Patent Description:
This specification relates to improving accelerated resource-intensive computation efficiency.

Hardware accelerators, such as Graphical Processing Units (GPUs) or Tensor Processing Units (TPUs), have much greater computational capacity compared to general-purpose processors, e.g., traditional Central Processing Units (CPUs). As a result, accelerators have started to drive much of the improvement in performance for critical workloads. For example, accelerators are widely used for machine learning training and inference tasks. <CIT> relates to thread contention in a multicore processor running high priority and low priority programs.

This specification describes a system implemented as computer programs on or more computers in one or more locations that manages the efficiency of a processing system that includes multiple general-purpose processing units.

The system splits a plurality of general-purpose processing units, e.g., CPU cores, into high-priority and low-priority domains. The general-purpose processing units in the high-priority domain are assigned to perform one or more tasks including one or more high-priority tasks, and the general processing units in the low-priority domain are assigned to perform one or more tasks including one or more low-priority tasks. Generally, the processing units in the low-priority domain are not assigned to perform any high-priority tasks. Moreover, the processing system generally also includes one or more hardware accelerators that are assigned a resource-intensive workload, e.g., a machine learning workload, and the high-priority tasks are tasks that are associated with that resource-intensive workload, i.e., tasks that support the workload assigned to the hardware accelerators.

During runtime of the processing system, the system obtains memory usage measurements that characterize usage of system memory by the high-priority domain and the low-priority domain. Based on the memory usage measurements, the system adjusts a configuration of (i) the high-priority domain, (ii) the low-priority domain, or (iii) both to adjust utilization of the system memory by the general-purpose processing units.

The system repeatedly obtains the usage measurements and adjusts the configurations during runtime to increase the efficiency of the processing system.

While hardware accelerators are responsible for the most heavily computational tasks in resource-intensive computation, general purpose processors, e.g., CPUs, often perform various supporting roles. For example, in a large-scale distributed machine learning system, CPUs may perform the supporting role of collecting and synchronizing machine learning model parameters. The supporting role of CPUs, however, may degrade system performance efficiency by competing with accelerators for shared resources, such as system memory. The described technology splits a plurality of general-purpose processing units into high-priority and low-priority domains, memory requests within each subdomain are handled by the corresponding memory controller and enjoy both lower memory latency and cache latency. Moreover, by filling the high-priority domain with low-priority CPU tasks, lost throughput due to fragmentation in domain-partitioning can be regained. Furthermore, by comparing measurements from performance counters during runtime, the system can choose to boost, throttle, or keep the resource configuration to reduce resource contention within and between high-priority and low-priority domains.

The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawing and description below.

Specific embodiments of the invention will now be described in detail with reference to the accompanying figures. Like elements in the various figures are denoted by like reference numerals for consistency.

<FIG> is a diagram of an example system <NUM> according to the invention that manages resource-intensive computation. The system <NUM> includes an accelerator package <NUM> designed to handle high-priority workloads such as machine learning tasks, and a processor package <NUM> designed to handle low-priority workloads such as CPU tasks. For example, a CPU task includes collecting the shared gradients from multiple accelerator packages. The accelerator package <NUM> is communicably coupled to the processor package <NUM> using one or more interfaces 112a and 112b. An optimization runtime system <NUM> manages the processor package <NUM> to improve its computation efficiency. For example, the optimization runtime system <NUM> is a set of computer programs running on a computer system including the processor package <NUM> and the accelerator package <NUM>.

The accelerator package <NUM> includes an accelerator engine <NUM> that performs the intensive computation associated with high-priority workloads. For example, the accelerator engine <NUM> can be a TPU or a GPU and the computation in the high-priority workload involves the training of a deep neural network, e.g., to repeatedly compute gradients of an objective function being used to train the neural network, or performing inference using the deep neural network, i.e., generating outputs using the deep neural network after the neural network has been trained.

The processor package <NUM> includes <NUM> includes cores 104a-104d responsible for performing computations, last-level-caches (LLCs) 106a-106d that store data for the computations, an interconnect 108a that connects different processing cores and LLCs, and memory controllers 1lOa-l lOb. While the processor package <NUM> mostly handles low-priority workloads, part of the computation from the high-priority workloads, e.g., memory -intensive computations, still runs on the processor package <NUM>. For example, the processor package <NUM> can play a supportive role of acting as a parameter server during the training of the neural network. As a parameter server, the processor package <NUM> during a machine learning task can collect shared gradients from multiple accelerator packages, can aggregate computed gradients, can update the parameters in real-time using the gradients, and then provide the updated parameter values to the accelerator packages. In another example, the processor packages <NUM> can perform an in-feed operation, in which the processor package interprets and reshapes input data before sending the data to the accelerator package <NUM>. In another example, the processor package <NUM> can handle irregular and complex supporting tasks such as beam search in machine translation applications.

As a result, in practice, the processor package <NUM> handles both low-priority tasks and certain parts of high-priority tasks. The low-priority tasks can interfere with the high-priority tasks by contending for shared resources such as in-pipeline resources, private caches shared through simultaneous multi-threading, last-level cache, and main memory bandwidth. To reduce performance bottlenecks, the optimization runtime system <NUM> splits the processor package <NUM> into a low-priority domain <NUM> and a high-priority domain <NUM>.

Each domain has its dedicated processing units, memory, and memory controllers. For example, the high-priority domain <NUM> includes dedicated cores l04a and l04b, dedicated LLCs l06a and l06b, and a dedicated memory controller llOa. The low-priority domain <NUM> includes dedicated cores l04c and l04d, dedicated LLCs l06c and l06d, and a dedicated memory controller 1lOb. For example, the optimization runtime system <NUM> can use Non-uniform memory access (NUMA) subdomain performance isolation technique to split the processor package <NUM>. As a result, the processor package <NUM> is exposed to an operating system running in a computer unit including the system <NUM> as two NUMA domains, e.g., the high-priority domain <NUM> and the low-priority domain <NUM>. Example techniques to implement NUMA subdomain performance isolation include sub-NUMA Clustering (SNC), Cluster-on-Die (CoD), and so on. A control groups interface <NUM> monitors, controls, and manages different groups of processes and their resource usages in the subdomains. Memory controllers 1lOa and 1lOb handle memory requests within each NUMA subdomain respectively. As a result, local memory requests experience both lower LLC and memory latency.

In an implementation according to the invention although the high-priority domain <NUM> has been isolated from the low-priority domain <NUM>, low-priority tasks can still interfere with the high-priority tasks due to a phenomenon called shared memory backpressure. Shared memory backpressure occurs when low-priority tasks in the low-priority domain <NUM> generate a large amount of memory traffic and saturate the corresponding memory controller 1lOb's bandwidth. In response, the memory controller 1lOb broadcasts a distress signal to all the cores 104a-104d across the processor package. When the cores 104a-104d receive the distress signal from the memory controller 1lO0b, they become throttled in order to avoid congesting the interconnect l08a. This mechanism is detrimental to the domain-splitting technique described above as each subdomain, e.g., the low-priority subdomain and the high-priority subdomain, already routes memory traffic internally. The memory saturation in the low-priority domain <NUM> itself has only minimal impact on the memory use in the high-priority domain <NUM>, but the shared memory backpressure causes the cores 104a- 104b in the high-priority domain <NUM> to be throttled nevertheless. As a result, the shared memory backpressure reduces the effectiveness of the memory interference protection implemented by the domain-splitting technique.

In an implementation according to the invention to reduce the effect of shared memory backpressure, the optimization runtime system <NUM> repeatedly measures the level of memory saturation in the low-priority domain <NUM>, the high-priority domain <NUM>, and/or the processor package <NUM> and, when appropriate, performs some actions to reduce the undesirable effects.

The optimization runtime system <NUM> uses existing hardware performance monitoring infrastructure such as measurements from the performance event FAST_ASSERTED from the Intel Uncore LLC coherence engine. This performance event reports the number of cycles in which the distress signal is asserted. The optimization runtime system <NUM> quantifies the memory saturation by dividing this cycle number by the number of total elapsed cycles between two measurements. The optimization runtime system <NUM> then disables cache prefetching for low-priority tasks in the low-priority domain <NUM> to reduce memory traffic. This disabling causes performance loss of low-priority tasks, but maintains performance in the high-priority domain <NUM>.

In some implementations, the optimization runtime system <NUM> backfills the high-priority domain <NUM> with low-priority tasks to improve system throughput. For example, the optimization runtime system <NUM> can be scheduled to run with the node-level scheduler runtime to gather necessary task information such as job priority and profile in both the high-priority domain <NUM> and the low-priority domain <NUM>. The optimization runtime system <NUM> assigns both high-priority tasks and low-priority tasks to designated domains, with low-priority tasks prioritized to be assigned to the low-priority domain <NUM> and high-priority tasks exclusively assigned to the high-priority domain <NUM>.

When a task is first scheduled on the processor package <NUM>, the optimization runtime system <NUM> receives high and low watermarks for each measurement of the task. The optimization runtime system <NUM> makes different measurements at specified time intervals, including:.

Where "socket-level" indicates that the measurements are taken across the entire processor package <NUM>. By comparing the measurements with the watermarks specified in the task profile, the optimization runtime system <NUM> can choose to boost, throttle, or keep the resource configuration for low-priority tasks in each domain. <FIG> and the related descriptions explain in detail the node-level resource management logic used by the optimization runtime system <NUM>.

In summary, the optimization runtime system <NUM> operates when the processor package <NUM> are assigned both high-priority tasks and low-priority tasks. The optimization runtime system <NUM> improves the performance of the processor package <NUM> by redistributing computing resources between the high-priority tasks and the low-priority tasks. As a result, the high-priority tasks are isolated from interference by the low-priority tasks, e.g., such as memory interference.

<FIG> is a flowchart of an example process <NUM> according to the invention for managing resources on multiple processing units. For convenience, the process <NUM> will be described as being performed by a system, e.g., the optimization runtime system <NUM> of in <FIG>.

The system can perform the process <NUM> to configure resources on multiple processing units, e.g., the processor package <NUM>, to improve performance for both high-priority and low-priority tasks.

As the first step, the system splits the multiple processing units into a high-priority domain and a low-priority domain (<NUM>). As described in <FIG>, the system assigns both high-priority tasks and low-priority tasks to the high-priority domain, and assigns only low-priority tasks to the low-priority domain. Low-priority tasks are prioritized to be assigned to the low-priority domain. Example high-priority tasks include machine learning tasks, and example low-priority tasks include CPU tasks.

The system then obtains shared system resource usage measurements across the high-priority and the low-priority domains (<NUM>). The system can make four types of measurements across the multiple processing units, including (<NUM>) socket-level memory bandwidth, (<NUM>) socket-level memory latency, (<NUM>) socket-level memory saturation, and (<NUM>) high-priority domain memory bandwidth. The system can take the measurement at a specified time interval to cause negligible performance overhead, e.g., every <NUM> seconds.

In some implementations, the system has previously collected a task profile when the task is first loaded onto the multiple processing units. The task profile includes high and low watermarks for each of the above-mentioned measurements.

By comparing the real-time measurement against the high and low watermarks, the system detects potential performance bottlenecks and configures the memory usage by the high-priority domain (<NUM>) and by the low priority domain (<NUM>). The system can disable or enable cache prefetching for processing cores in the low-priority domain, and can activate or deactivate processing cores in both domains. Configuring the high-priority and low-priority domain is described below with reference to <FIG>.

The system repeatedly performs steps <NUM>-<NUM> during the performance of the task to improve overall system performance.

<FIG> is a flowchart of an example software logic <NUM> for configuring resource on multiple processing units. For convenience, the software logic <NUM> is described as being performed by a system, e.g., the optimization runtime system <NUM> of <FIG>.

As described in <FIG>, after the system compares the real-time measurements against the high and low watermarks of the running tasks, the system configures resources on the processing unit to reduce performance bottlenecks.

The system measures socket-level memory latency, socket-level memory bandwidth, socket-level memory saturation, and high-priority domain memory bandwidth. By comparing the measurements against the high and low watermarks, the system determines whether the current measurements are "high" or "low. " For example, the system can determine that a measured value being greater than <NUM>% of the high watermark to be "high," and being smaller than <NUM>% of the low watermark to be "low. " The system configures the resources on the multiple processing units based on the following rules:.

To throttle or boost the high-priority domain, the system increases or reduces the number of cores in the high-priority domain, respectively. To throttle or boost the low-priority domain, the system increases or reduces the number of cores in the low-priority domain, respectively, and increase or reduce the number of cores using prefetching in the low-priority domain, respectively.

To throttle the high-priority domain, the system checks if the number of cores operating in the high-priority domain is greater than a minimum number of cores, e.g., as defined in the corresponding task profile (<NUM>). If so, the system reduces the number of operating cores in the high-priority domain by one.

To boost the high-priority domain, the system checks if the number of cores operating in the high-priority domain is smaller than a maximum number of cores, e.g., as defined in the corresponding task profile (<NUM>). If so, the system increases the number of operating cores in the high-priority domain by one.

To throttle the low-priority domain, the system checks if the number of cores using prefetching in the low-priority domain is greater than zero (307a). If so, the system closes half of the prefetching cores in the low-priority domain. Furthermore, if the number of operating cores in the low-priority domain is greater than a minimum number of cores (307b), the system reduces the number of operating cores in the low-priority domain by one.

To boost the low-priority domain, the system checks if the number of prefetching cores is smaller than the number of operating cores in the low-priority domain (309a). If so, the system increases the number of prefetching cores in the low-priority domain by one. Furthermore, the system checks if the number of operating cores is smaller than the maximum number of cores in the low-priority domain (309b). If so, the system increases the number of operating cores in the low-priority domain by one.

The system is more aggressive in disabling prefetching cores (closing half of the cores in throttle mode but only increase one core in boost mode) in order to prioritize high-priority task performance.

For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on its software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions.

Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible nontransitory storage medium for execution by, or to control the operation of, data processing apparatus.

A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or another unit suitable for use in a computing environment. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, subprograms, or portions of code.

Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions.

The processes and logic flow described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output.

Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks.

Also, a computer can interact with a user by sending text messages or other forms of a message to a personal device, e.g., a smartphone that is running a messaging application and receiving responsive messages from the user in return.

Particular embodiments of the subject matter have been described.

Claim 1:
A method (<NUM>) comprising:
obtaining (<NUM>) data splitting a plurality of general-purpose processing units (104a-104d) in a processor package (<NUM>) of a processing system (<NUM>) into a high-priority domain (<NUM>) and a low-priority domain (<NUM>), each domain being associated with a respective memory controller (110a, 110b),
wherein the processing system (<NUM>) further comprises an accelerator package (<NUM>) comprising a plurality of special-purpose hardware accelerators assigned to perform an accelerated workload (<NUM>), wherein
the general-purpose processing units (104a-104d) in the high-priority domain (<NUM>) are assigned to perform tasks comprising one or more high-priority tasks and one or more low-priority tasks,
the general-purpose processing units (104a-104d) in the low-priority domain (<NUM>) are assigned to perform only low-priority tasks, and
wherein the high-priority tasks are tasks which support the accelerated workload (<NUM>);
during runtime of the processing system (<NUM>), obtaining (<NUM>) shared memory usage measurements that characterize usage of shared system memory by the high-priority domain (<NUM>) and the low-priority domain (<NUM>); and
adjusting (<NUM>, <NUM>), based on the shared memory usage measurements, a configuration of (i) the high-priority domain, (ii) the low-priority domain, or (iii) both to adjust utilization of the shared system memory by the general-purpose processing units running low-priority tasks ;
wherein the shared memory usage measurements comprise a measurement of system memory saturation, including:
broadcasting a distress signal to all of the general-purpose processing units (104a-104d) in the processor package (<NUM>) when low-priority tasks in the low priority domain generate memory traffic that saturates the corresponding memory r controller's (110b) bandwidth;
using a performance monitoring infrastructure to report a number of cycles in which the distress signal is asserted;
quantifying the system memory saturation by dividing the number of cycles by a total number of cycles elapsed between two measurements;
determining that the system memory saturation is high when the shared memory usage measurements exceed a high system memory saturation watermark; and
in response to determining that the system memory saturation is high, reducing (<NUM>) a number of active general purpose processors utilizing cache prefetching in the low-priority domain (<NUM>).