Reducing computation in neural networks using self-modifying code

In various implementations, provided are systems and methods for reducing neural network processing. A compiler may generate instructions from source code for a neural network having a repeatable set of operations. The instructions may include a plurality of blocks. The compiler may add an overwrite instruction to the plurality of blocks that, when executed by one or more execution engines, triggers an overwrite action. The overwrite action causes the instructions of subsequent blocks to be overwritten with NOP instructions. The overwrite action is triggered only when a condition is satisfied.

BACKGROUND

Artificial intelligence is an area of research and engineering seeking to build intelligent machines that can make decisions in the same way that humans do. Artificial neural networks fall within a sub-field of artificial intelligence called machine learning. Machine learning is a field of study that investigates giving computers the ability to learn without being explicitly programmed. A program that implements a machine learning algorithm is able to learn to perform tasks without requiring explicit code in the program to account for every possibility or all possible behaviors.

The architecture of a neural network may include an input layer, an output layer, and a number of intermediate layers, often referred to as hidden layers. Each layer executes a computation on the outputs of the previous layer, with the last layer (the output layer) providing a final result. With more layers, a neural network can, theoretically, perform more complex tasks, such as language translations and identifying (or classifying) the contents of an image. A neural network with more than three hidden layers is sometimes referred to as a deep neural network. Deep neural networks can have many hidden layers, such as, for example, between five and more than a thousand layers.

Neural networks can be implemented using a Central Processing Unit (CPU) to perform the computations. CPUs, however, tend to be optimized for sequential rather than parallel computations, and thus can suffer from poor response times. Graphics Processing Units (GPUs) are optimized for parallel computations, but not necessarily for the result from one computation unit to be provided directly to another computation unit. Often, the result must first be written to a memory and then read back. Although GPUs can have better response times than CPUs, it would still be desirable to improve the execution time of a neural network.

DETAILED DESCRIPTION

In some neural networks, such as recurrent neural networks, the operations performed by a particular layer or node are repeated multiple times, each time with different input data based on the output data of the previous iteration. Such neural networks may be represented with a data flow graph having a feedback loop that indicates that the output data from a particular layer/node is fed back as the input data for the particular layer/node (or, more generally, the repeated set of operations may include a single layer/node and/or multiple layers/nodes). The number of iterations that are performed on the set of operations may be static or dynamic, as dictated by the control flow of the neural network. For example, the number of iterations may be static if the set of operations are implemented in a “for” loop with no break conditions. In contrast, the number of iterations may be dynamic if the set of operations are implemented in a “while” loop or in a “for” loop with break conditions, among other possibilities.

Many processors, such as single instruction multiple data (SIMD) processors, may not directly support the execution of a repeatable set of operations having a dynamic length. In many instances, the repeatable set of operations are compiled having a static length for the maximum number of iterations, and the output data from any redundant computations beyond the desired number of iterations is discarded. For example, a neural network may include a repeatable set of operations having 20 maximum iterations with break conditions that are highly likely to be satisfied much prior to the 20thiteration. Nonetheless, the instructions for the repeatable set of operations are still compiled to have 20 iterations. During runtime, a processor may execute the first 5 iterations, and determine after the 5thiteration or prior to the 6thiteration to exit the loop. The last 15 iterations are still executed by the processor, and the output data that is generated is discarded. The computations performed by the processor during the last 15 iterations are considered to be redundant, as the data has no influence on the final result of the neural network. Such redundant computation is inefficient and decreases workload throughput.

Examples described herein overcome the above-noted inefficiencies in processors, such as SIMD processors, that do not support dynamic control flow. Some examples provide for self-modifying code that, when executed, triggers an overwrite action that overwrites instructions in redundant code blocks with no operation (NOP) instructions (e.g., instructions that do nothing without disrupting the timing mechanisms of the processor). Placing NOP instructions in redundant code blocks can be advantageous because a NOP instruction can be performed in fewer cycles than the instruction that was overwritten. The self-modifying code may be added during the compilation process and may be executed during runtime by an execution engine.

In some examples, the compiler may first detect the presence of a repeatable set of operations having a dynamic length in the source code for a neural network. In some instances, the compiler scans the source code and identifies particular patterns indicative of a repeatable set of operations. For example, the compiler can parse the source code for “for” loops or “while” loops. The compiler then generates instructions having multiple blocks for the repeatable set of operations, each block corresponding to a single iteration, and the total number of blocks corresponding to the maximum number of iterations. The compiler may then generate additional instructions for each of the blocks. The additional instructions may include an evaluation instruction that, when executed, determines whether the break condition is satisfied (e.g., by determining the current value of a variable and comparing the current value to a predetermined value, etc.). The additional instructions may also include an overwrite instruction that, when executed, triggers an overwrite action. The overwrite action may only be triggered when the break condition is satisfied. Accordingly, the evaluation instruction may immediately precede the overwrite instruction or, alternatively or additionally, the overwrite instruction may incorporate the evaluation instruction.

Triggering of the overwrite action causes the instructions in each of the subsequent blocks to be overwritten with NOP instructions. The overwrite action may be carried out using a direct memory access (DMA) engine that transfers NOP instructions to the subsequent blocks immediately upon the execution engine initializing the DMA engine. The segments of instructions that are overwritten is determined by the location of the particular overwrite instruction that triggered the overwrite action. As one example, the compiler may add the additional instructions at the end of a block, and triggering of the overwrite action may cause instructions of each subsequent block to be overwritten. As another example, the compiler may add the additional instructions at the beginning of a block, and triggering of the overwrite action may cause instructions of the current block and of each subsequent block to be overwritten.

In various implementations, an integrated circuit can include an acceleration engine to make use of self-modifying code to reduce the number of computations that the integrated circuit needs to execute in order to perform a task for which the neural network was trained. The integrated circuit can include an array of processing engines for executing parallel, cascading computations. The integrated circuit can further include memory banks, placed local to the array of processing engines (e.g., on the same die), for fast, temporary storage of weight values and instructions for a neural network, and for storing intermediate results. The integrated circuit can further include DMA channels for performing transfers between the memory banks.

FIG.1illustrates an example of a computational flow model100for a neural network. Neural networks take inspiration from the mechanics of the operation of the human brain. According to various models of the brain, the main computational element of the brain is the neuron. Neurons are connected together with a number of elements, with elements entering a neuron being referred to as dendrites and an element leaving a neuron being referred to as an axon. A neuron accepts signals via dendrites, performs a computation on the signals, and outputs a signal on an axon. The input and output signals are referred to as activations. The axon of one neuron can branch out and be connected to the dendrites of multiple neurons. The connection between a branch of an axon and a dendrite is called a synapse.

A synapse can scale the signal crossing the synapse. The scaling factor is referred to as a weight, and is thought of as the way a brain is able to learn: different weights result from different responses to input. Learning can change the weights, but the organization of the neurons and synapses need not change to obtain the learning. The static structure of the brain can thus be used as a model for a program, and the weights can reflect tasks that the program has learned to perform.

Neural networks operate on the notion that a neuron's computation involves a weighted sum of input values. These weighted sums correspond to the value scaling performed by the synapses and the combining of those values in the neuron. A functional operation is performed in the neuron on the combined inputs. In the brain model, the operation appears to be a non-linear function that causes the neuron to generate an output only when the inputs cross some threshold. Thus, by analogy, the nodes of a neural network can apply a non-linear function to the weighted sum of the values input into the nodes.

In the illustrated example, the model100includes an input layer104, a middle layer that is often referred to as a hidden layer106, and an output layer108. Each layer includes some number of nodes102. In this example, the nodes102of the input layer104are connected to each node102of the hidden layer106. The connections, which would be referred to as synapses in the brain model, are referred to as weights110. Also in this example, each node102of the hidden layer106has a connection or weight110with each node102of the output layer. The input layer104can receive inputs and can propagate the inputs to the hidden layer106. A neural network implementation can include multiple hidden layers. Weighted sums computed by the hidden layer106(or multiple hidden layers) are propagated to the output layer108, which can present final outputs to a user. The outputs of the nodes102can be referred to as activations, in keeping with the brain model.

An example of a computation that can occur at each layer in the example model100is as follows:

In the above equation, Wijis a weight, xiis an input activation, yiis an output activation, f( ) is a non-linear function, and b is a bias term. Various non-linear functions can be used to achieve different purposes.

The model100can be referred to as a directed, weighted graph. In a directed graph, each connection to or from a node indicates a direction (e.g., into the node or away from the node). In a weighted graph, each connection can have a weight. Tools for developing neural networks can visualize the neural network as a directed, weighted graph, for ease of understanding and debuggability. In some cases, these tools can also be used to train the neural network and output trained weight values. Executing the neural network is then a matter of using the weights to conduct computations on input data.

Neural networks with many layers can be capable of learning high-level features having more complexity and abstraction than shallower networks. As an example, a neural network can be taught to recognize images. In this example, pixels of an image can be fed into the input layer of the neural network, and the outputs of the first layer can indicate the presence of low-level features in the image, such as lines and edges. At subsequent layers, these features can be combined to measure the likely presence of higher level features: the lines can be combined into shapes, which can be further combined into sets of shapes. Given all this information, the neural network can output a probability that the high-level features represent a particular object or scene. For example, the neural network can output whether an image contains a cat or does not contain a cat.

The learning phase of a neural network is referred to as training the neural network. During training, the neural network is taught to perform a task. In learning the task, values for the weights (and possibly also the bias) are determined. The underlying program for the neural network (e.g., the organization of nodes into layers, the connections between the nodes of each layer, and the computation executed by each node), does not need to change during training. Once trained, the neural network can perform the task by computing a result using the weight values that were determined during training. For example, the neural network can output the probability that an image contains a particular object, can output the probability that an audio sequence contains a particular word, can generate a bounding box around an object in an image, or can propose an action that should be taken, etc. Running the program for the neural network is referred to as inference.

There are multiple ways in which weights can be trained. One method is called supervised learning. In supervised learning, all training samples are labeled, so that inputting each training sample into a neural network produces a known result. Another method is called unsupervised learning, where the training samples are not labeled and training aims to find a structure in the data or clusters in the data. Semi-supervised learning falls between supervised and unsupervised learning. In semi-supervised learning, a subset of training data is labeled. The unlabeled data can be used to define cluster boundaries and the labeled data can be used to label the clusters.

Neural networks have been used for a variety of applications, including, for example, in the areas of image and video, speech and language, medicine, game play, and robotics. In image and video, neural networks have been used for image classification, object localization and detection, image segmentation, and action recognition. In speech and language, neural networks have been used for speech recognition, machine translation, natural language processing, and audio generation. In the medical field, neural networks have been used in genomics and medical imaging. In game play, neural networks have been used to play video and board games, including games with immense numbers of possible moves such as Go. In robotics, neural networks have been used for motion planning of a robot, visual navigation, control stabilization, and driving strategies for autonomous vehicles.

Different varieties of neural networks have been developed. Various examples of neural networks can be divided into two forms: feed-forward and recurrent.FIG.2illustrates an example of a computational flow model200for a neural network that includes feed-forward weights212between an input layer204and a hidden layer206, and recurrent weights214at the output layer208. In a feed-forward neural network, the computation is a sequence of operations on the outputs of a previous layer, with the final layer generating the outputs of the neural network. In the example illustrated inFIG.2, feed-forward is illustrated by the hidden layer206, whose nodes202operate only the outputs of the nodes202in the input layer204. A feed-forward neural network has no memory and the output for a given input can be always the same, irrespective of any previous inputs given to the neural network. The Multi-Layer Perceptron (MLP) is one type of neural network that has only feed-forward weights.

In contrast, recurrent neural networks have an internal memory that can allow dependencies to affect the output. In a recurrent neural network, some intermediate operations can generate values that are stored internally and can be used as inputs to other operations, in conjunction with the processing of later input. In the example ofFIG.2, recurrence is illustrated by the output layer208, where the outputs of the nodes202of the output layer208are connected back to the inputs of the nodes202of the output layer208. These looped-back connections can be referred to as recurrent weights214. Long Short-Term Memory (LSTM) is a frequently used recurrent neural network variant.

FIG.3illustrates an example of a two-dimensional convolution, which could be performed during image recognition. In this example, a filter plane304is a set of weights arranged in a matrix having a height R and a width S. The values in the filter plane304can be selected to filter for particular features, such as lines, edges, curves, corners, blobs, ridges, and so on. The filter plane304can also be referred to as a kernel or a feature detector.

The filter plane304is applied to a two-dimensional matrix of values that represent the input to the convolution. The two-dimensional matrix is referred to as an input feature map306. The input feature map306can include values for a component of the input. For example, when the input is a color image, the input feature map306can include the color values for one color, such as red, for each pixel in the image, with the values indicating an intensity of the color. In this example, additional feature maps can include the other color values for the pixels, one for blue and one for green. In this example, each input feature map is treated as a separate channel. In a black and white image, each pixel value can be represented using a single value that indicates an intensity between white and black. Thus, in some examples, black and white images can be represented using a single channel.

The convolution operation involves computing a value for each possible position of the filter plane304over the input feature map306by multiplying each filter plane304by the corresponding feature map value and summing the result. For example, at a first position316, multiply each value in the filter plane304by each corresponding value in the first position316results in a matrix {(1, 0, 1), (0, 1, 0), (0, 0, 1)}. In this example, the sum of the values in the matrix results in the value 4, which is placed in a corresponding first position318in an output feature map308. A region of values from the input feature map306can be referred to as input activations. The result of the multiplication and summation can be referred to as an output activation. The output feature map308represents a higher-level abstraction of the input feature map306, and has a height E and a width F. In various examples, additional filters can be applied to the same input feature map306to produce additional output feature maps.

FIG.4is an example configuration of a convolutional neural network400. The example ofFIG.4illustrates operations that can be included in a convolutional neural network, including convolution, application of non-linearity, pooling or sub-sampling, and output generation (e.g., a fully connected layer). Any given convolutional network includes at least one convolution layer, and can have tens of convolution layers. Additionally, each convolutional layer need not be followed by a pooling layer. In some examples, a pooling layer may occur after multiple convolution layers, or may not occur at all. The example convolution network illustrated inFIG.4classifies an input image420into one of four categories: dog, cat, boat, or bird. In the illustrated example, on receiving an image of a boat as input, the example neural network outputs the highest probability for “boat” (0.94) among the output predictions414.

To produce the illustrated output predictions414, the example convolutional neural network performs a first convolution402that can also include application of non-linearity; a first pooling404operation; a second convolution406that may also include application of non-linearity; a second pooling408operation; and then categorization using a first fully-connected410layer and a second fully-connected412layer. The output of the first convolution402is a set of one or more output feature maps422, which are provided as inputs to the first pooling404operation. The first pooling404operation produces a set of feature maps424that are provided as inputs to the second convolution406. The second convolution406also produces a set of output feature maps426, which can describe the input image420at a more abstract level. The second pooling408step also produces feature maps428, which are input into the first fully-connected410layer. The first fully-connected410layer accumulates the values in the feature maps428, and the result is input into the second fully-connected412layer. The outputs of the second fully-connected412layer are the output predictions414.FIG.4is an example configuration of a convolutional neural network. Other examples can include additional or fewer convolution operations, non-linearity operations, pooling operations, and/or fully-connected layers.

Non-linearity can be added after some convolution operations. Convolution is a linear operation, and in some examples, it is assumed that the real-world data being learned by the convolutional neural network is non-linear. Thus, a non-linear function can be applied, element-wise, to the output feature maps from a convolution. One such non-linear function is provided by a Rectified Linear Unit (ReLU), whose output is given by Output=Max (0, Input). Other non-linear functions may include tan h and sigmoid.

Pooling, which can also be referred to as sub-sampling or down-sampling, can reduce the dimensionality of a feature map while retaining the most important information. Pooling can include, for example, taking a region of values in the matrix of a feature map (e.g., a 2×2 neighborhood, or a neighborhood of another size), and determining a maximum value across the values in the region. Alternatively, average, sum, or another function can be used as the pooling function.

Pooling can be used to progressively reduce the spatial size of the input representation. For example, pooling can make the input representations (e.g., the feature dimension) smaller and more manageable. As another example, pooling can reduce the number of parameters and computations that need to be performed by the neural network. As another example, pooling can make the neural network invariant to small transformations, distortions, or translations in the input image. That is, a small distortion in the input is not likely to change the output of the pooling, since the maximum (or average, or sum, or some other operation) is taken in a local neighborhood. As a further example, pooling can assist in determining an almost scale invariant representation of the image (referred to as an equivariant representation). This means that an object can be detected in an image no matter where the object is located within the image.

As illustrated by the example ofFIG.4, a convolutional neural network can include multiple convolution layers, with each layer refining the features extracted by a previous layer. Each convolution layer may be, but need not be, followed by pooling. The output of a combination of these layers represent high-level features of the input image, such as the presence of certain shapes, colors, textures, gradients, and so on.

To turn these feature maps into a classification, a convolutional neural network can include one or more fully-connected layers. A Multi-Layer Perceptron that uses, for example, a softmax activation function or another logistic function, can be used after a fully-connected layer. A fully-connected layer can classify the input image into various classes based on training data. For example, the convolutional neural network ofFIG.4was trained to recognize dogs, cats, boats, and birds, and can classify an input image as including one of these classes.

Apart from classification, a fully-connected layer in a convolutional neural network might also provide an inexpensive (in computational and/or data storage terms) way to learn non-linear combinations of the extracted features. The features extracted by the convolution and pooling layers may be good for making a classification, but combination of the features may be better.

In the example ofFIG.4, the sum of the output predictions414is 1, due to the output layer using the softmax activation function. The softmax function takes a vector of arbitrary real-valued scores and compresses these values into a vector of values between zero and one that add up to one.

Research has found that the more convolution steps a neural network has, the more features the network will be able to learn to recognize. For example, in an image classification example, in a first layer, the neural network may learn to detect edges from the raw pixels, then in a second layer use the edges to detect shapes, and in a third layer, the neural network may be able to use the shapes to determine higher-level features, such as facial shapes, in higher layers.

In the training of a convolutional neural network, parameters such as the number of filters, the filter sizes, and the organization of the layers remain unchanged. During training, only the values of the filter matrices and connection weights are changed. Once trained, a neural network includes the weights determined during the training and a set of instructions describing the computation to be executed at each layer and/or node of the network. In some examples, the number of weights can be on the order of 5 million to 100 million. In some examples, a weight value can be represented using a 32-bit number, in which case 5 million to 100 million weights can require about 20 megabytes (MB) to 400 MB to store. In some examples, the number of weights can be as few as 1.5 million.

Operation of a neural network (e.g., conducting inference) involves fetching input data or input activations, executing multiply-and-accumulate operations in parallel for each node in a layer, and providing output activations. Optimum performance of a neural network, measured by accuracy and/or response time, can be achieved when a hardware architecture is capable of highly parallelized computations. Central Processing Units (CPUs), which can also be referred to as general purposed processing units, can have multiple cores, (e.g., 2 to 64 or more cores) and can increase parallelism through use of multiple execution threads. CPU cores, however, tend to be optimized for sequential processing. For example, a computation engine (e.g., an arithmetic logic unit (ALU)) of a core obtains operands from memory and writes a result to memory, such that memory operations are required for sequential computations. In this example, each memory operation can require management by control logic of the CPU. For this and other reasons, CPUs tend to have slow response times when performing inference for a neural network.

In contrast to CPUs, Graphics Processing Units (GPUs) achieve parallelism by having thousands of small and efficient cores, configured specifically for conducting parallel computations. GPUs thus can achieve far better performance than a CPU when executing a neural network. Individual GPU computation engines, however, can still be primarily sequential in nature, such that memory operations are required for the outputs of one computation engine to be provided to the inputs of another.

Special-purpose acceleration engines can achieve better performance than both CPUs and GPUs when executing a neural network. Acceleration engines can employ a spatial architecture, in which computation engines form processing chains and can pass data directly from one computation engine to another. This can significantly reduce the number of memory transactions needed to conduct inference. In some examples, acceleration engines can also include an on-chip buffer that can store values read from processor memory, and that can distribute values to multiple computation engines in the acceleration engine. The computation engines can further include a small, local register file (e.g., a small memory) for storing intermediate results. Having an on-chip memory hierarchy can improve the efficiency of the operation of a neural network by reducing memory latencies.

FIG.5illustrates a simplified example of the organization of a neural network500. The neural network500of this example includes an input layer504, a hidden layer506, and an output layer508, which produces a result510. The layer506can include a repeatable set of operations that are performed N times. During the first iteration, the output data from the preceding layer (e.g., the input layer504) can be used as the input data for the layer506. During subsequent iterations, the output data from the preceding iteration can be used as the input data for the layer506. After the last iteration, the output data is fed into the subsequent layer (e.g., the output layer508). The layer506may include a break condition that, when satisfied, causes the layer506to stop repeating and the output data of the current layer to be fed into the subsequent layer. The layers of the neural network500can be fully connected layers, sparsely connected, and/or can include recurrent weights.

FIG.6illustrates a simplified example of the organization of a neural network600. The neural network600may be similar to the neural network500. For example, the neural network600may be an unrolled version of the neural network500, illustrating each of the iterations of the layer506. The neural network600of this example includes an input layer604, unrolled hidden layers606, and an output layer608, which produces a result610. The hidden layers606may include a first layer606-1, a second layer606-2, a third layer606-3, through an Nthlayer606-N. The layer606-2receives as input the output of the layer606-1, the layer606-3receives as input the output of the layer606-2, and the like. Each of the layers606can include the same set of operations, such as convolutional operations, non-linearity, and/or pooling. Alternatively or additionally, the layers can be fully connected, sparsely connected, and/or can include recurrent weights.

The example neural network600also includes an intermediate operation between the layer606-1and the layer606-2, which can be grouped into what is referred to herein as a conditional layer626. In various implementations, the conditional layer626can include a function622for computing an intermediate result from the outputs of the layer606-1or from a variable associated with the layer606-1. Execution of the conditional layer626can further include testing624the result computed by the function622against a break condition628. When testing624determines that the break condition628is not satisfied by the result, an overwrite action630is not triggered and execution of the neural network proceeds with the layer606-2and continues to the output layer608. When testing624determines that the break condition628is satisfied, in this example, the overwrite action630is triggered, causing the layers606-2through606-N to be modified as described herein.

The function622used in the conditional layer626can include a logistic function, such as softmax. The softmax function combines the values in a vector of arbitrary values to a vector of values in the range of (0, 1) that add up to 1. The output of the softmax function can be used to represent a categorical distribution. Some or all of the outputs of the layer606-1can be input into softmax to produce a result. In various examples, other logistic functions can be used.

The break condition628can describe circumstances under which the result of the function622satisfies the break condition. For example, the break condition628can include a test value, against which the result is tested. Additionally, in some implementations, the condition628can include a test that is to be applied at the testing624step. For example, the test can be to compare the result against the test value, and when the result is greater than (or less than, or equal to, or not equal to, or some other comparison) the test value, then the break condition is met. For example, the break condition628can provide a test value of 90%, and indicate that, when the result indicates a greater than 90% probability, then the break condition628is satisfied.

In some examples, the function622outputs a set of values. For example, softmax can output a vector. In these examples, the break condition628can be that one or more values from the set of values should meet a condition. For example, the five largest values from the set (or the two largest, or seven largest, or some other number of largest values) can be summed, and the testing624can determine whether the sum meets the condition. For example, the testing624can determine whether the sum is greater than a threshold value. In these and other examples, the condition can be less stringent than when only the largest value from the set is tested against the condition.

In some implementations, the test value and/or the test being applied can be specified separately from the definition of the neural network. For example, an acceleration engine can include a register or set of registers which can store the test value, a test, and/or to which conditional layer626the test value and/or test applies. In some examples, the acceleration engine can include a test value and/or test for each conditional layer626, or for all conditional layers. In some examples, the acceleration engine can include separate test values and/or tests for each task the neural network is trained to perform.

FIG.7illustrates a simplified example of the organization of a neural network700. The neural network700may be similar to the neural networks500and600. The neural network700includes an input layer704, hidden layers706, and an output layer708, which produces a result710. The layers706may include a first layer706-1, a second layer706-2, a third layer706-3, through an Nthlayer706-N. The example neural network700ofFIG.7includes multiple conditional layers726positioned between each of the layers706and multiple overwrite actions730. For example, a first conditional layer726-1may be positioned between the layers706-1and706-2and may trigger a first overwrite action730-1, a second conditional layer726-2may be positioned between the layers706-2and706-3and may trigger a second overwrite action730-2, and the like. In some examples, the conditional layers726may be incorporated into the layers706. For example, the conditional layer726-1may be incorporated into the layer706-1and the conditional layer726-2may be incorporated into the layer706-2, and the like.

FIG.8illustrates an example neural network800and the associated instructions corresponding to different layers806of the neural network800. The neural network800may be similar to the neural networks500,600, and700. The layers806may include a first layer806-1, a second layer806-2, a third layer806-3, through an Nthlayer806-N. When the neural network800is compiled into instructions, the compiler may generate N blocks of instructions for the layers806. Each of the blocks of instructions may include instructions832corresponding to the repeatable set of operations and additional instructions834may correspond to the conditional layers626and726described in reference toFIGS.6and7. In some examples, the additional instructions834contain an evaluation instruction and an overwrite instruction. For each block of instructions, the compiler may add/insert the additional instructions834before, after, or in the middle of the instructions832. In the illustrated embodiment, the additional instructions834are added after the instructions832for each block except for the last block, which does not contain any additional instructions834.

In one example, during runtime, an execution engine may execute the instructions832-1to perform the set of operations on the input data. The execution engine may then execute the evaluation instruction of the additional instructions834-1, causing the execution engine to determine whether the break condition is satisfied. If it is determined that the break condition is satisfied, then the overwrite action730-1is triggered, causing the instructions832-2through832-N to be overwritten with NOP instructions. In some examples, the additional instructions834of subsequent blocks may also be overwritten with NOP instructions. If it is determined that the break condition is not satisfied, then the overwrite action730-1is not triggered and the execution engine continues executing instructions.

Continuing with the above example, assuming the overwrite action730-1was not triggered, the execution engine may then execute the instructions832-2to perform the set of operations on the output data of the layer806-1that was generated by the instructions832-1. The execution engine may execute the evaluation instruction of the additional instructions834-2, causing the execution engine to determine whether the break condition is satisfied. If it is determined that the break condition is satisfied, then the overwrite action730-2is triggered, causing the instructions832-3through832-N to be overwritten with NOP instructions. If it is determined that the break condition is not satisfied, then the overwrite action730-2is not triggered and the execution engine continues executing instructions.

Continuing with the above example, assuming the overwrite actions730-1and730-2were not triggered, the execution engine may then execute the instructions832-3to perform the set of operations on the output data of the layer806-2that was generated by the instructions832-2. The execution engine may execute the evaluation instruction of the additional instructions834-3, causing the execution engine to determine whether the break condition is satisfied. If it is determined that the break condition is satisfied, then the overwrite action730-3is triggered, causing the instructions832-4(not shown) through832-N to be overwritten with NOP instructions. If it is determined that the break condition is not satisfied, then the overwrite action730-3is not triggered and the execution engine continues executing instructions.

FIG.9includes a block diagram illustrating an example of a host system900on which a compiler930can run. The illustrated host system900is an example of a computing device, and includes a processor902, a processor memory904, at least one storage device906, various Input/Output (I/O) devices908, and at least one network interface910. In the example ofFIG.9, the host system900also includes an acceleration engine912, which is an integrated circuit device that can accelerate certain operations or computations performed by the host system900. In various examples, the host system900can be implemented as a server in a data center, a desktop computer, a laptop computer, a tablet computer, or a smartphone, among other examples. In some examples, operations or components discussed below as performed or included in the host system900can be performed or included in other computer devices. For example, the compiler930can execute on the host system900while the acceleration engine912is located at a different host system.

The processor902is an integrated circuit device that can execute program code, in the form of instructions. The program code can be used for various software applications or tools, such as an operating system920. While the processor902is executing a program, the instructions for the program can be stored in the processor memory904. The instructions can also be stored elsewhere, such as on the storage device906, and can be loaded into the processor memory904when needed by the processor902. The processor902can also use the processor memory904for temporary storage of other data on which the processor902is operating. In various examples, the processor memory904is a volatile memory type, such as a type of Random Access Memory, though non-volatile memory types can, alternatively or additionally, be used for the processor memory904.

The storage device906is an example of a device that can include non-volatile memory. For example, the storage device906can be a magnetic disk drive, a solid state drive, or an optical drive, among other examples. The storage device906can further be non-transitory, such that program code and other data stored on the storage device906remains present when the storage device906is not powered on.

The storage device906is one example of a peripheral device, which are components that can be coupled to the host system900to add functionality to the host system900. Other examples of peripheral devices include the Input/Output devices908and the network interface910. The Input/Output devices908can include user input and output devices, such as keyboards, mice, touch screens, microphones, display screens, speakers, printers, and scanners, among other examples. The network interface910, which can be implemented using a network interface card, can provide access to one or more networks. The network interface910can include, for example, a physical port for connecting a network cable and/or wireless antennas for communicating with Wi-Fi and/or cellular networks. The network interface910can also be described as an I/O device.

The acceleration engine912is also another type of peripheral device or I/O device. The acceleration engine912is a device that is purpose built to perform certain operations that can be performed by the processor902, but can be performed faster by the acceleration engine912. For example, the acceleration engine912can be a neural network accelerator, and, as such, may be able to perform the large scale, parallel computations of a neural network more efficiently than when the computations are performed by the processor902. As another example, the acceleration engine912can be a graphics processing unit (GPU), and may be optimized to perform the computations needed for graphics rendering. Other examples of devices that can be implemented by the acceleration engine912include cryptographic accelerators, compression and decompression accelerators, 3-D accelerators, regular expression accelerators, security accelerators, and others.

In various examples, the acceleration engine912can execute program code to perform certain operations. For example, when the acceleration engine912is a neural network accelerator, the acceleration engine912can be programmed to execute a particular neural network, such as one that performs image recognition or one that performs machine translation. As a further example, to support the execution of a neural network, the acceleration engine912can be programmed to perform operations such as copying data for the neural network from processor memory904(for example) into the acceleration engine912, copying input data for the neural network from processor memory904into the acceleration engine912, and/or copying results from the acceleration engine912into the processor memory904, among other examples.

To generate program code for the acceleration engine912, in various examples, the host system900can execute the compiler930. Compilers, in general, are software programs that translate program code written in a human-readable language into a format (e.g., machine instructions) that can be read and processed by an integrated circuit device. In the example ofFIG.9, the acceleration engine912is a neural network accelerator and the compiler930is for compiling a neural network description into instructions to be executed on the acceleration engine912. When the acceleration engine912implements a different type of accelerator, another compiler can be used.

The compiler930can be activated, for example, when the operating system920receives keyboard, mouse, touchscreen, voice commands, or other inputs from the Input/Output devices908. The inputs can further include parameters for the compiler930, such as the input code942to compile and configure options for the compilation process. Once the compiler930is activated, the processor902can load the instructions for the compiler930into the processor memory904, and can execute the instructions.

In the example ofFIG.9, the compiler930includes a first stage932, a second stage936, and a third stage940, which each perform different operations to produce compiled code944. In other examples, the compiler930can combine the operations of the first stage932, second stage936, and/or third stage940into fewer stages, or can divide the operations of one of the stages into multiple stages.

The first stage932can receive and process input code942. The input code942can describe a program in a high-level programming language, such as Java, C++, or Tensorflow, among many other examples. The input code942can describe, for example, steps to perform image recognition, speech recognition, machine translation, or other operations. The input code942can be obtained, for example, from the storage device906. Alternatively, though not illustrated here, the input code942may be located in the processor memory904or can be obtained from a network location, using the network interface910. Processing of the input code942can include sorting the operations described in the input code942into layers, where the outputs of one layer provide the inputs to a next layer. Processing can also include identifying steps to be performed by the processor902, rather than by the acceleration engine912. For example, the processor902, through the execution of a driver922, may need to perform steps such as configuring DMA descriptors for moving data into or out of the acceleration engine912, among other examples.

The output934of the first stage932can be organized, for example, in the layers, nodes, and connections between nodes of a neural network. The second stage936can perform intermediate processing on this output934. For example, the operations performed in any one layer, or at any one node in a layer, may be too many for the acceleration engine912to perform at the same time. The acceleration engine912may, for example, have a limited amount of local storage space for the data needed for a computation, or the computations may be more than the acceleration engine912can perform at one time. In this example, the first stage932can break the operations of the layer or node down into smaller operations, which can fit into the acceleration engine's local memory and/or can fit into the computing capacity of the acceleration engine912. Processing of the output934of the first stage932can include other steps, such as scheduling, or determining the order in which the acceleration engine912and/or processor902will perform operations, among other examples.

In various examples, the output938of the second stage936includes the various steps to be performed by components of the acceleration engine912, in the order that the steps are to be performed. The output938can be represented, for example, as a data flow graph, where the nodes in the graph represent memory operations, computations, and other operations, and the edges or connections between the nodes represent dependencies between the nodes, such as data dependencies, memory dependencies, or operational dependencies, among other examples.

The third stage940can operate on the output938of the second stage936, and perform various steps before producing the instructions that are to be executed by the acceleration engine912. These steps can include, for example, removing redundant dependencies, resolving or handling dependencies between nodes by inserting synchronization instructions into the code, identifying possible optimizations in memory usage or memory bandwidth usage, and other operations.

The output of the third stage940is compiled code944, which may include machine instructions in binary format. In some examples, the compiled code944can be stored in the processor memory904. Alternatively or additionally, the compiled code944can be copied to the storage device906or to a network location. As noted above, the acceleration engine912may be located at a different host system, in which case the compiled code944can be sent over the network interface910to the other host system.

In the example ofFIG.9, the host system900can execute a driver922, which can also be referred to as a device driver or runtime driver, that manages the acceleration engine912. The driver922can provide an interface between applications executing on the host system900(or on another host system) and the acceleration engine912. For example, the driver922can provide an Application Program Interface (API) that defines functions for feeding input data to the acceleration engine912and defining the operation to perform on the input data. In this and other examples, the driver922can configure the acceleration engine912to perform the operation. For example, the driver922can identify a neural network that the acceleration engine912is to execute, as well as the location in the processor memory904or on the storage device906where the compiled code944for the neural network is located. The driver922can further load into the acceleration engine912or cause the acceleration engine912to load the compiled code944, can load or cause the acceleration engine912to load the input data on which the neural network is to operate, and/or can cause the acceleration engine912to begin executing on the input data. Once the acceleration engine912has finished, the acceleration engine912can notify the driver922, and the driver922can deliver a result back to the application that requested the result.

FIG.10illustrates an example of an accelerator1002of an acceleration engine. In various implementations, the accelerator1002, for a set of input data, can execute a neural network to perform a task the neural network was trained for (e.g., conduct inference), including executing conditional layers. In various implementations, the example accelerator1002is an integrated circuit component of an acceleration engine. The accelerator can have other integrated circuit components, including additional acceleration engines. In various implementations, the accelerator1002can include a memory subsystem1004and a processing engine array1010. When in operation (e.g., when computing a result for a set of input data1050), the processing engine array1010can read weight1006and state1008values from the memory subsystem1004. The processing engine array1010can output computation results to a results buffer1012. In some cases, the example accelerator1002can perform an activation function (using an activation1016block) and/or pooling (using a pooling1018block) on the results from the processing engine array1010, before the results are written to the memory subsystem1004.

Weights1006, in this example, are the weight values for a neural network. In various implementations, the weights1006are post-training weights, meaning that values for the weights1006were previously determined. State1008, in this example, can include input data1050when a computation begins, as well as intermediate values that reflect an in-progress computation. State1008, for example, can include partial sums determined by the processing engine array1010. State1008can also include instructions for the processing engine array1010, where the instructions may be associated with a particular layer. The instructions can, for example, instruct the processing engine array1010, and possibly also the activation1016and/or pooling1018blocks, to execute a certain computation. The weights1006and the state1008can be read from the memory subsystem1004for operating on by the processing engine array1010. In some examples, the memory subsystem can also include a separate memory or buffer for instructions.

In various implementations, the memory subsystem1004can include multiple memory banks1014. In these implementations, each memory bank1014can be independently accessible, meaning that the read of one memory bank is not dependent on the read of another memory bank. Similarly, writing to one memory bank does not affect or limit writing to a different memory bank. In some cases, each memory bank can be read and written at the same time. Various techniques can be used to have independently accessible memory banks1014. For example, each memory bank can have at least one read channel and may have at least one separate write channel that can be used at the same time. In these examples, the memory subsystem1004can permit simultaneous access to the read or write channels of multiple memory banks. As another example, the memory subsystem1004can include arbitration logic such that arbitration between, for example, the outputs of multiple memory banks1014can result in more than one memory bank's output being used. In these and other examples, though globally managed by the memory subsystem1004, each memory bank can be operated independently of any other.

Having the memory banks1014be independently accessible can increase the efficiency of the accelerator1002. For example, weights1006and state1008can be simultaneously read and provided to each row of the processing engine array1010, so that the entire processing engine array1010can be in use in one clock cycle. As another example, weights1006and state1008can be read at the same time that intermediate results are written to the memory subsystem1004. In contrast, a single memory, while still able to provide weights1006and state1008to the processing engine array1010faster than off-chip memory, may be able to service only one read or write at a time. With a single memory, multiple clock cycles can be required, for example, to read weights for each row of the processing engine array1010before the processing engine array1010can be started.

In various implementations, the memory subsystem1004can be configured to simultaneously service multiple clients, including the processing engine array1010, the activation1016block, the pooling1018block, and any external clients that access the memory subsystem1004over a chip interconnect1020. In some implementations, being able to service multiple clients can mean that the memory subsystem1004has at least as many memory banks as there are clients. In some cases, each row of the processing engine array1010can count as a separate read client. In these cases, weights1006and state1008can be stored separately, and thus require two reads, or can be concatenated and stored together, thus requiring one read. In some cases, each column of the processing engine array1010can output an intermediate value, such that each column can count as a separate write client. In some cases, output from the processing engine array1010can be written into the memory banks1014that can then subsequently provide input data for the processing engine array1010. The memory banks1014can be implemented, for example, using static random access memory (SRAM).

In various implementations, the memory subsystem1004can include control logic. The control logic can, for example, keep track of the address spaces of each of the memory banks1014, identify memory banks1014to read from or write to, and/or move data between memory banks1014, if needed. In some implementations, the memory subsystem1004can include multiplexors for selecting which memory bank to output to a particular client and/or to receive input from a particular client. In these implementations, the control logic can generate select signals for the multiplexors, which can enable some or all of the memory banks1014to service each client. In some implementations, memory banks1014can be hardwired to particular clients. For example, a set of memory banks1014can be hardwired to provide weights1006and state1008to the rows of the processing engine array1010. In these examples, the control logic can move data between memory banks1014, for example, to move intermediate results from the memory banks1014to which the intermediate results are written, to the memory banks1014from which the intermediate results will be read for the next round of computation.

The processing engine array1010is the computation matrix of the accelerator1002. The processing engine array1010can, for example, execute parallel integration, convolution, correlation, and/or matrix multiplication, among other things. The processing engine array1010includes multiple processing engines1011, arranged in rows and columns, such that results output by one processing engine1011can be input directly into another processing engine1011. Processing engines1011that are not on the outside edges of the processing engine array1010thus can receive data to operate on from other processing engines1011, rather than from the memory subsystem1004.

In various examples, the processing engine array1010uses systolic execution, in which data arrives at each processing engine1011from different directions at regular intervals. In some examples, input data can flow into the processing engine array1010from the left and weight values can be loaded at the top. In some examples weights and input data can flow from the left and partial sums can flow from top to bottom. In these and other examples, a multiply-and-accumulate operation moves through the processing engine array1010as a diagonal wave front, with data moving to the right and down across the array. Control signals can be input at the left at the same time as weights1006, and can flow across and down along with the computation.

In various implementations, the number of columns in the processing engine array1010determines the computational capacity of the processing engine array1010, and the number of rows determines the required memory bandwidth for achieving maximum utilization of the processing engine array1010. The processing engine array1010can have, for example, 64 columns and 256 rows, or some other number of columns and rows.

An example of a processing engine1011is illustrated inFIG.10in an inset diagram. As illustrated by this example, a processing engine1011can include a multiplier-accumulator circuit. Inputs from the left can include, for example, input data i and a weight value w, where the input data is a value taken from either a set of input data or a set of intermediate results, and the weight value is from a set of weight values that connect one layer of the neural network to the next. A set of input data can be, for example, an image being submitted for identification or object recognition, an audio clip being provided for speech recognition, a string of text for natural language processing or machine translation, or the current state of a game requiring analysis to determine a next move, among other things. In some examples, the input data and the weight value are output to the right, for input to the next processing engine1011.

In the illustrated example, an input from above can include a partial sum, p_in, provided either from another processing engine1011or from a previous round of computation by the processing engine array1010. When starting a computation for a new set of input data, the top row of the processing engine array1010can receive a fixed value for p_in, such as zero. As illustrated by this example, i and w are multiplied together and the result is summed with p_in to produce a new partial sum, p_out, which can be input into another processing engine1011. Various other implementations of the processing engine1011are possible.

Outputs from the last row in the processing engine array1010can be temporarily stored in the results buffer1012. The results can be intermediate results, which can be written to the memory banks1014to be provided to the processing engine array1010for additional computation. Alternatively, the results can be final results, which, once written to the memory banks1014can be read from the memory subsystem1004over the chip interconnect1020, to be output by the system.

In some implementations, the accelerator1002includes an activation1016block. In these implementations, the activation1016block can combine the results from the processing engine array1010into one or more output activations. For example, for a convolutional neural network, convolutions from multiple channels can be summed to produce an output activation for a single channel. In other examples, accumulating results from one or more columns in the processing engine array1010may be needed to produce an output activation for a single node in the neural network. In some examples, activation1016block can be bypassed.

In some implementations, the accelerator1002can include a pooling1018block. Pooling is the combining of outputs of a cluster of nodes from a layer of a neural network. The combined output can be provided to the next layer. Combining can include for example, computing a maximum value, a minimum value, an average value, a median value, or some other value determined from the outputs of the cluster of nodes. In various examples, the pooling1018can be selectively activated, as needed for any particular neural network.

In various implementations, instructions provided to the processing engine array1010can include instructions for executing a conditional layer, including instructions that configure the processing engine array1010to compute a result from the outputs of the most recent layer that was executed, and testing the result against a condition. For example, an instruction included in state1008read from the memory subsystem1004can configure the processing engine array1010to read the outputs of the preceding layer, and to compute a result from these outputs. In this example, the preceding layer is a hidden layer or any other layer other than the output layer. Also in this example, the processing engine array1010can compute the result using a particular logistic function, which may be identified by an instruction. As a further example, the same instruction or different instruction can include a condition against which to compare the result. In this example, the instruction can configure the processing engine array1010to test the result to determine whether the result meets the condition. The condition can, for example, call for testing the result against a test value, where the test is to see whether the result is greater than, less than, equal to, and/or not equal to the test value, or to conduct another type of comparison. In some examples, the processing engine array1010can compute the result and test the condition at the same time. In some examples, the processing engine array1010can compute the result directly from intermediate results being computed by the processing engine array1010. In some examples, the processing engine array1010can compute the result and then store the result in the memory subsystem1004, and then read the result to test the condition. In these and other examples, once the processing engine array1010has tested the result against the condition, the processing engine array1010can write the outcome of the test to the memory subsystem1004, or to a register in the accelerator1002, or to a storage location outside of the accelerator1002.

In some examples, the pooling1018block can be used to assist in executing a conditional layer. For example, when the outputs of the preceding layer are computed by the processing engine array1010, the pooling1018block can be configured to compute a result from the outputs. In this example, the outputs and the result can both be written to the memory subsystem1004. Also in this example, the result output by the pooling1018can be input to the processing engine array1010for the processing engine array1010to test the result against a condition.

In some examples, the activation1016block can be configured to test the result against the condition. For example, the activation1016block can be configured such that, when the result is output from the processing engine array1010, the activation1016block can test the result against the condition. In this example, the activation1016block can write an outcome of the test to the memory subsystem1004or to a register.

In some examples, the pooling1018block can be used when the condition requires manipulating multiple values. For example, the result may include a set of values, such as may be included in a vector computed by softmax. In this example, cascading sub-blocks in the pooling1018block can compare the values in the set against one another to identify the largest two, five, seven, or some other number of values. A final block in the cascade can compute a sum of the largest values, and then compare the sum against a test value. In this example, the result of the comparison determines whether the condition has or has not been met.

In the various examples discussed above, execution of the conditional layer completes with a value written to the memory subsystem1004or to a register, which indicates the outcome of testing the condition. Alternatively or additionally, the accelerator1002can write the value to a storage location outside of the accelerator1002, such as in processor memory. In these and other examples, the accelerator1002can then wait for further instructions. For example, the accelerator1002may wait for input data1050and an instruction to continue processing. In this example, the input data1050can be a set of weights for the next layer that the accelerator1002is to execute. Alternatively or additionally, the input data1050can include an instruction for the accelerator1002to start a new inference, and the input data1050can include the data upon which to operate.

In some examples, in addition to or instead of writing a value indicating the outcome of the condition, the accelerator1002can determine the next action to take. For example, the conditional instruction can include a pointer, memory address, or other identifier for the next layer to execute when the condition is met. In this example, the conditional instruction can also include an identifier for a layer to execute when the condition is not met, or else indicate that the next sequential layer should be executed. In this example, the accelerator1002may be able to begin executing the layer identified by the conditional instruction. For example, the weights for the identified layer may already be present in the memory subsystem1004. Alternatively, the accelerator1002may be able to request that the appropriate weights be loaded into the memory subsystem1004.

In some examples, the accelerator1002may be instructed to stop in-progress computations, and reset to a start state. This may occur, for example, when the condition is met and the accelerator1002is being instructed to not continue with the current inference. To terminate an in-progress computation, the accelerator1002can, for example, flush all values in the processing engine array1010and discard the outputs. As a further example, the accelerator1002can delete values from the memory subsystem1004and/or move values to be ready to start a new inference. In some examples, the accelerator1002can immediately begin a new inference on input data1050that is waiting to be processed.

Input data1050can arrive over the chip interconnect1020. The chip interconnect1020can connect the accelerator1002to other components of an acceleration engine, such as a DMA engine that can obtain input data1050from an I/O device, a storage drive, or a network interface. The input data1050can be, for example one-dimensional data, such as a character string or numerical sequence, or two-dimensional data, such as an array of pixel values for an image or frequency and amplitude values over time for an audio signal. In some examples, the input data1050can be three-dimensional, as may be the case with, for example, the situational information used by a self-driving car. In some implementations, the memory subsystem1004can include a separate buffer for the input data1050. In some implementations, the input data1050can be stored in the memory banks1014along with the weights1006.

In various implementations, the weights1006stored in the memory subsystem1004can have been determined by training the neural network to perform one or more tasks. The input data1050can include an instruction indicating the task to perform (e.g., image processing, speech recognition, machine translation, etc.). In various implementations, the accelerator1002is configured for conducting inference (e.g., performing a task), rather than for training of the neural network. In some implementations, the accelerator1002can be used for training, though perhaps with assistance from software to update the stored weights1006.

In various implementations, the memory subsystem1004can include enough memory to store both intermediate results and all of the weight values for a neural network. The memory subsystem1004should have, at a minimum, enough memory in the memory subsystem1004to store intermediate results, but in many cases the memory subsystem1004can include many more memory banks1014than are needed to store just intermediate results. This additional space can be used to store all of the weight values for a neural network. For example, a neural network may have 1.5 million weights, which, when each is represented by 32 bits, can require about 6 MB of memory. Intermediate results can require, for example, 10 MB of storage space, at most. On-chip memory of 20 MB is a reasonable size, and, in the preceding example, can readily accommodate the weight values, intermediate results, and any other data that the accelerator1002can need during operation.

FIG.11illustrates an example of an acceleration engine1100that has multiple accelerators1102a-1102n. Each of the accelerators1102a-1102ncan include a memory subsystem and processing engine array, and can execute the computation required for a neural network to perform a task for which the neural network was programmed. In the illustrated example, the acceleration engine1100includes n accelerators1102a-1102n.

The example acceleration engine1100further includes DRAM controllers1142a-1142kfor communicating with processor memory, implemented in this example using DRAM1130. In the illustrated example, the acceleration engine1100includes k DRAM controllers1142a-1142k, each of which may be able to communicate with an independent set of banks of DRAM. In other examples, other types of RAM technology can be used for the processor memory. The DRAM controllers1142a-1142kcan also be referred to as memory controllers.

The example acceleration engine1100further includes Input/Output (I/O) controllers1144a-1144pfor communicating with I/O devices1132in the system. The acceleration engine1100can communicate with I/O devices over, for example, a processor bus. In some examples, the processor bus can be implemented using Peripheral Component Interconnect (PCI) and/or a variation of the PCI bus protocol. The processor bus can connect the acceleration engine1100to I/O devices1132such as, for example, input and output devices, memory controllers, storage devices, and/or network interface cards, among other things. In this example, the acceleration engine1100includes p I/O controllers1144a-1144p, each of which may include a separate root complex and may communicate with a separate set of I/O devices1132. In other examples, other standardized bus protocols, such as Ultra Path Interconnect (UPI) can be used for the host bus. In other examples, a proprietary bus protocol can be used.

The example acceleration engine1100further includes DMA engines1146a-1146dthat can move data between the accelerators1102a-1102n, DRAM controllers1142a-1142k, and I/O controllers1144a-1144p. In the illustrated example, the acceleration engine1100includes d DMA engines1146a-1146d. In some implementations, the DMA engines1146a-1146dcan be assigned to specific tasks, such as moving data from the DRAM controllers1142a-1142kto the accelerators1102a-1102n, or moving data between the I/O controllers1144a-1144pand the accelerators1102a-1102n. In some implementations, at least one DMA engine1146a-1146dcan be dedicated to each accelerator1102a-1102n. In some implementations, the DMA engines1146a-1146dcan be treated as a pool instead of being dedicated to a function or component, such that whenever data needs to be moved, an available DMA engine1146a-1146dis engaged.

In the example acceleration engine1100, the various components can communicate over a chip interconnect1120. The chip interconnect1120primarily includes wiring for routing data between the components of the acceleration engine1100. In some cases, the chip interconnect1120can include a minimal amount of logic, such as multiplexors to control the direction of data, flip-flops for handling clock domain crossings, and timing logic.

In some examples, each of the accelerators1102a-1102ncan simultaneously be executing a different neural network. In some examples, two or more of the accelerators1102a-1102ncan execute the same neural network for different inputs. In some examples, two or more of the accelerators1102a-1102ncan be executing parts of the same neural network (e.g., parts of the same layer or different layers). In some examples, two or more of the accelerators1102a-1102ncan sequentially execute layers of a neural network, such that inputs can be pipelined through the acceleration engines.

FIG.12illustrates an example of a method1200for reducing computation in neural network processing. The method1200may be implemented by the systems described above, such as for example the host system ofFIG.9, the accelerator ofFIG.10, and/or the acceleration engine ofFIG.11. One or more steps of method1200may be performed in a different order than that shown in the illustrated example, and one or more steps of method1200may be omitted during performance of method1200.

At step1202, a compiler generates instructions from source code for a neural network. The neural network may include a repeatable set of operations that may be performed up to a number of iterations based on a condition. The number of iterations may be a maximum number of iterations that the repeatable set of operations may be performed if the condition is never satisfied (e.g., the length of the “for” loop). The instructions that are generated by the compiler may include a plurality of blocks, where each block contains the instructions for the repeatable set of operations for a single iteration. Accordingly, the number of blocks generated by the compiler may be equal to the maximum number of iterations.

At step1204, the compiler generates at least one additional instruction for at least one of the plurality of blocks. In some examples, the compiler may generate at least one additional instruction for each of the plurality of blocks. The at least one additional instruction may include an evaluation instruction that, when executed, causes a determination of whether the condition is satisfied. The at least one additional instruction may also include an overwrite instruction that, when executed, triggers an overwrite action when the condition is satisfied. The overwrite action causes the instructions of subsequent blocks to be overwritten with NOP instructions. In some examples, the overwrite action causes a DMA engine to overwrite the instructions of the subsequent blocks with the NOP instructions.

At step1206, the compiler adds the at least one additional instruction to at least one of the plurality of blocks. In some examples, the compiler may add the at least one additional instruction to each of the plurality of blocks. The compiler may add the at least one additional instruction by overwriting existing instructions, by inserting between existing instructions, and/or by modifying existing instructions, etc. The at least one additional instruction may be added at the end of a block, at the beginning of a block, or in the middle of a block. In examples where the at least one additional instruction is added at the end of the block, the compiler may skip the last block when adding the instruction(s). In examples where the at least one additional instruction is added at the beginning of the block, the compiler may skip the first block when adding the instruction(s). After adding the at least one additional instruction, the blocks may be transferred to one or more execution engines for runtime execution.

At step1208, one or more execution engines execute the instructions of a first block of the plurality of blocks. Executing the instructions of the first block may cause the one or more execution engines to perform the repeatable set of operations for a single iteration.

At step1210, the one or more execution engines execute the at least one additional instruction of the first block. In response to executing the evaluation instruction of the first block, the one or more execution engines may determine that the condition is satisfied. In response to executing the overwrite instruction of the first block, the one or more execution engines may trigger the overwrite action, causing the instructions of subsequent blocks of the plurality of blocks to be overwritten with the NOP instructions. In some examples, the overwrite action also causes the at least one additional instruction of the subsequent blocks to be overwritten with the NOP instructions.

At step1212, the one or more execution engines execute the NOP instructions of the one or more subsequent blocks. In some examples, executing the NOP instructions does not cause variables stored in a processor memory to change. In some examples, executing the NOP instructions does not cause computational data values stored in a processor memory to change.