Reducing dynamic power consumption in arrays

Systems and methods are provided to skip multiplication operations with zeros in processing elements of the systolic array to reduce dynamic power consumption. A value of zero can be detected on an input data element entering each row of the array and respective zero indicators may be generated. These respective zero indicators may be passed to all the processing elements in the respective rows. The multiplication operation with the zero value can be skipped in each processing element based on the zero indicators, thus reducing dynamic power consumption.

BACKGROUND

Artificial neural networks are computing systems with an architecture based on biological neural networks. A neural network may be implemented by circuitries and data paths, such as a systolic array, which can comprise an array of processing elements capable of performing concurrent arithmetic operations, for example, floating-point multiplications and additions, etc. Power consumption of the systolic array can become critical when a large number of processing elements are performing arithmetic operations concurrently.

DETAILED DESCRIPTION

A convolutional neural network (CNN) is generally a feed-forward artificial neural network, which may include multiple intermediate layers and an output from one layer may be used as an input to the next layer. Systolic arrays may be used to accelerate the workload in neural networks by reading the data from the memory once, and reusing it in multiple computations. A systolic array may be implemented using a two-dimensional array of processing elements (PEs). The PEs can be divided into layers including, e.g., an input layer, a number of intermediate layers (also known as hidden layers), and an output layer. Each PE of the input layer may receive an element of an input data set, and scale the element with a weight (or a filter) to indicate the element's degree of influence on the output. The PEs in the intermediate layers may combine the scaled elements received from each PE of the input layer to compute a set of intermediate outputs. For example, each PE in the intermediate layers may compute a sum of the element-weight products, and then generate an intermediate output by applying an activation function to the sum. The intermediate outputs from each PE of one intermediate layer may be considered as an activated vote (or no-vote), associated with a weight indicating the vote's influence, to determine the intermediate output of the next intermediate layer. The output layer may generate a sum of the scaled intermediate outputs from the final intermediate layer, and generate a binary output (e.g., “yes” or “no”) based on whether the sum of the scaled intermediate outputs exceeds a threshold.

Generally, an input data set (e.g., an input feature map) may be fed, one input data element at a time, into its respective row of the systolic array, and passed from one PE to another PE in a given row starting from a leftmost PE. In some implementations, the weights may be cached in the respective PEs. As the input data element passes through a PE, the input data element can be multiplied with the cached weight value, and accumulated with a partial sum provided by a neighboring PE in a row above. In most implementations, the systolic array may include a large number of PEs (e.g., several thousands), therefore dynamic power consumption can become critical due to the multiply-accumulate operations performed by the large numbers of PEs in the systolic array concurrently.

A common CNN may generally have several weights as well as input data elements with a value of zero. Additionally, a number of data elements generated by the intermediate layers may include a zero value due to the commonly used activation functions such as ReLu or Sigmoid. In these cases, the multiplication operation may generate a zero result due to multiplication with a zero value. The zero result accumulated with a partial sum may not alter the functional result, however, the multiply-accumulate operation may waste dynamic power since the multiplication operation is still being performed.

Embodiments of the disclosed technologies can provide systems and methods to reduce dynamic power consumption in a systolic array by skipping multiplication operations in a PE under certain conditions. The multiplication operation can be skipped when a zero is detected on an input data element for a current operation or a no-operation (NOP) is received by a PE. For example, each PE may receive an opcode indicating an operation to be executed by the PE. The PE may decode the opcode to determine if the opcode value corresponds to a NOP. According to certain embodiments, a value of zero on an input data element may be detected by respective zero input data detector circuits for each row of the systolic array as the input data element enters the systolic array. A value of zero may correspond to a logical zero or a logical low, and a value of one may correspond to a logical one or a logical high. For example, in some implementations, the logical zero may be represented by a first range of voltage levels (e.g., 0-2 volts), and the logical one may be represented by a second range of voltage levels (e.g., 3-5 volts). The respective zero input data detector circuits may generate a zero input data indicator signal for each row, which may be sequentially passed to all the PEs in that row. A respective weight value may be pre-loaded in each PE of the systolic array. For each row of the systolic array, a respective zero weight detector circuit may be used to detect a value of zero on each weight value entering the respective row and generate a respective zero weight indicator signal for each weight value. The respective zero weight indicator signals may be sequentially passed to subsequent PEs in each row along with a corresponding weight value and may be cached in a respective PE for each column. Thus, instead of having respective zero detector circuits in each PE, embodiments of the disclosed technologies can reduce the gate count and dynamic power consumption by having the zero detector circuits that are external to the PEs, and can be used by all the PEs in a given row.

The zero input data indicator and an opcode indicating a NOP may be used to gate a register in each PE, which stores the input data element. Thus when a zero input data indicator is asserted indicating that an input data element associated with the zero input data indicator is “0”, or the opcode indicates a NOP, the register may hold the previous value of the input data element. Stored value of the input data element may be sequentially passed to other PEs in that row along with the respective zero input data indicator signal. However, a zero value of the input data element may not be stored in the register, and therefore not passed to other PEs in that row. The respective stored value of the input data element may be used by a multiplier in each PE to perform the multiplication operation with a weight value cached in the respective PE. When the zero input data indicator corresponding to an input data element is asserted or the opcode indicates a NOP, the stored value of the input data element may not toggle. Since the weight may already be pre-loaded in the PE, the inputs to the multiplier may not change, and the multiplication operation may be skipped, thus reducing the dynamic power consumption by the PE for that operation. In such cases, the multiply-accumulate operation may not provide a correct result, therefore, the result of the multiply-accumulate operation may be bypassed, and an input partial sum from a neighboring PE in a row above may be provided as an output partial sum to another neighboring PE in a row below. An example systolic array is explained with reference toFIG. 1.

FIG. 1illustrates an example 4×4 systolic array100. For example, the systolic array100may include four PEs in each row, and four PEs in each column. It will be understood that the systolic array100may include any number of PEs in each row and column. The systolic array100may be part of a neural network processor in a computer system. For example, the computer system may be configured to provide multi-tenant compute services for data processing applications such as an image recognition service, text-based data processing (e.g., processing of search queries), audio or video data processing, etc.

Each PE may include a row input bus102, a column input bus104, a column output bus106, and a row output bus108. A PE may receive inputs from a left PE of the same row (or from external circuitries) via the row input bus102. The PE may also receive inputs from a PE of the same column above (or from external circuitries) via the column input bus104. The PE may perform arithmetic computations based on the inputs, and transmit the result of the arithmetic computations to a PE of the same column below (or to the external circuitries) via the column output bus106. The PE may also forward the inputs received via the row input bus102to a right PE of the same row via the row output bus108.

The systolic array100may be configured to perform the arithmetic computations, including multiplication and addition operations, for the processing elements of a neural network. For example, each PE may include arithmetic units such as a multiplier and an adder, or a fused multiplier adder. In the example ofFIG. 1, each row of the PEs may be configured to handle one set of input data, and each column of the PEs may generate one set of output data based on the sets of input data received by each PE in a given column.

In one implementation, a column112of the PEs (the leftmost column) may receive four sets of input data, with each set of input data being handled by one row of the PEs. Each PE in the column112may obtain, from the corresponding input data set received via the row input bus102, an input data element and an associated weight value, and multiply the input data element with the weight value to generate a scaled input. The scaled inputs generated by the PEs within any column (including the column112) can be accumulated by the adder of each PE. For example, a PE112a(of the column112) may generate a first scaled input (from the first input data set), and transmit the first scaled input to a PE112bvia the column output bus106as a partial sum. The PE112bmay also generate a second scaled input (from the second input data set) and add the second scaled input to the partial sum. The updated partial sum, accumulated with the first scaled input and the second scaled input, is then transmitted to a PE112cvia the column output bus106. The partial sums are updated and propagated across the column112, and a PE112dmay generate a sum of the scaled inputs from the four input data sets.

The sum generated by the PE112dmay correspond to an output data set, and may be fed back to the leftmost PEs after going through an activation function. Moreover, each PE in the column112can also propagate the input data sets to other PE columns (e.g., a column114), which can scale the input data sets with a different set of weights from the column112. Each column of the PEs can perform the arithmetic operations (multiplications and summations) to generate the output data elements for other processing elements in parallel. In the example ofFIG. 1, the systolic array100can generate output data elements for four PEs corresponding to the four columns of the systolic array100.

The systolic array100may perform convolution computations in multiple waves. A wave may be defined as streaming of input data elements while reusing the same weights in the systolic array100. For example, the respective weights may have been pre-loaded in each PE in the systolic array100, sequentially or in parallel prior to starting a wave computation. The partial sums generated by the PEs may correspond to a single wave. As the PEs of the systolic array100perform arithmetic operations for the convolution computations, dynamic power dissipated by all the multipliers in the PEs may be significant. This problem may be further exacerbated for a systolic array comprising a large number of PEs (e.g., several thousands). The arithmetic operations performed by a PE are further explained with reference toFIG. 2.

FIG. 2illustrates a PE200for neural network computations, according to certain embodiments of the disclosed technologies. The PE200may be part of a systolic array similar to the systolic array100inFIG. 1. Some embodiments may be described with reference to neural networks, however it will be understood that certain embodiments may be used in other applications, e.g., pattern recognition, image processing, audio processing, video processing, etc., without deviating from the scope of the technologies.

The PE200may include a data element load generator202, a data register204, a weight register206, a multiplier208, an adder210, a skip calculation generator212, a skip calculation register214, a selector216, an input partial sum register218, a cached weight register220, and an operation decoder256. The PE200may be configured to receive an input data element222, a weight224, a zero data element indicator226, a zero weight indicator228, an opcode230, a weight load signal232, and an input partial sum234to perform the convolution computations according to some embodiments.

The PE200may be configured to receive the input data element222via a first port. The input data element222may correspond to an input data set, or any array of input data elements. The PE200may receive one input data element at a time, in uniform time periods, from the input dataset. For example, a uniform time period may correspond to a clock cycle. The input data set may be similar to an input feature map comprising input feature map elements. As an example, the input data set may correspond to an input image, an audio clip, a video clip, a text portion, or any other data which may be provided for data processing to identify a certain pattern or an object. In some instances, the input data set may correspond to an intermediate output dataset, which has gone through an activation function, e.g., ReLu or Sigmoid, as discussed with reference toFIG. 1. Each input data element222may include 8-bits, 16-bits, or any suitable number of bits.

The PE200may be configured to receive the weight224via a second port. In some implementations, the weight224may belong to a set of weight values corresponding to a convolution filter. The weight224may be pre-loaded in the PE200prior to receiving the input data element222. In some embodiments, the PE200may receive one weight value at a time, in the uniform time periods, from the set of weight values, to pre-load each PE in a given row with a respective weight value. The PE may pass the weight value to the next PE in the respective row until each PE in the given row has been pre-loaded. Each PE may cache the respective weight value to use for computations with the input data elements. The weight values in the convolution filter may have been pre-determined based on supervised learning, unsupervised learning, or any other method suitable for determining convolutional filters. For example, given an input image, the weight values in the convolution filter can represent a spatial distribution of pixels for certain features to be detected from the input image. The weight224may include 8-bits, 16-bits, or any suitable number of bits.

The PE200may be configured to receive the zero data element indicator226for a current operation via a third port. The zero data element indicator226may include a single bit or multiple bits. The zero data element indicator226may be used to indicate whether the input data element222associated with the zero data element indicator226is zero. For example, a value of “1” for the zero data element indicator226may indicate that the input data element222associated with the zero data element indicator226is zero, and a value of “0” for the zero data element indicator226may indicate that the input data element222associated with the zero data element indicator226is not zero. A “0” may correspond to a logical zero or a logical low, and a “1” may correspond to a logical one or a logical high. For example, in some implementations, the logical zero may be represented by a first range of voltage levels (e.g., 0-2 volts), and the logical one may be represented by a second range of voltage levels (e.g., 3-5 volts). It will be understood that other implementations to represent a “0” value and a ‘1” value are possible without deviating from the scope of the disclosed technologies. The zero data element indicator226may be generated by a circuit external to the PE200, and passed to all the PEs in the same row sequentially, in the uniform time periods.

The PE200may be configured to receive the zero weight indicator228via a fourth port. The zero weight indicator228may include a single bit or multiple bits. The zero weight indicator228may be used to indicate whether the weight224associated with the zero weight indicator228is zero. For example, a value of “1” for the zero weight indicator228may indicate that the weight224is zero, and a value of “0” for the zero weight indicator228may indicate that the weight224is not zero. The zero weight indicator228may be generated by a circuit external to the PE200, and passed to all the PEs in the same row sequentially along with the weight224.

The weight load signal232may be used to load the weight224into the cached weight register220to provide a cached weight246. The weight load signal232may be asserted to cache the weight224for the PE200in the cached weight register220before the input data element222is fed into the array. As the weights are shifted into the array to pre-load each PE with a respective weight value, the weight load signal232may be asserted for each PE at certain time periods in order to pre-load each PE with the appropriate weight value.

The operation decoder256may be configured to decode the opcode230to determine an operation to be executed by the PE200for different instructions represented by different opcode values. In some embodiments, a first opcode value may correspond to an instruction to shift the weights from one PE to another in the systolic array. A second opcode value may correspond to an instruction to start the arithmetic computations by the PE. For example, once the weights have been pre-loaded in the systolic arrays, the input data elements may be read from the memory and the arithmetic computations may be performed as the input data elements pass through the array. A third opcode value may correspond to an instruction to execute NOPs. The NOPS may be used to space two systolic array instructions, or when there are no input data elements to be read from the memory. For example, the NOPs may be used to space the instructions to shift the weights, and the instructions to start the arithmetic computations. For example, for a 4×4 array, it may take up to 15 cycles to shift the weights into all the PEs in the array before starting the arithmetic computations so 15 NOP cycles may be needed. The operation decoder256may be configured to decode the opcode230to generate a NOP258, and a start computations signal260. The opcode230may include any suitable number of bits, e.g., two, four, etc.

In some implementations, the input data element222, the weight224, the opcode230, the zero data element indicator226, and the zero weight indicator228may belong to the row input bus102, as discussed with reference toFIG. 1. The row input bus102may also include one or more control signals, e.g., a data type. In some implementations, a splitter (not shown) may be used in the PE200to split the row input bus102into different internal buses to carry the input data element222, the weight224, the opcode230, the zero data element indicator226, and the zero weight indicator228within the PE200.

The data element load generator202may be configured to generate a data load signal242that may be used to allow the data register204to skip storing of the input data element222in certain conditions. In some embodiments, the input data element222may be loaded into the data register204when the data load signal242is asserted based on the zero data element indicator226and the NOP258. The data load signal242may be asserted when the zero data element indicator226corresponding to the input data element222is “0” and the opcode230does not indicate a NOP (e.g., the NOP258is “0”). The data load signal242may not be asserted when the zero data element indicator226corresponding to the input data element222or the NOP258is “1.” The data element load generator202may be implemented using an OR, NOR, NAND, or any suitable circuit.

The data register204may be configured to store the input data element222, or skip storing of the input data element222to provide a stored input data element244based on the data load signal242for a current operation. In some implementations, the data register204may store a Din input if a load input is “1”, and may hold the previous value if the load input is “0.” For example, if the data load signal242is “1”, the data register204may store a new value for the input data element222, and if the data load signal242is “0”, the data register204may skip storing the new value for the input data element222. Thus, in some instances, the data register204may only store non-zero value of the input data element222. According to certain embodiments, skipping the storing of the new value by the data register204may result in not toggling the stored input data element244and holding the previous value of the stored input data element244.

The weight register206may be configured to store the cached weight246to provide a stored weight value248based on the start computations signal260. In some implementations, the weight register206may store a Din input if a load input is “1”, and may hold the previous value if the load input is “0.” For example, if the start computations signal260is asserted (e.g., the start computations signal260is “1”), the cached weight246may be loaded into the weight register206, else the weight register206may hold the previous value. Thus, the weight224previously loaded into the cache register220using the weight load signal232may be shifted into the weight register206at the start of the arithmetic computations. In some embodiments, the stored weight value248, once loaded at the start of the arithmetic computations, remains unchanged as the input data element is fed into the PE200, one element at a time, for computations corresponding to one or more waves through the systolic array.

The multiplier208may be configured to perform a multiplication operation between the stored input data element244and the stored weight value248to provide a multiplication result250. The multiplier208may be implemented using a multiplier circuit. Generally, when there is a change in the value of any of the inputs of the multiplier208(e.g., a “1” to “0”, or vice-versa), the multiplier208performs the multiplication operation, and the output of the multiplier208changes resulting in dynamic power dissipation. For a systolic array comprising hundreds or thousands of PEs similar to the PE200, the power consumption can be substantial. According to certain embodiments, power consumption of the PE200can be reduced by avoiding the toggling of all the inputs to the multiplier208under certain conditions so that the multiplication operation can be skipped altogether. In some implementations, when the zero data element indicator226or the NOP258is asserted, storing of the input data element222in the data register204can be skipped using the data load signal242, thus keeping the stored input data element244input going into the multiplier208unchanged. For example, the zero data element indicator226or the NOP258may generate a value of “0” for the data load signal242, which can disable loading of the input data element222into the data register204. Since the weight224has been pre-loaded into the PE200, input to the weight register206may not change even if the cached weight246is zero. Therefore, the stored weight value248may not change as the input data element222is received by the PE200for the current operation. In this case, the stored input data element244and the stored weight value248may hold their values from the previous operation and may not toggle. Thus, the multiplication result250may not change and the dynamic power consumption can be reduced. Since the multiplication result250may not be accurate for the current operation, the multiplication result250is not propagated to other PEs in the array.

The PE200may be configured to receive the input partial sum234via a fifth port. The input partial sum234may be a partial sum generated from a neighboring PE in a row above and in the same column of the systolic array. In some instances, the input partial sum234may include inputs from external circuitries. For example, when the PE200is a PE in a first row of the systolic array, the input partial sum234may include default values. As discussed with reference toFIG. 1, the input partial sum234may be part of the column input bus104. In some instances, the column input bus104may also be used to load weights in the PE200. The column input bus104may also include control signals, e.g., an overflow bit. In some implementations, the opcode230may be provided to the PE200via the column input bus104. The input partial sum234may be stored in an input partial sum register218to provide a stored input partial sum236.

The adder210may be configured to perform an addition operation on the multiplication result250and the stored input partial sum236to provide an addition result238. The adder210may be implemented using an adder circuit. In some embodiments, the multiplication and addition operations may be fused or integrated together to perform a single step multiply add operation with a single rounding using a fused multiplier adder, or fused multiplier accumulator instead of performing multiplication and addition operations in different steps. The fused multiplier adders may be used to improve the speed and accuracy of the floating point arithmetic operations. For example, in place of the multiplier208and the adder210, a fused multiplier adder (FMA) may be used to perform both the multiplication and addition operations in a single step.

The skip calculation generator212may be configured to generate a skip calculation indicator252which may be used to bypass the multiplication result250under certain conditions. For example, when a zero is detected on the input data element222or the weight224, or the opcode230indicates a NOP, the multiplier result250may be inaccurate for the current operation and may need to be bypassed based on the NOP258, the zero data element indicator226and the zero weight indicator228. In some embodiments, the skip calculation generator212may assert the skip calculation indicator252to “1”, when the NOP258, the zero data element indicator226or the zero weight indicator228is “1.” The skip calculation generator212may use OR, NOR, NAND or other suitable circuits to generate the skip calculation indicator252. The skip calculation indicator252may be stored in a skip calculation register214to provide a stored skip calculation indicator254that may be used by the selector216.

The selector216may be configured to select either the addition result238or the stored input partial sum236based on the stored skip calculation indicator254to provide an output partial sum240via a sixth port. According to some embodiments, when a value of either the input data element222or the weight224for a current operation is zero, or the NOP258is asserted, the addition result238may not provide a correct result for the current operation since the multiplication result250may hold a value for the previous operation. In such cases, the stored skip calculation indicator254may allow bypassing the addition result238, and selecting the stored input partial sum236to provide the output partial sum240. For example, when the stored skip calculation indicator254is “1”, the stored input partial sum236may be selected as the output partial sum240, and when the stored skip calculation indicator254is “0”, the addition result238may be selected as the output partial sum240. The selector216may be implemented using a multiplexer, or any suitable circuit.

According to certain embodiments, when the input data element222, or the weight224is zero, selecting the input partial sum236as the output partial sum240based on the stored skip calculation indicator254can provide the same functionality as adding a zero multiplication result250to the stored input partial sum236by the adder210. Thus, bypassing the output of the adder210may not result in any change in the functionality of the PE200to perform convolution computations. Generation of the zero data element indicator226and the zero weight indicator228for each row of an array is discussed with reference toFIG. 3.

FIG. 3illustrates an apparatus300including zero detector circuits for input data elements and weights entering a systolic array for neural network computations, according to certain embodiments of the disclosed technologies.

The apparatus300may include a two-dimensional systolic array302comprising PEs arranged into rows and columns. The systolic array302may be similar to the systolic array100inFIG. 1. A first row of the systolic array302may include PE 00, PE 01, PE 02, . . . , PE 0y, a second row of the systolic array302may include PE 10, PE 11, PE 12, . . . , PE 1y, a third row of the systolic array302may include PE 20, PE 21, PE 22, . . . , PE 2y, and an Xth row of the systolic array302may include PE x0, PE x1, PE x2, . . . , PE xy. The x and y may include positive integers, e.g., 32, 64, 128, or any suitable number. Each PE of the systolic array302may be similar to the PE200, and include means to perform arithmetic computations using power efficient methods, as discussed with reference toFIG. 2.

In certain embodiments, a first (e.g., leftmost) PE in each row of the systolic array302may be coupled to a respective zero input data detector circuit to detect a zero value on an input data element, and a respective zero weight detector circuit to detect a zero value on a weight value entering the systolic array302. For example, the PE 00 in the first row may be coupled to a first zero input data detector306aand a first zero weight detector308a, the PE 10 in the second row may be coupled to a second zero input data detector306band a second zero weight detector308b, the PE 20 in the third row may be coupled to a third zero input data detector306cand a third zero weight detector308c, and the PE x0 in the Xth row may be coupled to an Xth zero input data detector306xand an Xth zero weight detector308x. The first zero input data detector306a, the second zero input data detector306b, the third zero input data detector306c, . . . , and the Xth zero input data detector306xmay be configured to detect a zero value on a respective input data element in an input dataset0, an input dataset1, an input dataset2, . . . , and an input datasetx respectively. Similarly, the first zero weight detector308a, the second zero weight detector308b, the third zero weight detector308c, . . . , and the Xth zero weight detector308xmay be configured to detect a zero value on a respective weight value in a filter0, a filter1, a filter2, . . . , and a filterx respectively.

Each of the input dataset0, the input dataset1, the input dataset2, . . . , and the input datasetx may belong to an image, a text, a video clip, an audio clip, or another type of data set which may need to be processed by a neural network processor for convolution computations. In some instances, the input dataset0, the input dataset1, the input dataset2, . . . , and the input datasetx may be associated with output dataset0, output dataset1, output dataset2, . . . , output datasety generated by an intermediate layer of the convolution operation. For example, the output dataset0, output dataset1, output dataset2, . . . , output datasety may go through activation functions and fed back to the systolic array302as the input dataset0, the input dataset1, the input dataset2, . . . , and the input datasetx. The filter0, the filter1, the filter2, . . . , and the filterx may include different sets of weight values to convolve with the input dataset0, the input dataset1, the input dataset2, . . . , and the input datasetx. The weight values in the filter0, the filter1, the filter2, . . . , and the filterx may be pre-determined using supervised learning, non-supervised learning, or any suitable method of determining convolution filters.

Each zero input data detector for the respective row may be configured to detect whether an input data element from the input dataset entering the respective row is “0” and generate a corresponding zero input data indicator for that input data element. The corresponding zero data element indicator may be passed into the first PE of the respective row along with the input data element. For example, the PE 00 may be the first PE of the first row in the systolic array302. The PE 00 may be configured to receive input data elements from the input dataset0 prior to other PEs in the first row (e.g., PE 01, PE 02, . . . , PE 0y). In some embodiments, one input data element at a time may be fed sequentially, in uniform time periods, from the input dataset0 to the PE 00. The first zero input data detector306amay be configured to generate the corresponding zero data element indicator226in each of the uniform time periods (e.g. clock cycles) for each input data element from the input dataset0. The zero data element indicator226corresponding to each input data element may be fed to the PE 00 sequentially, in uniform time periods, along with each input data element. The PE 00 may store or skip storing the received input data element222based on the value of the respective data load signal242. In some implementations, the first zero input data detector306amay include a comparator to compare the incoming input data element with a zero to assert (e.g., set to “1”) or de-assert (e.g., set to “0”) the zero data element indicator226based on the value of the incoming input data element. For example, the comparator may be implemented using an OR, XOR, NAND, or any suitable circuit.

Each zero weight detector for the respective row may be configured to detect whether a weight value from a set of weight values entering the respective row is zero and generate a corresponding zero weight indicator for that weight value. For example, the first zero weight detector308amay be configured to detect whether a weight value from the filter0 (e.g., the weight224) includes a zero value and generate the corresponding zero weight indicator228for the weight. In some implementations, the first zero weight detector308amay include a comparator to compare the weight value with a zero to assert (e.g., set to “1”) or de-assert (e.g., set to “0”) the zero weight indicator228. For example, the comparator may be implemented using an OR, XOR, NAND, or any suitable circuit. In one embodiment, one weight value at a time may be fed sequentially, in uniform time periods, from the filter0 to the PE 00 for pre-loading the respective weight values in the PE 00 to the PE 0y prior to starting the arithmetic computations. The first zero weight detector308amay generate a corresponding zero weight indicator for each of those weight values which may be fed to the PE 00 sequentially, in uniform time periods, along with the corresponding weight value. The PE 00 may pass the respective weight values and the corresponding zero weight indicators sequentially to the next neighboring PE until all the PEs in the first row have been preloaded with the respective weight values and the corresponding zero weight indicators. The respective weight value and the corresponding zero weight indicator may be cached in each PE before the respective input data elements are fed to each row in the systolic array302.

The second zero input data detector306b, the third zero input data detector306c, . . . and the Xth zero input data detector306xmay be similar to the first zero input data detector306a, and may generate a respective zero data element indicator, similar to the zero data element indicator226, to provide to the PE 10, PE 20, . . . , and PE x0, sequentially, in the uniform time periods, for power optimization. The respective zero data element indicator generated for each row may be received by a respective first PE in each row via the respective row input bus102, and propagated, sequentially, in the uniform time periods, by the first PE to all the PEs in the given row. The second zero weight detector308b, the third zero weight detector308c, . . . , and the Xth zero weight detector308xmay be similar to the first zero weight detector308a, and may generate a respective zero weight indicator, similar to the zero weight indicator228, to provide to the PE 10, PE 20, . . . , and PE x0, sequentially, to pre-load each PE in the respective row along with the respective weight value prior to starting the arithmetic computations.

In some embodiments, the zero input data detectors306a-306x, and the zero weight detectors308a-308xmay be implemented as a separate entity external to the systolic array302. For example, the zero input data detectors306a-306x, and the zero weight detectors308a-308xmay be part of a circuit304. In other embodiments, the circuit304and the systolic array302may be part of a computing engine, which may be configured to perform arithmetic computations for the convolution operations. Some embodiments of the disclosed technologies can provide reduced gate count and dynamic power consumption by detecting zeros on the input data elements and the weights entering a respective first PE in each row of the systolic array, and passing the zero indicators to all the PEs in the array as compared to using respective zero detectors within each PE in the systolic array302.

Note thatFIG. 3only shows the respective zero data element indicator and the zero weight indicator entering the first PE in each row of the systolic array302for ease of illustration, however it will be understood that each PE in the respective row of the systolic array302may also receive the respective input data element and the respective weight value along with some control signals (e.g., opcode230, weight load232, data type, etc.), which may be propagated from the left to the right of the systolic array302for each row. This is further explained with reference toFIG. 4.

FIG. 4illustrates an apparatus400showing propagation of zero indicators, the input data element and the weight value from one PE to another, according to a first embodiment of the disclosed technologies.

In some embodiments, each PE of the systolic array302may include components, in addition to the components of the PE200as shown inFIG. 2, to cache the weight, and the zero indicators before passing them to a neighboring PE in a given row. For example, each PE may include a zero data element indicator register402, a zero weight indicator register404, and an output weight register406, in addition to the components of the PE200. This is further explained inFIG. 4using the PE 00 and the PE 01 as an example. The PE 00 and the PE 01 are part of the systolic array302as discussed with reference toFIG. 3.

The zero data element indicator register402may be configured to store the zero data element indicator226received by the PE 00 to provide a stored zero data element indicator408. The zero data element indicator226may correspond to the input data element222received by the PE 00. As discussed with reference toFIG. 3, one input data element at a time may be fed sequentially, in uniform time periods, from the input dataset0 to the PE 00. The data register204may store or skip storing the input data element222for a current operation based on the data load signal242. For example, if the zero data element indicator226is “1” or the NOP258is received by the PE 00 for the current operation, the data register204may hold the value from the previous operation or a default value. The stored input data element244may be provided to the PE 01 as the input data element222.

Thus, in certain embodiments, if a zero is detected on the input data element222received by the PE 00 for the current operation, the zero value of the input data element222may not be propagated to the PE 01-PE 0y since the stored input data element244may hold the value from the previous operation or the default value. However, the stored zero data element indicator408corresponding to the zero value of the input data element222may be propagated to the neighboring PEs. The PE 01 may receive the stored zero data element indicator408as the zero data element indicator226, store it, and propagate its stored zero data element indicator408to the neighboring PE (e.g., PE 02). The PE 01 may also propagate its stored input data element244from the previous operation to PE 02 along with the stored zero data element indicator408. Thus, the zero data element indicator226may only be generated once by the first zero input data detector306a, and passed sequentially, in uniform time periods, from the PE 00 to the PE 0y. The respective stored zero data element indicator408in each PE may be used to bypass the respective multiplier result250in each PE if the corresponding input data element222includes a zero value. Thus, the respective output partial sum240in each PE may be the respective input partial sum234if the respective input data element222or the respective weight224includes a zero value or a NOP is received for that operation.

The zero weight indicator register404may be configured to store the zero weight indicator228received by the PE 00 to provide a stored zero weight indicator412. The zero weight indicator228may correspond to the weight224received by the PE 00. The PE 00 may be configured to receive the weight224for pre-loading the weights in the systolic array302prior to starting the arithmetic computations. For example, in one embodiment, one weight value at a time may be fed sequentially, in uniform time periods, from the filter0 to the PE 00. The PE 00 may store the received weight value in the output weight register406to provide a stored weight value410based on a shift weight signal414. The stored weight value410may be shifted into the PE 01 as the weight224. The shift weight signal414may be generated by the operation decoder256based on the opcode230. For example, the opcode230may include a certain opcode value to indicate shifting of the weight value from one PE to another PE. The PE 01 may receive the stored zero weight indicator412as the zero weight indicator228in the next time period, store it, and propagate its stored zero weight indicator412to the neighboring PE (e.g., PE 02). Thus, the zero weight indicator228may only be generated once by the first zero weight detector308a, and passed sequentially, in uniform time periods, from the PE 00 to the PE 0y along with the corresponding weight value.

In certain embodiments, the same weight value may be used by all the PEs in a given row for convolving with each input data element for an input data set to optimize the memory bandwidth. In some embodiments, instead of pre-loading the weights in the systolic array, respective weights may be fed into each row along with the input data elements to perform arithmetic computations. This is further explained with reference toFIG. 5.

FIG. 5illustrates an apparatus500showing propagation of zero detectors, the input data element and the weight value from one PE to another, according to a second embodiment of the disclosed technologies.

In the second embodiment, instead of pre-loading the weights in the systolic array, one weight value at a time may be fed sequentially, in uniform time periods, from the filter0 to the PE 00, along with the input data element222. The input data element222and the weight224may be cached in their respective registers only if no zero is detected on both the input data element222and the weight224. Thus, the multiplier inputs may not toggle if a zero is detected on either the input data element222or the weight224resulting in reduced power consumption. The input data element222and the weight224may be propagated to the neighboring PEs along with the corresponding zero data element indicator226and the zero weight indicator228.

In the second embodiment, a skip calculation generator502may be configured to generate a skip calculation indicator526using the zero data element indicator226and the zero weight indicator228. The skip calculation indicator526may be used by a data register504and a weight register506to skip storing of a zero value on the input data element222or the weight224respectively for a current operation. In some embodiments, the skip calculation generator502may perform an OR, or an NOR operation on the zero data element indicator226and the zero weight indicator228to generate the skip calculation indicator526. The skip calculation indicator526may be stored in a skip calculation register514to provide a stored skip calculation indicator540that may be used by a selector516.

The data register504may be configured to store the input data element222, or skip storing of the input data element222to provide a stored input data element528based on the skip calculation indicator526for a current operation. For example, if the skip calculation indicator526is “0”, the data register504may store a new value for the input data element222, and if the skip calculation indicator526is “1”, the data register504may skip storing the new value for the input data element222. According to certain embodiments, skipping the storing of the new value by the data register504may result in not toggling the stored input data element528and holding the previous value of the stored input data element528.

The weight register506may be configured to store the weight224, or skip storing of the weight224to provide a stored weight value530based on the skip calculation indicator526for the current operation. For example, if the skip calculation indicator526is “0”, the weight register506may store a new value for the weight224, and if the skip calculation indicator526is “1”, the weight register506may skip storing the new value for the weight224. According to certain embodiments, skipping the storing of the new value by the weight register506may result in not toggling the stored weight value530and holding the previous value of the stored weight value530.

The multiplier508may be configured to perform a multiplication operation between the stored input data element528and the stored weight value530to provide a multiplication result532. In some implementations, when a value of either the input data element222, or the weight224for a current operation is zero, storing of both the input data element222and the weight224in the data register504and the weight register506respectively can be skipped using the skip calculation indicator526. For example, the zero data element indicator226or the zero weight indicator228may generate a value of “1” for the skip calculation indicator526, which can disable loading of the respective inputs into the data register504and the weight register506. In this case, the stored input data element528and the stored weight value530may hold their values from the previous operation and may not toggle. Thus, the multiplication result532may not change and the dynamic power consumption can be reduced. Since the multiplication result532may not be accurate for the current operation, the multiplication result532is not propagated to other PEs in the array.

The PE 00 may receive an input partial sum534, which may be stored in an input partial sum register512to provide a stored input partial sum536. The adder510may be configured to perform an addition operation on the multiplication result532and the stored input partial sum536to provide an addition result538. In some embodiments, in place of the multiplier508and the adder510, an FMA may be used to perform both the multiplication and addition operations in a single step.

The selector516may be configured to select either the addition result538or the stored input partial sum536based on the stored skip calculation indicator540to provide an output partial sum542via a sixth port. According to some embodiments, when a value of either the input data element222, or the weight224for a current operation is zero, the addition result538may not provide a correct result for the current operation since the multiplication result532may hold a value for the previous operation. In such cases, the stored skip calculation indicator540may allow bypassing the addition result538, and selecting the stored input partial sum536to provide the output partial sum542. For example, when the stored skip calculation indicator540is “1”, the stored input partial sum536may be selected as the output partial sum542, and when the stored skip calculation indicator540is “0”, the addition result538may be selected as the output partial sum542. The selector516may be implemented using a multiplexer, or any suitable circuit.

In some embodiments, generation of the skip calculation indicator526may also be based on a value of an operation to be executed by the PE 00 as determined by the opcode230(not shown inFIG. 5). For example, for a NOP, the data register504and the weight register506can hold their values from a previous operation using the skip calculation indicator526, thus reducing power consumption. The selector516may select the stored input partial sum536as the output partial sum542instead of the addition result538.

The zero data element indicator register518may be configured to store the received zero data element indicator226to provide a stored zero data element indicator544to the neighboring PE 01 in the first row in the next time period.

The zero weight indicator register524may be configured to store the received zero weight indicator228to provide a stored zero weight indicator550to the neighboring PE 01 in the first row in the next time period.

The output data register520may be configured to store the received input data element222to provide a delayed input data element546to the neighboring PE 01 in the first row in the next time period.

The output weight register522may be configured to store the received weight224to provide a delayed weight value548to the neighboring PE 01 in the first row in the next time period.

The stored zero data element indicator544, the stored zero weight indicator550, the delayed input data element546, and the delayed weight value548may be provided to the PE 01 via the row output bus108as discussed with reference toFIG. 1. In some embodiments, the control signals received by the PE 00 from external circuitries may also be cached in the PE 00 and provided to the PE 01 via the row output bus108. Thus, the respective input data element, weight value, zero data element indicator, and the zero weight indicator received by each PE every clock cycle may be cached in the respective PE, and the cached (delayed) values may be passed to the next neighboring PE in the next time period.

The stored zero data element indicator544, the stored zero weight indicator550, the delayed input data element546, and the delayed weight value548may be received by the PE 01 as the zero data element indicator, the zero weight indicator, the input data element, and the weight respectively via the row input bus102. The PE 01 may perform the arithmetic computations on the delayed input data element546, and the delayed weight value548according to certain embodiments. The PE 01 may skip the multiplication operation if the delayed input data element546, or the delayed weight value548includes a zero value based on the stored zero data element indicator544and the stored zero weight indicator550, thus optimizing the dynamic power consumption of the PE 01. The PE 01 may store the stored zero data element indicator544, the stored zero weight indicator550, the delayed input data element546, and the delayed weight value548in respective registers in the PE 01, and pass the delayed values to the neighboring PE 02 in the next time period.

Thus, the input dataset0 may be fed, one input data element every time period, into the first row of the systolic array302, and passed sequentially from the PE 00 to the PE 0y. As the input data element222passes through a PE, the stored input data element528can be multiplied with the stored weight value530, and accumulated with the stored input partial sum536by the adder510. If either the input data element222or the weight224is zero, the inputs to the multiplier508may not change to reduce power consumption, and the stored input partial sum536may be provided as the output partial sum542via the column output bus106. The output partial sum542of the PE 00 may be passed on as the input partial sum534for the neighboring PE 10 in the second row. The same operations may be repeated by each row of the systolic array302and corresponding output datasets may be generated.

FIG. 6shows an apparatus600for neural network computations according to some embodiments of the disclosed technologies. The apparatus600may be part of a computer system, e.g., a host server. For example, the host server may provide multi-tenant compute services for data processing applications such as an image recognition service, text-based data processing (e.g., processing of search queries), audio data processing, video data processing, etc. In some embodiments, a host device may operate a software application and communicate with the apparatus600to make a prediction based on computations with a prediction model utilizing a neural network processor. For example, the host device can make the prediction by identifying information included in an input data set for an image, text, audio, video, etc. using the prediction model.

The apparatus600may include a neural network processor602coupled to memory614, a host interface616, and a direct memory access (DMA) controller618via an interconnect620. The neural network processor602may include a computing engine604, a computation controller606, a state buffer608, an output buffer610, and an activation engine612. The neural network processor602can provide the computing resources to support the computations with the prediction model. The neural network processor602may be implemented as a system on chip (SoC), a field programmable gate array (FPGA), or any suitable circuit.

The memory614may be configured to store instructions, input data sets (e.g., pixel data of an image) and the weights (e.g., weights corresponding to certain visual and/or non-visual features) received from the host device. The memory614may also be configured to store outputs of the neural network processor602(e.g., one or more image recognition decisions on the input images in the form of output data sets). The memory614may include any suitable memory, e.g., dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate DRAM (DDR DRAM), storage class memory (SCM), flash memory, etc.

The host interface616may be configured to enable communication between the host device and the neural network processor602. For example, the host interface616may be configured to transmit memory descriptors including the memory addresses of the stored data (e.g., input data sets, weights, results of computations, etc.) between the host device and the neural network processor602. The host interface616may include, e.g., a peripheral component interconnect express (PCIe) interface, or any suitable interface for communicating with the host device. The host device may include a host processor and a host memory.

The DMA controller618may be configured to perform DMA operations to transfer data between the neural network processor602and the host device. For example, as discussed above, the host device can store the instructions, input data sets, and the weights in the memory614. The host device can provide the memory addresses for the stored instructions, data, and the weights to the neural network processor602(e.g., in the form of memory descriptors). The neural network processor602can then obtain the stored instructions, data, and the weights based on the memory addresses provided by the host device. The neural network processor602can also store the results of computations (e.g., one or more image recognition decisions) in the memory614, and provide the memory addresses for the stored results to the host device.

The state buffer608may be configured to provide caching of data used for computations at the computing engine604. The data cached at the state buffer608may include, e.g., the input data sets and the weights acquired from the memory614, as well as intermediate outputs of computations at the computing engine604. The caching can reduce the effect of memory access bottleneck (e.g., caused by the latencies at the memory614, the DMA controller618, the interconnect620, etc.) on the performance of the computing engine604. The state buffer608can be an on-chip memory device and may include a static random access memory (SRAM) or any suitable memory.

The computation controller606may be configured to provide controls to various components of the neural network processor602to perform neural network computations. In some implementations, the computation controller606may read the instructions stored in the memory614and schedule the executions of the instructions by the computing engine604. In the first embodiment, the computation controller606may perform scheduling of loading the weights into the computing engine604prior to reading the input data elements from the state buffer608. For example, as discussed with reference toFIG. 2andFIG. 4, the computation controller606may provide the opcode230and the weight load signal232to the computing engine604based on the instructions received from the host device. The computation controller606may provide appropriate values of the opcode230to the computing engine604which may be decoded by each PE in the computing engine to perform a corresponding operation. For example, the computing engine604may use the weight load signal232and the opcode230to pre-load the weights in all the PEs in the computing engine604. Once the weights have been pre-loaded, the computation controller606may perform scheduling of loading the input data elements into the computing engine604, sequentially, in uniform time periods, from the state buffer608to start the arithmetic computations.

In the second embodiment, the computation controller606may perform scheduling of loading the weights and the input data elements into the computing engine604, sequentially, in uniform time periods, from the state buffer608. The computation controller606may schedule loading of the weights and the input data elements in a respective first PE of each row in the systolic array302using a respective row data bus. For example, a respective input data element and a weight value may be loaded per cycle in the first PE of the respective row.

In another embodiment, the computation controller606may schedule loading of the weights in the systolic array302in parallel for each row using a respective column data bus for each PE in a given row. For example, weights for each row may be loaded in parallel per cycle. In some implementations, the computation controller606may determine a data type for the input data set based on the instructions received from the host device. The instructions may be in the form of an opcode. The data type may indicate a size and a type of the input data element, e.g., 4-bit, 8-bit, 16-bit, signed, unsigned, or floating point.

The computing engine604may be configured to perform computations for the neural network. In some embodiments, the computing engine604may include a set of PEs configured to perform one or more arithmetic operations involved in the neural network computations. Each PE may perform multiply-accumulate operations using input data sets and associated weights. For example, the computing engine604may include the systolic array302, and the circuit304comprising the zero input data detectors306a-306x, and the zero weight detectors308a-308x. In some embodiments, the zero input data detectors306a-306x, and the zero weight detectors308a-308xmay be external to the computing engine604. The computing engine604may execute instructions as scheduled by the computation controller606to load the weights and the input datasets sequentially from the state buffer608into the computing engine604.

In the first embodiment, the weights may be pre-loaded prior to reading the input datasets from the state buffer608, as discussed with reference toFIG. 4. The respective zero weight indicators corresponding to each weight may be cached locally in each PE and the cached values may be used to perform arithmetic computations with the respective input data element as the input data element is fed into the computing engine604along with the corresponding zero data element indicator. In the second embodiment, the weights and the input datasets may be read simultaneously from the state buffer608, as discussed with reference toFIG. 5. The corresponding zero data element indicator and the zero weight indicator may be provided by the respective zero detector circuits and propagated sequentially from one PE to another for the respective row. The weights and the input datasets can be obtained from the state buffer608using one or more interfaces. In certain embodiments, the computing engine604may perform the arithmetic computations to reduce the dynamic power consumption of the systolic array302using the respective zero data element indicator and the zero weight indicator signals as discussed with reference toFIGS. 2-5, and provide the computations results to be stored in the output buffer610.

The output buffer610may include a set of registers to store the output data sets generated by the computing engine604. In some implementations, the output buffer610may also enable additional processing such as, e.g., a pooling operation to reduce the size of the stored outputs. In some implementations, the computing engine604can be operated to perform computations for a particular neural network layer, and the output buffer610can process the outputs of that neural network layer and store the processed output datasets (with or without processing by the activation engine612) at the state buffer608. The processed output datasets may be used by the computing engine604as the intermediate outputs. In some embodiments, the output buffer610may include adders to accumulate the partial sums generated for different sets of filters and input data sets to generate a convolution output array. The final output value of the convolution output array stored in the state buffer608can be retrieved by the computation controller606for storing at the state buffer608.

The activation engine612may be configured to apply one or more activation functions (e.g., ReLu function) on the output of the output buffer610. For example, the activation engine612may include one or more lookup tables (e.g., in the form of multiplexer circuits) that can map the input to one of the candidate outputs representing the result of applying the activation function to the input. In some examples, the activation engine612may also include a bypass path to allow outputs from the output buffer610to be stored directly at the state buffer608when activation functions are not to be applied.

FIG. 7shows a method700executed by a PE for neural network computations, according to some embodiments of the disclosed technologies. The PE may be part of the systolic array302, e.g., the PE 00. The systolic array302may be part of the computing engine704.

In a step702, the PE may receive a zero weight indicator, via a first port, to indicate whether a weight value is zero. For example, the PE 00 may include means to receive the zero weight indicator228to indicate that the weight224is zero. The weight224may have been received from the host device into the memory614. The PE 00 may receive the zero weight indicator228from the first zero weight detector308avia the row input bus102. As discussed with reference toFIG. 3, the first zero weight detector308amay include comparators or other circuits to determine that the weight224is zero. For example, the first zero weight detector308amay set the zero weight indicator228to “1” when it detects that the weight224is zero. The computation controller606may schedule loading of the weight224corresponding to the filter0, sequentially, in uniform time periods, from the state buffer608into the PE 00 of the computing engine604prior to scheduling loading of the input data elements into the PE 00.

In a step704, the PE may store the weight value to provide a stored weight value for pre-loading the weight value in the PE. For example, the PE 00 may include means for pre-loading the weight224in the PE 00. The computation controller606may provide the opcode230to the computing engine604with a certain opcode value for loading the respective weights in each PE of the computing engine604. As discussed with reference toFIG. 2, the weight224may be stored in the cached weight register220using the weight load signal232. The cached weight246previously loaded into the cache register220may be shifted into the weight register206at the start of the arithmetic computations based on the start computations signal260. The stored weight value248may be used to perform arithmetic computations with the stored input data element228.

In a step706, the PE may receive, via a second port, a zero data element indicator for a current operation to indicate whether an input data element associated with the zero data element indicator is zero. For example, the PE 00 may include means to receive the zero data element indicator226to indicate that the input data element222associated with the zero data element indicator226is zero. As discussed with reference toFIG. 3, the PE 00 may receive the zero data element indicator226via the second port from the first zero input data detector306ausing the row input bus102. The first zero input data detector306amay include comparators or other circuits to determine that the input data element222is zero. For example, the first zero input data detector306amay set the zero data element indicator226to ‘1” when it detects that the input data element222associated with the zero data element indicator226is zero. The input data element222may correspond to an input data set, e.g., the input dataset0. In one instance, the input dataset0 may belong to a dataset received from a host device into the memory614to perform data processing. In another instance, the input dataset0 may belong to an intermediate output generated by the computing engine604, which have gone through an activation function in the activation engine612. The computation controller606may schedule loading of the input data elements corresponding to the input dataset0, sequentially, in uniform time periods, from the state buffer608into the PE 00 of the computing engine604after loading of the weights into the systolic array.

In a step708, the PE may skip storing of the input data element to provide a stored input data element based on the zero data element indicator indicating that the input data element associated with the zero data element indicator is zero. For example, the PE 00 may include means to skip storing of the input data element222to provide the stored input data element228based on the zero data element indicator226indicating that the input data element222associated with the zero data element indicator226is zero. Referring back toFIG. 4, the PE 00 may include the data element load generator202to generate the data load signal242, which can be used to skip storing of the input data element222in the data register204. For example, when the zero data element indicator226or the NOP258is asserted, storing of the input data element222in the data register204can be skipped. In some embodiments, the data element load generator202may include a NOR circuit to perform a NOR operation to set the data load signal242to “0” when the zero data element indicator226is “1”, or the NOP258is “1.” The data load signal242may be used to disable the loading of the input data element222in the data register204under certain conditions and therefore the stored input data element244may hold a value from the previous operation.

In a step710, the PE may perform a multiplication operation between the stored input data element and the stored weight value to generate a multiplication result upon change in a value of any one of the stored input data element or the stored weight value. For example, the PE 00 may include means to perform a multiplication operation between the stored input data element244and the stored weight value248to generate the multiplication result250upon change in a value of the stored input data element244or the stored weight value248. Referring back toFIG. 4, the PE 00 may include the multiplier208to perform the multiplication operation between the stored input data element244and the stored weight value248to generate the multiplication result250. The multiplier208may perform the multiplication upon a change in a value of either the stored input data element244in the data register204, or the stored weight value248in the weight register206. As discussed previously, the value of the stored input data element244in the data register204may change based on the data load signal242. For example, the data load signal242may disable loading a respective new value in the data register204when the zero data element indicator226or the NOP258is asserted, which may avoid toggling the stored input data element244. Since the weight224has been pre-loaded in the PE 00, the stored weight value248does not change during the computations. Thus, dynamic power consumption can be reduced which may have occurred as a result of a multiplication operation performed by the multiplier208.

Embodiments of the disclosed technologies can provide systems and methods to reduce dynamic power consumption in the PEs using zero detector circuits by skipping multiplication operations with a zero value on the input data element. Additionally, use of the respective zero detector circuits for detecting zeros on the input data elements and the weights entering each row of the systolic array, and passing the zero indicators to all the PEs in the array can minimize the gate count and power consumption as compared to using respective zero detectors within each PE in the array.

FIG. 8illustrates an example of a computing device800. Functionality and/or several components of the computing device800may be used without limitation with other embodiments disclosed elsewhere in this disclosure, without limitations. A computing device800may perform computations to facilitate processing of a task. As an illustrative example, computing device800can be part of a server in a multi-tenant compute service system. Various hardware and software resources of computing device800(e.g., the hardware and software resources associated with data processing) can be allocated to a client upon request.

In one example, the computing device800may include processing logic802, a bus interface module804, memory806, and a network interface module808. These modules may be hardware modules, software modules, or a combination of hardware and software. In certain instances, modules may be interchangeably used with components or engines, without deviating from the scope of the disclosure. The computing device800may include additional modules, which are not illustrated here for the ease of illustration. In some implementations, the computing device800may include fewer modules. In some implementations, one or more of the modules may be combined into one module. One or more of the modules may be in communication with each other over a communication channel810. The communication channel810may include one or more busses, meshes, matrices, fabrics, a combination of these communication channels, or some other suitable communication channel.

The processing logic802may include application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), systems-on-chip (SoCs), network processing units (NPUs), processors configured to execute instructions or any other circuitry configured to perform logical arithmetic and floating point operations. Examples of processors that may be included in the processing logic802may include processors developed by ARM®, MIPS®, AMD®, Intel®, Qualcomm®, and the like. In certain implementations, processors may include multiple processing cores, wherein each processing core may be configured to execute instructions independently of the other processing cores. Furthermore, in certain implementations, each processor or processing core may implement multiple processing threads executing instructions on the same processor or processing core, while maintaining logical separation between the multiple processing threads. Such processing threads executing on the processor or processing core may be exposed to software as separate logical processors or processing cores. In some implementations, multiple processors, processing cores or processing threads executing on the same core may share certain resources, such as for example busses, level 1 (L1) caches, and/or level 2 (L2) caches. The instructions executed by the processing logic802may be stored on a computer-readable storage medium, for example, in the form of a computer program. The computer-readable storage medium may be non-transitory. In some cases, the computer-readable medium may be part of the memory806. The processing logic802may also include hardware circuities for performing artificial neural network computations including, for example, the neural network processor602, etc.

The access to the processing logic802can be granted to a client to provide the personal assistant service requested by the client. For example, the computing device800may host a virtual machine, on which an image recognition software application can be executed. The image recognition software application, upon execution, may access the processing logic802to predict, for example, an object included in an image. As another example, access to the processing logic802can also be granted as part of bare-metal instance, in which an image recognition software application executing on a client device (e.g., a remote computer, a smart phone, etc.) can directly access the processing logic802to perform the recognition of an image.

The memory806may include either volatile or non-volatile, or both volatile and non-volatile types of memory. The memory806may, for example, include random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory, and/or some other suitable storage media. In some cases, some or all of the memory806may be internal to the computing device800, while in other cases some or all of the memory may be external to the computing device800. The memory806may store an operating system comprising executable instructions that, when executed by the processing logic802, provides the execution environment for executing instructions providing functionality to perform convolution computations for the computing device800. The memory806may also store, for example, software applications for performing artificial neural network computations. The memory may also store and maintain several data structures and tables for facilitating the functionality of the computing device800.

The bus interface module804may enable communication with external entities, such as a host device and/or other components in a computing system, over an external communication medium. The bus interface module804may include a physical interface for connecting to a cable, socket, port, or other connection to the external communication medium. The bus interface module804may further include hardware and/or software to manage incoming and outgoing transactions. The bus interface module804may implement a local bus protocol, such as Peripheral Component Interconnect (PCI) based protocols, Non-Volatile Memory Express (NVMe), Advanced Host Controller Interface (AHCI), Small Computer System Interface (SCSI), Serial Attached SCSI (SAS), Serial AT Attachment (SATA), Parallel ATA (PATA), some other standard bus protocol, or a proprietary bus protocol. The bus interface module804may include the physical layer for any of these bus protocols, including a connector, power management, and error handling, among other things. In some implementations, the computing device800may include multiple bus interface modules for communicating with multiple external entities. These multiple bus interface modules may implement the same local bus protocol, different local bus protocols, or a combination of the same and different bus protocols.

The network interface module808may include hardware and/or software for communicating with a network. This network interface module808may, for example, include physical connectors or physical ports for wired connection to a network, and/or antennas for wireless communication to a network. The network interface module808may further include hardware and/or software configured to implement a network protocol stack. The network interface module808may communicate with the network using a network protocol, such as for example TCP/IP, Infiniband, RoCE, Institute of Electrical and Electronics Engineers (IEEE) 802.11 wireless protocols, User Datagram Protocol (UDP), Asynchronous Transfer Mode (ATM), token ring, frame relay, High Level Data Link Control (HDLC), Fiber Distributed Data Interface (FDDI), and/or Point-to-Point Protocol (PPP), among others. In some implementations, the computing device800may include multiple network interface modules, each configured to communicate with a different network. For example, in these implementations, the computing device800may include a network interface module for communicating with a wired Ethernet network, a wireless 802.11 network, a cellular network, an Infiniband network, etc. In some embodiments, the computing device800may receive a set of parameters, such as the aforementioned weight values for convolution computations, from a server through network interface module808.

The various components and modules of the computing device800, described above, may be implemented as discrete components, as a System on a Chip (SoC), as an ASIC, as an NPU, as an FPGA, or any combination thereof. In some embodiments, the SoC or other component may be communicatively coupled to another computing system to provide various services such as traffic monitoring, traffic shaping, computing, etc. In some embodiments of the technology, the SoC or other component may include multiple subsystems as disclosed herein.