Patent Description:
Accordingly, it may be understood that these statements are to be read in this light, and not as admissions of prior art.

Integrated circuits, such as field programmable gate arrays (FPGAs), may include circuitry to perform various mathematical operations. For example, a deep learning neural network may be implemented in one or more integrated circuit devices for machine learning applications. The integrated circuit devices may perform several operations to output results for the neural network. However, in some instances, throughput of mathematical operations in neural networks may be limited by the hardware of the integrated circuit. Because of these limitations, the neural network may perform at a rate slower than desired. <NPL> relates to FPGA-based vector processing. <NPL> relates to an architecture for Linear Quadratic Regulator control.

It may be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another. Moreover, it may be appreciated that such a development effort might be complex and time consuming, but would nevertheless be a routine undertaking of design, fabrication, and manufacture for those of ordinary skill having the benefit of this disclosure.

Machine learning is used in a variety of settings to perform tasks through the use of examples. For example, neural networks may be used to perform a task without task-specific programming. That is, neural networks may be trained from prior data to classify or infer information from current data. For instance, training data may be used to identify images that contain an object by analyzing other images that include and do not include the object. While images are used as an example, this is simply meant to be illustrative and any suitable neural network task may be performed in the embodiments described below.

Configurable devices, such as programmable logic devices (PLDs), may perform one or more operations to execute tasks via machine learning. For example, integrated circuits (ICs), such as field programmable gate arrays (FPGAs), may include one or more digital signal processing (DSP) blocks, or DSP circuits, having one or more specialized processing blocks to perform arithmetic operations on data received by the DSP block. One type of specialized processing blocks in the DSP block may be multiply-accumulate (MAC) blocks, or MAC circuits, that include one or more multiplier circuits and/or one or more accumulator circuits. For instance, in some FPGAs, the MAC blocks may be hardened intellectual property (IP) blocks having specialized multiply circuitry coupled to specialized adder circuitry. Examples of operations performed by the MAC blocks include dot products, vector multiplications, and the like. As described below, the one or more multipliers of the DSP blocks may be used to perform neural network arithmetic operations during the classification or inference stage. However, throughput of the digital signal processor (DSP) may be limited by hardware of the IC. For example, the number of MAC blocks may limit the performance (e.g., speed) of the IC in performing arithmetic operations of the neural network.

Some arithmetic operations in neural networks may not involve the same precision as the precision designed to be processed in the MAC blocks. For example, the MAC block may include circuitry that processes <NUM> bit operands, but the neural network may involve multiplying lower precision <NUM> bit operands. The systems and methods described below improve neural network performances in ICs by better utilizing capacity of operands in multiply operations. By combining more than one quantity into each operand of multiply operations, speed of performing MAC operations (e.g., weightings and summations) by MAC blocks of ICs in the neural network applications may be improved. For example, two or more quantities may be packed into a first operand that is received by multiplier circuitry. Two or more quantities may be packed into a second operand received by the multiplier circuitry. The multiplier circuitry may then perform the multiplication operation between the first operand and the second operand to determine a product between each of the respective quantities. The multiplier circuitry may then output each of the products to be accumulated. To prevent overflow from the multiplication, a gap may be included between each of the quantities combined in the operands.

Further, to reduce likelihood of overflow from the accumulation, accumulator circuitry of the MAC block may be bypassed to a soft logic accumulator. That is, the multiplication of the MAC operation may be performed in a hardened multiplier that is specialized at performing multiplications and accumulations, and the accumulation of the MAC operation may be performed in soft logic to prevent overflow due to accumulating several products output from the multiplication.

With the foregoing in mind, <FIG> illustrates a block diagram of a data processing system <NUM> that may be used to perform one or more tasks via machine learning. The data processing system <NUM> may include a processor <NUM> operatively coupled to a memory <NUM>. The processor <NUM> may execute one or more instructions stored on the memory <NUM> to perform the one or more tasks. The data processing system <NUM> may include a network interface <NUM> to send and/or receive data via a network to communicate with other electronic devices. The data processing system <NUM> may include one or more inputs/outputs (I/O) <NUM> that may be used to receive data via I/O devices, such as a keyboard, mouse, display, buttons, or other controls. The data processing system <NUM> may include a machine learning circuit <NUM> that performs one or more tasks using machine learning methods and techniques. The machine learning circuit <NUM> may include a PLD, such as an FPGA. Each of the processor <NUM>, the memory <NUM>, the network interface <NUM>, the I/O <NUM>, and the machine learning circuit <NUM> may be communicatively coupled to one another via interconnection circuitry <NUM>, such as a communication bus.

The hardware of the machine learning circuit <NUM> may perform one or more tasks using neural networks <NUM> and <NUM>. Turning now to a more detailed discussion of an example of the machine learning circuit <NUM>, <FIG> illustrates an IC <NUM>, which may be a programmable logic device, such as a field-programmable gate array (FPGA) <NUM>. For the purposes of this example, the device is referred to as an IC <NUM>, though it should be understood that the device may be any suitable type of device (e.g., application-specific standard product) may be used. As shown, IC <NUM> may have input/output circuitry <NUM> for driving signals off IC <NUM> and for receiving signals from other devices via input/output pins <NUM>. Interconnection resources <NUM>, such as global and local vertical and horizontal conductive lines and buses, may be used to route signals on IC <NUM>. Additionally, interconnection resources <NUM> may include fixed interconnects (conductive lines) and programmable interconnects (i.e., programmable connections between respective fixed interconnects). Programmable logic <NUM> may include combinational and sequential logic circuitry. For example, programmable logic <NUM> may include look-up tables, registers, and multiplexers. In various embodiments, the programmable logic <NUM> may be configured to perform a custom logic function. The programmable interconnects associated with interconnection resources may be considered to be a part of programmable logic <NUM>. The IC <NUM> may include programmable elements <NUM> with the programmable logic <NUM>. The programmable elements <NUM> may be based on any suitable programmable technology, such as fuses, antifuses, electrically-programmable read-only-memory technology, random-access memory cells, mask-programmed elements, and so forth.

The circuitry of IC <NUM> may be organized using any suitable architecture. As an example, the logic of IC <NUM> may be organized in a series of rows and columns of larger programmable logic regions, each of which may have multiple smaller logic regions. The logic resources of IC <NUM> may be interconnected by interconnection resources <NUM> such as associated vertical and horizontal conductors. For example, in some embodiments, these conductors may include global conductive lines that span substantially all of IC <NUM>, fractional lines such as half-lines or quarter lines that span part of IC <NUM>, staggered lines of a particular length (e.g., sufficient to interconnect several logic areas), smaller local lines, or any other suitable interconnection resource arrangement. Moreover, in further embodiments, the logic of IC <NUM> may be arranged in more levels or layers in which multiple large regions are interconnected to form still larger portions of logic. Still further, other device arrangements may use logic that is not arranged in a manner other than rows and columns. As explained below, the machine learning circuit <NUM> may perform the one or more tasks using hardware of the IC <NUM>. For example, the machine learning circuit <NUM> may utilize arithmetic logic circuitry to perform arithmetic operations used in machine learning methods and techniques.

<FIG> is a network diagram of an example of a machine learning network, such as a neural network <NUM>, which may be utilized to perform one or more tasks on the machine learning circuit <NUM>. While the neural network <NUM> is described in detail as an example, any suitable machine learning methods and techniques may be used. The neural network <NUM> includes a set of inputs <NUM>, <NUM>, <NUM>, and <NUM>, a set of weights <NUM>, <NUM>, <NUM>, and <NUM>, a set of summations <NUM> and <NUM>, and a resultant value <NUM>. Each of the inputs <NUM>, <NUM>, <NUM>, and <NUM> is weighted with a respective weight to determine a respective weighted value <NUM>, <NUM>, <NUM>, and <NUM>. The weighted values <NUM> and <NUM> may be summed at the summation <NUM>, and the weighted values <NUM> and <NUM> may be summed at the summation <NUM>. The resultant value <NUM> may be output from the summations <NUM> and <NUM> and used to perform one or more tasks from prior data. While four inputs and two summations are shown, this is meant to be illustrative and any suitable combination of inputs, weightings, summations, and connections therebetween may be used.

<FIG> is a flow diagram of a process <NUM> that may be performed in conjunction with the neural network <NUM> on the IC <NUM>. At block <NUM>, the IC <NUM> may perform training in which the weighted values <NUM>, <NUM>, <NUM>, and <NUM> are determined and/or adjusted such that the weights applied to the inputs <NUM>, <NUM>, <NUM>, and <NUM> indicate a likelihood that the respective inputs <NUM>, <NUM>, <NUM>, and <NUM> predict the resultant value <NUM>.

Upon training the neural network <NUM>, at block <NUM>, the IC <NUM> may perform inferences and/or classifications on new data. In an example involving image recognition, for example, the neural network <NUM> may be trained using images of shapes (e.g., circles, triangles, squares) in which the shape in the image is known. Then, the IC <NUM> may classify the shapes of new data using the neural network <NUM> after the weights have been adjusted from the training data. By adjusting the weights applied to the inputs <NUM>, <NUM>, <NUM>, and <NUM> based on the training data, weights may be obtained that, when applied to new images, reflect a likelihood that the respective input of the new image includes a certain shape. In some embodiments, continued learning may occur in which new data is then verified and the weights are continually adjusted. Each of the blocks <NUM> may be performed via the machine learning circuit <NUM> and/or some operations of each of the blocks <NUM> may be performed via the processor <NUM>.

<FIG> is a network diagram of an example of a neural network <NUM> having an input layer <NUM>, more than one computational layers <NUM>, and an output layer <NUM>. The illustrated embodiment may be referred to as a deep neural network due to having more than one computational layers <NUM>, also referred to as hidden layers. As the number of computational layers <NUM> increases, the complexity and processing of inputs increases. In the illustrated embodiment, each input is weighted and summed at four summations, and each respective summation is then weighted and summed at three summations, which are then used to output a resultant value.

As explained below the circuitry of the IC <NUM> may further include one or more DSP blocks. The DSP block may include one or more (multiply-accumulate) MAC blocks, or MAC circuits. Each MAC block may include hardened circuitry (e.g., multiplier circuitry and accumulator circuitry) that is designed and specialized to perform multiplication and accumulation operations. While the MAC block may include circuitry that performs multiplication and accumulation of inputs having a certain amount of precision, the neural network <NUM> may have inputs <NUM>, <NUM>, <NUM>, and <NUM> and weights of lower precision than the circuitry of the MAC block. For example, while the neural network <NUM> may utilize weights and inputs <NUM>, <NUM>, <NUM>, and <NUM> of six-bit precision, the MAC block may include circuitry designed to process eighteen-bit inputs. By combining more than one value from the neural network <NUM> into the same operand of the MAC block, each multiplication of the MAC block may process additional values associated with the neural network <NUM> to improve throughput of the neural network <NUM>.

<FIG> is an example of a set of data structures <NUM> of the IC <NUM> having combined values in the same operand to allow the IC <NUM> to process values of the neural network <NUM> at a faster rate. The IC <NUM> may combine a first value <NUM> and a second value <NUM> into a first operand <NUM>. That is, the IC <NUM> may pack each bit of the first value <NUM> and each bit of the second value <NUM> into the first operand <NUM>. For example, a first component (e.g., first set of bits) of the first operand <NUM> may represent a first value <NUM> and a second component (e.g., second set of bits) of the first operand <NUM> may represent a second value <NUM>. Further, the operand <NUM> may include a gap between the first value <NUM> and the second value <NUM> to prevent overflow. For example, the gap <NUM> may be at least the number of bits of the first value <NUM> or the second value <NUM>. The first value <NUM> may be the first input <NUM> and the second value <NUM> may be the second input <NUM>.

Similarly, the IC <NUM> may combine a third value <NUM> and a fourth value <NUM> into a second operand <NUM>. The second operand <NUM> may include a gap <NUM> between the third value <NUM> and the fourth value <NUM> to prevent overflow. The gap <NUM> may be at least the number of bits of the third value <NUM> or the fourth value <NUM>. In the example described above in which the neural network <NUM> utilizes six-bit precision, the first value <NUM>, the second value <NUM>, the third value <NUM>, the fourth value <NUM>, and the gaps <NUM> and <NUM> may each be six bits. The third value <NUM> may be a first weight to be applied to the first input <NUM> and the fourth value <NUM> may be a second weight applied to the second input <NUM>.

The IC <NUM> may perform a multiplication operation on the first operand <NUM> and the second operand <NUM> such that a multiplied product <NUM> includes a first product <NUM> of the first value <NUM> multiplied with the third value <NUM> and a second product <NUM>, from the same multiplication operation, of the second value <NUM> multiplied with the fourth value <NUM>. That is, by combining or packing more than one value into each operand <NUM> and <NUM> with sufficient gap <NUM> and <NUM> between the values, the multiplied product <NUM> may include each respective product without overflow. For example, in the neural network <NUM>, the first product <NUM> may be the weighted value <NUM> from the first weight applied to the first input <NUM> and the second product <NUM> may be the second weighted value <NUM> from the second weight applied to the second input <NUM>. By combining the values from the neural network <NUM> into each operand <NUM> and <NUM>, the resultant value <NUM> may be determined at a faster rate due to increased throughput.

Each of the first product <NUM> and the second product <NUM> may subsequently be split from the multiplied product <NUM> and accumulated. Because the accumulation may be a faster operation than the multiplication, the performance of the neural network <NUM> may be improved by determining more than one product from a single multiplication operation using more of the available precision in the hardened multiplier circuitry of the IC <NUM>. Further, the hardened circuitry of MAC blocks in the IC <NUM> may be specialized to perform the multiplications to determine the weighted values <NUM>, <NUM>, <NUM>, and <NUM> at a faster rate than in circuitry that executes multiplications in soft logic due to the specialization of the hardened circuitry.

<FIG> is a block diagram of circuitry of the IC <NUM> that performs the arithmetic operations described with respect to <FIG>. The IC <NUM> may include a DSP block <NUM> having first input circuitry <NUM> and second input circuitry <NUM> to receive a first operand <NUM> and a second operand <NUM> respectively. The DSP block <NUM> may include a MAC block <NUM> having multiplier circuitry <NUM> that multiplies the first operand <NUM> with the second operand <NUM> and outputs a product. That is, the multiplier circuitry <NUM> may be designed or hardened with circuitry to perform multiplication operations on operands of a certain precision. By including more than one value of lower precision than the designed operand precision into the operand prior to executing the multiplication operation, more than one product may be determined from the multiplication operation.

In some embodiments, the MAC block <NUM> may include adder circuitry <NUM> that may add the products from the multiplier circuitry <NUM>. Upon completing the MAC operation, the MAC block <NUM> may output a result via the output circuitry <NUM>. In the illustrated embodiment, the IC <NUM> may include more than one DSP block <NUM> (e.g., <NUM>, <NUM>, <NUM>, <NUM>, or more), and each DSP block may include more than one MAC block <NUM> (e.g. <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or more).

The MAC block <NUM> may include bypass circuitry <NUM> (e.g., multiplexor) to bypass the adder <NUM> and to provide the multiplied product <NUM> to soft logic <NUM> of the IC <NUM>. Further, the IC <NUM> may then perform the summations <NUM> and <NUM> of the neural network in the soft logic <NUM> of the IC <NUM>. The soft logic <NUM> may refer to programmed instructions (e.g., code) stored in memory on the IC <NUM> to execute operations of the IC <NUM>. The IC <NUM> may be programmed to execute instructions to split the first product <NUM> and the second product <NUM> from the multiplied product <NUM>. The IC <NUM> may then execute instructions to accumulate <NUM> the first product (e.g., first weighted value <NUM>) with one or more other products (e.g., weighted value <NUM>) to determine a total from the summation <NUM>. The IC <NUM> may execute instructions to accumulate <NUM> the second product (e.g., weighted value <NUM>) with one or more other products (e.g., weighted value <NUM>). For example, the first product (e.g., weighted value <NUM>) may be held at block <NUM>. The IC <NUM> may then perform another multiplication to determine third and fourth products (e.g., the weighted values <NUM> and <NUM>) by combining the fifth and sixth values (e.g., inputs <NUM> and <NUM>) into a third operand and seventh and eighth values (e.g., the weights for the respective inputs) into a fourth operand. The third and fourth products (e.g., weighted values <NUM> and <NUM>) may then be added to each respective total being held <NUM> and <NUM>. By implementing the accumulators in soft logic <NUM>, more accumulation operations may be performed with less or no risk of overflow. By moving to lower precision neural networks than the six bit example, the IC <NUM> may obtain additional products in each multiplication operation.

<FIG> is a more generalized example of a data structure <NUM> used in performing multiplication operations in the neural network <NUM> on the IC <NUM>. The IC <NUM> may combine values A[<NUM>] to A[n] into a first operand. Similarly, the IC <NUM> may combine values B[<NUM>] to B[n] into a second operand. Each of the values <NUM> and <NUM> may be separated from other values <NUM> and <NUM> by gaps <NUM> and <NUM> to prevent overflow.

Upon performing the multiplication operation, the multiplied product <NUM> may include a set of multiplied values <NUM> C[<NUM>] to C[n] from multiplying each respective value of A to B. Lower precision levels of the neural network may allow for additional values to be included in each multiplication operation. For example in an eighteen bit multiplication operation the following table may reflect precision levels with respect to the number of values in each operand:.

This relationship may be more generalized according to the following equation: <MAT> Where numcomp refers to the number of values that may be included in each operand, WidthMult refers to the number of bits in each operand, and precision refers to the number of bits used in operations in the neural network <NUM>.

<FIG> is a generalized block diagram of circuitry of the IC <NUM> that performs arithmetic operations for the neural network of <FIG> using the data structure of <FIG>. The IC <NUM> includes similar circuitry to the circuitry described with respect to <FIG>. Further, additional accumulators (e.g., in code) may be used for each of the values <NUM> in the multiplied product <NUM>.

<FIG> is a block diagram of another data structure <NUM> that may be used in conjunction with the circuitry described with respect to <FIG>. The data structure <NUM> includes a first operand having N values <NUM> A[<NUM>] to A[n] with gaps <NUM> between each of the values <NUM>. The data structure <NUM> includes a second operand having a single value B[<NUM>] <NUM> and padding <NUM> throughout the remainder of the first operand. The single value B[<NUM>] may be the same precision as each of the N values of the first operand. Upon performing the multiplication operation, the IC <NUM> may determine a first multiplied product <NUM> by multiplying a first value A[<NUM>] of the first operand with B[<NUM>] <NUM>. The IC <NUM> may determine a second multiplied product <NUM> by multiplying a second value A[<NUM>] of the first operand with B[<NUM>] <NUM>. That is, B[<NUM>] <NUM> may be multiplied with each of the N values <NUM> of the first operand.

<FIG> is a flow diagram of a process <NUM> performed by the IC <NUM> to perform the arithmetic operations of the neural network <NUM> to output a resultant value <NUM> described in conjunction with the example of <FIG> and <FIG>. At block <NUM>, the IC <NUM> may combine (e.g., pack) a first value and a second value into a first operand. In some embodiments, the IC <NUM> may combine (e.g., pack) a third value and a fourth value into a second operand. As mentioned above with respect to <FIG>, additional values may be included in each of the first operand and the second operand. Further, in certain embodiments described with respect to <FIG>, the IC <NUM> may simply have a single value in the second operand. At block <NUM>, the IC <NUM> may multiply the first operand with the second operand to determine a first multiplied product based at least in part on the first value and a second multiplied product based at least in part on the second value. In the example in which the second operand includes a third value and a fourth value, for instance, the first multiplied product may be the first value multiplied by the third value and the second multiplied product may be the second value multiplied by the fourth value. In this manner, more than one multiplied products may be determined from the same multiplication operation performed by the hardened multiplier circuitry. In some embodiments, the multiplication operation may be performed in the hardened multiplier circuitry and the multiplied result having both the first multiplied product and the second multiplied product may be output to soft logic where the first multiplied product and the second multiplied product may be split from one another.

Claim 1:
An integrated circuit device comprising:
multiplier circuitry;
first input circuitry to the multiplier circuitry, wherein the first input circuitry is configurable to receive a first operand, wherein a first component of the first operand comprises a first value, a second component of the first operand comprises a second value, and a third component of the first operand comprises a third value;
second input circuitry to the multiplier circuitry, wherein the second input circuitry is configurable to receive a second operand, wherein a first component of the second operand comprises a fourth value, a second component of the second operand comprises a fifth value, and a third component of the second operand comprises a sixth value, wherein the multiplier circuitry is configurable to multiply the first operand and the second operand to produce a plurality of results corresponding to a plurality of distinct multiply operations, wherein the plurality of results comprises:
a first result of the plurality of results, wherein the first result is equivalent to a first multiply operation based on the first value and the fourth value;
a second result of the plurality of results, wherein the second result is equivalent to a second multiply operation based on the second value and the fifth value; and
a third result of the plurality of results, wherein the third result is equivalent to a third multiply operation based on the third value and the sixth value; and
an accumulator circuitry, wherein the accumulator circuitry is configurable to perform a separate accumulation of each of the plurality of results to a respective total.