Patent Description:
The present disclosure relates to a convolutional computation device that performs a convolution calculation.

In the convolution calculation, output data is generated by convolving weight data forming a predetermined filter into the input data. In the conceivable convolutional computation device, the convolution calculation is processed by converting the convolution operation into a matrix operation (see, for example, Patent Literature <NUM>).

Patent Literature <NUM> discloses technologies related to a processor and method for performing outer product and outer product accumulation operations on vector operands requiring large numbers of multiplies and accumulations.

Patent Literature <NUM> discloses technologies that include executing a convolutional neural network layer on an image processor having an array of execution lanes and a two-dimensional shift register. The two-dimensional shift register provides local respective register space for the execution lanes. The executing of the convolutional neural network includes loading a plane of image data of a three-dimensional block of image data into the two-dimensional shift register.

Non-Patent Literature <NUM> discloses technologies related to Deep Neural Network, Convolution Neural Network, and use of the Winograd transform to significantly boost the performance of FPGA.

As a result of detailed examination by the inventor, in the conceivable convolution computation device, since the convolution operation is converted into the matrix operation, it is necessary to convert the input data so that the matrix operation can be performed, and since it is necessary to handle the converted data in which the original input data is overlapped, the difficulty that the hardware and data processing increase and the power consumption increases has been found.

An object of the present disclosure is to provide a convolutional computation device with reducing power consumption.

This object is solved by the subject matter of claim <NUM>. Further aspects of the invention are disclosed in the dependent claims.

According to this, the power consumption in the convolutional computation device is reduced.

The above and other objects, features and advantages of the present disclosure will become more apparent from the below-described detailed description made with reference to the accompanying drawings. In the present description, the first and the second embodiments and their potential modifications relate to unclaimed combinations of features, which are nevertheless useful to understand the invention, whereas the third embodiment relates to a claimed combination of features. In the drawings:.

A first embodiment of the present disclosure will be described below with reference to the drawings of <FIG>.

The convolutional computation circuit is used for a data flow processor (Data Flow Processor, hereinafter referred to as "DFP") for image recognition and object detection in an autonomous driving system.

In the convolutional computation circuit of the present embodiment, in the shift register <NUM>, data elements are stored in a large number of storage elements <NUM> arranged two-dimensionally, and the data elements in the storage elements <NUM> within the input window <NUM> set in a predetermined area is selected as the input data d. In the multiplier-accumulator <NUM>, the input data d input from the shift register <NUM> and the weight data g forming a predetermined filter are processed by the multiply-accumulate operation to generate output data Y. Here, the total data elements are sequentially shifted in the shift register <NUM>, the data elements in the input window <NUM> are sequentially selected as the input data d, and the total output data Z is generated from the output data Y sequentially generated from the input data d.

The DFP system <NUM> will be outlined with reference to <FIG>.

In the DFP system <NUM> of the present embodiment, the DFP <NUM> functions as an individual master for handling the heavy calculation load of the host CPU <NUM>, can execute a program and an original fetch instruction, and supports an interruption process generated by the event handler <NUM>. The DFP <NUM>, the host CPU <NUM>, the ROM <NUM>, the RAM <NUM>, and the external interface <NUM> transmit and receive data via the system bus <NUM>.

The DFP11 will be outlined with reference to <FIG>.

With regard to the DFP11, it is possible to execute a plurality of threads in parallel even for different instruction streams by dynamically allocating registers and thread scheduling by hardware for a large number of threads. It is possible to generate such a large number of threads by automatically vectorizing the program code by the compiler and extracting the graph structure that maintains the task parallelism and graph parallelism of the program.

In the DFP11, the plurality of execution cores <NUM> each have a large number of pipelines independently scheduleable, and share resources among the four processing elements PE # <NUM>, PE # <NUM>, PE # <NUM>, and PE # <NUM>. The thread scheduler <NUM> realizes scheduling across a large number of threads and executes a large number of threads at the same time. The command unit <NUM> sends and receives data to and from the configuration interface <NUM>, and functions as a command buffer. The memory sub system <NUM> is formed by an arbiter <NUM>, an L1 cache 34a, and an L2 cache 34b, and transmits / receives data to / from the system bus interface <NUM> and the ROM interface <NUM>.

In the present embodiment, the convolutional computation circuit is used as one execution core <NUM> among the plurality of execution cores <NUM> included in the DFP <NUM>.

The convolutional computation circuit will be described with reference to <FIG>.

The convolutional computation circuit is formed by a two-dimensional circulation shift register <NUM> and a multiplier-accumulator <NUM>.

In the shift register <NUM>, a large number of storage elements <NUM> are arranged two-dimensionally. In this embodiment, <NUM> storage elements <NUM> arranged in <NUM> by <NUM> having the zero-th row to the seventh row and the zero-th column to the seventh column.

Data elements are stored in each storage element <NUM>. Each storage element <NUM> is connected to four storage elements <NUM> arranged vertically and horizontally of the storage element <NUM>, and the data element of each storage element <NUM> can be shifted to the storage elements <NUM> arranged vertically and horizontally. The shift register <NUM> is a two-dimensional circulation type, and the storage elements <NUM> disposed on the bottom column, the top column, the right end row and the left end row is arranged on an up side, a down side, a left side and a right side of the storage elements <NUM> disposed on the top column, the bottom column, the left end row, and the right end row, respectively. Here, when each storage element <NUM> is connected to n adjacent storage elements <NUM> and, when the data element is shifted between the storage element <NUM> and n adjacent storage elements <NUM>, the shift amount is defined as n. In the present embodiment, each storage element <NUM> is connected only to the storage elements <NUM> adjacent to each other vertically and horizontally, and the data element of each storage element <NUM> can be shifted only to the storage elements <NUM> adjacent to each other vertically and horizontally, so that the shift amount is defined as <NUM>.

A memory interface <NUM> is connected to the shift register <NUM>. Data elements are sequentially input from the memory interface <NUM> to each storage element <NUM> in the bottom row of the shift register <NUM>. By sequentially shifting the data elements, input to each storage element <NUM> in the bottom row, upward, so that the data elements are stored in all the storage elements <NUM>.

In the shift register <NUM>, the input window <NUM> is set in a predetermined area. The data element stored in the storage element <NUM> in the input window <NUM> is selected as the input data d and outputs to the multiplier-accumulator <NUM>. In the present embodiment, the input window <NUM> is set in the area including the storage elements <NUM> in the <NUM> by <NUM> matrix having the <NUM>-th row to the second row and the <NUM>-th column to the second column. Then, the input data d with of <NUM> rows and <NUM> columns is selected from the total data elements of <NUM> rows and <NUM> columns, and the input data d of <NUM> rows and <NUM> columns is output to the multiplier-accumulator <NUM>.

The storage element <NUM> of the shift register <NUM> will be described in detail with reference to <FIG>.

The storage element <NUM> is formed of a multiplexer <NUM> (MUX) and a flip-flop <NUM> (FF). In the present embodiment, data elements are input to the multiplexer <NUM> from flip-flops <NUM> of storage elements <NUM> adjacent to each other in the vertical and horizontal directions. The multiplexer <NUM> selects one data element from the four input data elements and outputs it to the flip-flop <NUM> of the storage element <NUM>. The flip-flop <NUM> holds the data element input from the multiplexer <NUM>.

The multiplier-accumulator <NUM> will be described with reference to <FIG>.

The multiplier-accumulator <NUM> performs a multiply-accumulate operation of the input data d and the weight data w forming a predetermined filter to generate the output data Y. That is, as shown by the following equation (<NUM>), the Frobenius product of the input data d and the weight data g is defined as the output data Y. Here, dqr is an input data element forming the input data d, and gqr is a weight data element forming the weight data g. Equation <NUM> <MAT>.

In the present embodiment, according to the input data d (dqr: q = <NUM> to <NUM>; r = <NUM> to <NUM>) in <NUM> rows by <NUM> columns and the weight data g (gqr: q = <NUM> to <NUM>; r = <NUM> to <NUM>) in <NUM> rows by <NUM> columns, the output data Y of <NUM> row by <NUM> column is generated. Therefore, the number of multiplications in the multiply-accumulate operation is nine.

The multiplier-accumulator <NUM> includes an input register <NUM>, a weight register <NUM>, a multiplier <NUM>, and an adder tree <NUM>. The input register <NUM> holds an input data element dqr that forms the input data d input from the shift register <NUM>. The weight register <NUM> holds a weight data element gqr that forms the weight data g input from an interface (not shown). In this embodiment, the input register <NUM> and the weight register <NUM> each include nine storage areas. Each multiplier <NUM> multiplies each input data element dqr of the input register <NUM> with each weight data element gqr of the weight register <NUM>, and the adder tree <NUM> calculates the total multiplication result calculated by each multiplier <NUM>. In this embodiment, nine multipliers <NUM> are used, and nine multiplier results are added by the adder tree <NUM>.

The convolutional computation process of the present embodiment will be described with reference to <FIG>.

As shown in <FIG>, in the shift register <NUM>, the total data elements are sequentially shifted, and the data elements in the input window <NUM> are sequentially selected as the input data d. In the present embodiment, the total data elements ij (i = <NUM> to <NUM>; j = <NUM> to <NUM>) of <NUM> rows by <NUM> columns are sequentially shifted, and the data elements in the input window <NUM> having <NUM> by <NUM> matrix with the 0th row to the 2nd row and the 0th column to the 2nd column are sequentially selected as the input data d.

As shown in <FIG>, the selection of the input data d by the shift operation in the shift register <NUM> corresponds to the movement of the total data elements in the input data range D. Here, the shift operation in the up / down / left / right directions in the shift register <NUM> corresponds to the movement of the total data elements in the input data range D in the opposite direction. In the following, the input data of the input data range D displaced downward and to the right by k and l with respect to the upper left as a reference is defined as dkl, and the output data generated from the input data dkl is defined as Ykl. In the present embodiment, in the total data element ij (i = <NUM> to <NUM>; j = <NUM> to <NUM>) in the <NUM> rows and <NUM> columns matrix, the input data range D in the <NUM> rows and <NUM> columns matrix is displaced by k and l in the downward direction and the right direction, respectively(k = <NUM> to <NUM>; l = <NUM> to <NUM>).

As shown in <FIG>, in a series of shift operations, a shift operation with a shift amount of <NUM> and the numerical number of shift times of <NUM> is performed in one cycle from input data selection to the next input data selection, and <NUM> shift operations are performed in <NUM> cycles. By repeating the <NUM> cycles including before the start of the cycle, <NUM> input data dkl in the <NUM> rows and <NUM> columns matrix are selected, and <NUM> output data Ykl in the <NUM> row and <NUM> column matrix are generated based on <NUM> input data dkl in the <NUM> rows and <NUM> columns matrix. Then, the total output data Z in the <NUM> rows and <NUM> columns matrix having <NUM> output data Ykl as the element zkl is generated.

In the present embodiment, the number of shifts in the convolution operation is <NUM> times. Further, the total number of multiplications is <NUM> times by multiplying <NUM> times, which is the number of multiplications in the product-sum operation, and <NUM> times, which is the number of times of input data selection.

The convolutional computation circuit of the present embodiment has the following effects.

In the convolutional operation circuit of the present embodiment, since the convolutional operation is not converted into the matrix operation, it is not necessary to convert the input data so that the matrix operation can be performed, and it is not necessary to handle the converted data which is prepared by duplicating the original input data. Therefore, the increase in hardware and data processing is avoided, and the power consumption in the convolutional computation device is reduced.

Hereinafter, a first modification of the first embodiment of the present disclosure will be described.

In the convolutional computation circuit of this modification, a plurality of input window areas can be switched with each other in the shift register <NUM>. That is, all the storage elements <NUM> included in the plurality of input window areas are connected to the multiplexer, and by selecting the input data elements input from the storage elements <NUM> in the multiplexer, the plurality of input window areas can be switched to each other.

For example, when a filter of the <NUM> rows <NUM> columns matrix, a filter of the <NUM> rows <NUM> columns matrix, or a filter of the <NUM> rows <NUM> columns matrix is used in the convolutional operation, the 0th to 2nd input window areas including the storage elements <NUM> in the <NUM> rows <NUM> columns matrix, the <NUM> rows <NUM> columns matrix, or the <NUM> rows <NUM> columns matrix corresponding to filters can be switched therebetween. As such 0th to 2nd input window areas, for example, (<NUM>) the zero-th input window area including the storage elements <NUM> in the <NUM> by <NUM> matrix from the 0th row to 2nd row and the 0th column to the 2nd column, (<NUM>) the 1st input window area including the storage elements <NUM> in the <NUM> by <NUM> matrix from the 0th row to 3rd row and the Oth column to the 3rd column, and (<NUM>) the 2nd input window area including the storage elements <NUM> in the <NUM> by <NUM> matrix from the 0th row to 4th row and the 0th column to the 4th column are available.

Regarding the convolutional computation circuit of this modification, since a plurality of input window areas can be switched with each other in the shift register <NUM>, it is possible to handle various types of convolutional operations, and a highly versatile convolutional computation circuit is provided.

Hereinafter, a second modification of the first embodiment of the present disclosure will be described.

In the convolutional computation circuit of this modification, the shift amount can be switched in the shift register <NUM>. That is, the flip-flops <NUM> of the storage elements <NUM> disposed s1 storage elements away, s2 storage elements away,. , and sn storage elements away therefrom are connected to the multiplexer <NUM> of each storage element <NUM> in each of the up, down, left, and right directions. Here, n is equal to or larger than two. Then, in the multiplexer <NUM> of each storage element <NUM>, the shift amount can be switched by changing the selection of the data element input from the flip-flop <NUM> in the storage element <NUM> disposed s1 storage elements away, s2 storage elements away,. , and sn storage elements away therefrom. Here, n is equal to or larger than two.

For example, in the shift register <NUM>, in addition to the flip-flops <NUM> of the adjacent storage elements <NUM>, the flip-flop <NUM> of the storage element <NUM> disposed two storage elements away from the storage element is connected to the multiplexer <NUM> of the storage element <NUM> in each of the up, down, left, and right directions. In the multiplexer <NUM> of each storage element <NUM>, the shift amount is changed between one and two by switching which data element is selected to be input from the adjacent storage element or the storage element disposed two storage elements away from the storage element.

In the convolutional computation circuit of this modification, since the shift amount can be switched in the shift register <NUM>, it is possible to handle various types of convolution calculations, and a highly versatile convolutional computation circuit is realized.

Further, by combining the first modification and the second modification of the first embodiment to switch the input data area and the shift amount in the shift register <NUM>, a more versatile convolutional computation circuit can be obtained.

A second embodiment of the present disclosure will be described below with reference to the drawings of <FIG>.

In the convolutional computation circuit of the present embodiment, a plurality of input windows 44a-44d are set in the shift register <NUM>.

In <FIG>, for the sake of simplification of the drawings, the signal lines are not shown between the storage elements <NUM> between the top row and the bottom row, and between the storage elements <NUM> between the leftmost column and the rightmost column.

As shown in <FIG>, in the shift register <NUM> of the present embodiment, the 0th to 3rd input windows 44a-44d are set. As the 0th to 3rd input windows 44a-44d, (<NUM>) the 0th input window 44a including the storage elements <NUM> in the <NUM> rows and <NUM> columns matrix with the 0th row to the 2nd row and the 0th column to the 2nd column, (<NUM>) the first input window 44b including the storage elements <NUM> in the <NUM> rows and <NUM> columns matrix with the 0th row to the 2nd row and the third column to the 5th column, and (<NUM>) the second input window 44c including the storage elements <NUM> in the <NUM> rows and <NUM> columns matrix with the third row to the 5th row and the 0th column to the 2nd column, and (<NUM>) the third input window 44d including the storage elements <NUM> in the <NUM> rows and <NUM> columns matrix with the third row to the 5th row and the third column to the 5th column are available. The input data da to dd from the storage elements <NUM> in the 0th to 3rd input windows <NUM> are input to the 0th to 3rd multiplier-accumulator 50a to 50d, respectively, and output data Ya to Yd are generated in the 0th to 3rd multiplier-accumulator 50a to 50d.

In the convolutional computation process, a series of shift operations are executed in the shift register <NUM>. As shown in <FIG>, in a series of shift operations, a shift operation with a shift amount of <NUM> and the numerical number of shift times of <NUM> is performed in one cycle from input data selection to the next input data selection, and <NUM> shift operations are performed in <NUM> cycles. By executing the eight cycles, the input data dm n, dm <NUM> + n, d<NUM> + m n, d<NUM> + m <NUM> + n, each of which has nine <NUM> rows and <NUM> columns matrixes, are selected by the 0th to 3rd input windows 44a to 44d, in addition to before the start of the cycle (m = <NUM> to <NUM>; n = <NUM> to <NUM>), and nine output data Ym n, Ym <NUM> + n, Y<NUM> + m n, Y<NUM> + m <NUM> + n, each of which has <NUM> row and <NUM> column matrix, are generated from the input data dm n, dm <NUM> + n, d<NUM> + m n, d<NUM> + m <NUM> + n, each of which has nine <NUM> rows and <NUM> columns matrixes. Then, as in the first embodiment, the total output data Z of <NUM> rows and <NUM> columns matrix having <NUM> output data Ykl as the element zkl is generated.

In the present embodiment, the number of shifts in the convolution operation is <NUM> times. Further, the total number of multiplications is <NUM> times by multiplying <NUM> times, which is the number of multiplications in the multiply-accumulate operation, <NUM> times, which is the number of input windows 44a to 44d, and <NUM> times, which is the number of selection times of input data.

In the convolutional computation circuit of the present embodiment, since a plurality of input windows 44a to 44d are set in the shift register <NUM>, the same output is acquired with a smaller number of shift operations as compared with the case where a single input window <NUM> is set. Thus, the convolutional operation can be executed at high speed.

A third embodiment of the present disclosure will be described below with reference to the drawings of <FIG>.

The convolutional computation circuit of the present embodiment performs a multiply-accumulate calculation based on the Winograd algorithm, selects input data of <NUM> rows and <NUM> columns matrix for a filter of <NUM> rows and <NUM> columns matrix, and generates the output data of <NUM> rows and <NUM> columns matrix.

In <FIG>, similar to <FIG>, for the sake of simplification of the drawings, the signal lines are not shown between the storage elements <NUM> between the top row and the bottom row, and between the storage elements <NUM> between the leftmost column and the rightmost column.

As shown in <FIG>, in the shift register <NUM>, an input window <NUM> including the storage elements <NUM> of the 0th row to the 4th row and the 0th column <NUM> to the 4th column in the <NUM> rows and <NUM> columns matrix is set.

The multiplier-accumulator <NUM> performs a multiply-accumulate calculation of the input data d and the weight data g based on the Winograd algorithm, as shown in the following equation (<NUM>). Here, G, B, and A are constant matrices. Equation (<NUM>)<MAT>.

In the present embodiment, the output data Y of <NUM> rows and <NUM> columns matrix is generated by the multiply-accumulate calculation of the input data d of <NUM> rows and <NUM> columns matrix and the weight data g of <NUM> rows and <NUM> columns matrix. The constant matrixes G, B, and A are as shown in <FIG>. The weight term GgGT is calculated in advance. Therefore, in the present embodiment, the number of multiplications in the multiply-accumulate operation is <NUM> times. Further, as shown in <FIG>, the constant matrices B and A have one of the elements <NUM>, <NUM>, <NUM>, and <NUM>, and the multiplication operation can be executed only by the bit shift operation and the addition operation.

In the present embodiment, as shown in <FIG>, in the total data element ij (i = <NUM> to <NUM>; j = <NUM> to <NUM>) of <NUM> rows and <NUM> columns matrix, the input data range D of <NUM> rows and <NUM> columns matrix is displaced by k and l (k = <NUM>,<NUM>; l = <NUM>,<NUM>) in the right direction and the down direction, respectively.

As shown in <FIG>, in a series of shift operations, a shift operation with a shift amount of <NUM> and the numerical number of shift times of <NUM> is performed in one cycle from input data selection to the next input data selection, and <NUM> shift operations are performed in <NUM> cycles. By repeating the <NUM> cycles in addition to before the start of the cycle, <NUM> input data dkl in the <NUM> rows and <NUM> columns matrix are selected, and <NUM> output data Ykl in the <NUM> row and <NUM> column matrix are generated based on <NUM> input data dkl in the <NUM> rows and <NUM> columns matrix. Then, a total output data Z of <NUM> rows and <NUM> columns matrix is generated in which the output data element yklop (o = <NUM> to <NUM>; p = <NUM> to <NUM>) in the output data Ykl, each of which has <NUM> rows and <NUM> columns matrix, is the element zk + o l + p.

In the convolutional computation circuit of the present embodiment, the number of multiplications and the number of shifts can be significantly reduced by performing the multiply-accumulate calculation based on the Winograd algorithm. In addition, by selecting the input data of <NUM> rows and <NUM> columns matrix for the filter of <NUM> rows and <NUM> columns matrix and generating the output data of <NUM> rows and <NUM> columns matrix, the multiplication operation can be performed only by the bit shift operation and the addition operation. Therefore, it is possible to execute the convolutional operation at sufficiently high speed and with low power consumption.

Hereinafter, a first modification of the third embodiment will be described.

Regarding the convolutional computation circuit of this modification, the shift amount is set to <NUM> in the shift register <NUM> of the third embodiment. As described in the first embodiment, each storage element <NUM> is connected to three adjacent storage elements <NUM>, and the data element is shifted between each storage element <NUM> and the three adjacent storage elements <NUM>. Thus, it is possible to realize a shift with a shift amount of <NUM>. In the shift register <NUM>, a shift operation with a shift amount of <NUM> and a number of shift times of <NUM> is performed in one cycle from input data selection to the next input data selection, and three shifts are executed in three cycles. Therefore, the number of shifts in this modified example is three.

As described above, in this modification, the number of shifts in the convolutional operation is further reduced, and the convolutional operation can be executed at higher speed and lower power consumption.

Hereinafter, a second modification of the third embodiment will be described.

Regarding the convolutional computation circuit of this modification, the shift register <NUM> of the third embodiment is capable of switching between the 0th to 3rd input window areas instead of the shift operation. As the 0th to 3rd input window areas, (<NUM>) the 0th input window area including the storage elements <NUM> in the <NUM> rows and <NUM> columns matrix with the 0th row to the 4th row and the 0th column to the 4th column, (<NUM>) the first input window area including the storage elements <NUM> in the <NUM> rows and <NUM> columns matrix with the 0th row to the 4th row and the third column to the 7th column, and (<NUM>) the second input window area including the storage elements <NUM> in the <NUM> rows and <NUM> columns matrix with the third row to the 7th row and the 0th column to the 4th column, and (<NUM>) the third input window area including the storage elements <NUM> in the <NUM> rows and <NUM> columns matrix with the third row to the 7th row and the third column to the 7th column are available. Then, the 0th to 3rd input window areas are sequentially switched to select input data, sequentially output data is generated from the input data, and total output data is generated from the output data.

As described above, in this modification, by switching the input window area in the convolutional operation, the shift operation is unnecessary, and the convolutional operation can be executed at higher speed and lower power consumption.

Claim 1:
A convolutional computation device comprising:
a two-dimensional circulation shift register unit (<NUM>) that has a plurality of storage elements (<NUM>) arranged two-dimensionally and respectively storing data, and that is configured to cyclically shift the data among the plurality of storage elements, to provide at least one input window (<NUM>, 44a to 44d) in a predetermined area, and to select the data stored in one of the storage elements disposed in the input window as input data; and
at least one multiplier-accumulator (<NUM>, 50a to 50d) that is configured to generate output data (Y, Ya to Yd) by performing a multiply-accumulate operation on the input data input from the two-dimensional circulation shift register unit and weight data for providing a predetermined filter,
wherein:
the multiplier-accumulator is configured to perform the multiply-accumulate operation based on a Winograd algorithm,
characterized in that:
the filter is a three rows and three columns matrix filter;
the input data is a <NUM> rows and <NUM> columns matrix input data;
the output data is a <NUM> rows and <NUM> columns matrix output data;
the multiplier-accumulator is configured to perform the multiply-accumulate operation only by a bit shift operation and an addition operation; and
the Winograd algorithm is shown in an equation of <MAT> wherein
Y is an output data Y;
G, B, and A are constant matrices;
g is a weight data;
d is an input data;
GgGT is a weight term calculated in advance; and
the constant matrices of B and A have one of the elements <NUM>, ±<NUM>, ±<NUM>, ±<NUM>, and ±<NUM>.