NEURAL NETWORK COMPUTING METHOD AND NEURAL NETWORK COMPUTING DEVICE

A neural network computing method and a neural network computing device are provided. The neural network computing method includes the following steps. At least one chosen layer is decided. A plurality of front layers previous to the chosen layer are decided. A selected element is selected from a plurality of chosen elements in the chosen layer. A front computing data group related to the selected element is defined. The front computing data group is composed of only part of a plurality of front elements in the front layers. The selected element is computed according to the at least one front computing data group.

TECHNICAL FIELD

The disclosure relates in general to a computing method and a computing device, and more particularly to a neural network computing method and a neural network computing device.

BACKGROUND

For the neural network (NN) computation, the size of the intermediate data reflects the required memory size. The larger or enough capacity of the working memory (e.g., SRAM, DRAM, register) would improve the efficiency of the NN calculation. However, the larger working capacity consumes more die area and increases the cost.

In the ordinary computation sequence of the neural network model, the data (input and weight values of each layer) should be calculated layer by layer. The whole intermediate data of a certain layer would be calculated and stored to the working memory. The required working memory capacity is quite large.

SUMMARY

The disclosure is directed to a neural network computing method and a neural network computing device. The data in the neural network could be computed via front computing data, and only part of the front elements in any one of the front layers are needed to be stored. Therefore, the memory usage is greatly reduced, and the memory area as well as the cost can be decreased.

According to one embodiment, a neural network computing method is provided. The neural network computing method includes the following steps. At least one chosen layer is decided. A plurality of front layers previous to the chosen layer are decided. A selected element is selected from a plurality of chosen elements in the chosen layer. A front computing data group related to the selected element is defined. The front computing data group is composed of only part of a plurality of front elements in the front layers. The selected element is computed according to the at least one front computing data group.

According to another embodiment, a neural network computing device is provided. The neural network computing device includes a deciding unit, a selecting unit, a defining unit and a computing unit. The deciding unit is configured to decide at least one chosen layer and decide a plurality of front layers previous to the chosen layer. The selecting unit is configured to select a selected element from a plurality of chosen elements in the chosen layer. The defining unit is configured to define a front computing data group related to the selected element. The front computing data group is composed of only part of a plurality of front elements in the front layers. The computing unit is configured to compute the selected element according to the at least one front computing data group.

According to an alternative embodiment, a neural network computing method is provided. The neural network computing method includes the following steps. At least one chosen layer is decided. A plurality of front layers previous to the chosen layer are decided. More than one selected elements from a plurality of chosen elements in the chosen layer are selected. More than one front computing data groups related to the selected elements are defined. Each of the front computing data groups is composed of only part of a plurality of front elements in the front layers. The selected elements are computed according to the more than one front computing data groups.

DETAILED DESCRIPTION

Please refer toFIGS.1A to1C, which illustrate how the neural network is calculated via a “layer-by-layer” approach. As shown inFIG.1A, the neural network has, for example, multiple convolutional layers Ly1to Ly4. In the “layer by layer” approach, all elements on the convolutional layer Ly1need to be stored (the slash indicates the part that needs to be stored) before any element in the convolutional layer Ly2is calculated. After all the elements on the convolutional layer Ly2are calculated, the process will enters the calculation ofFIG.1B.

InFIG.1B, all elements in the convolutional layer Ly2need to be stored (the slash indicates the part that needs to be stored) before any element in the convolutional layer Ly3is calculated. After all the elements on the convolutional layer Ly3are calculated, the process will enter the calculation ofFIG.1C.

InFIG.1C, all elements in the convolutional layer Ly3need to be stored (the slash indicates the part that needs to be stored) before any element in the convolutional layer Ly4is calculated.

As mentioned above, in the “layer by layer” approach, it is necessary to store the entire convolutional layer Ly1, Ly2, Ly3, or Ly4each time, consuming a large amount of memory capacity.

Please further refer toFIG.2, which illustrates how the neural network is calculated via a “multi-layer jump” approach. As shown inFIG.2, the elements in the range R1iin the convolutional layer Ly1can be used to calculate all of the elements in the range R2iin the convolutional layer Ly2. The elements in the range R2iin the convolutional layer Ly2can be used to calculate all of the elements in the range R3iin the convolutional layer Ly3. The elements in the range R3iin the convolutional layer Ly3can be used to calculate all of the elements in the range R4iin the convolutional layer Ly4. That is to say, a certain element E4iin the convolutional layer Ly4can be calculated by the range R1iof the convolutional layer Ly1, the range R2iof the convolutional layer Ly2, and the range R3iof the convolutional layer Ly3. After each element in the convolutional layer Ly4is calculated one by one, the calculation of the convolutional layer Ly4can also be completed. Therefore, using this “multi-layer jump” approach, only the range R1iof the convolutional layer Ly1, the range R2iof the convolutional layer Ly2, and the range R3iof the convolutional layer Ly3need to be stored for calculating an element of E4iin the convolutional layer Ly4.

Please refer toFIG.3, which illustrates a neural network computing method according to one embodiment. In the present embodiment, the neural network is not computed via “layer-by-layer” approach. Instead, via the “multi-layer jump” approach, several front layers L1, L2previous to one chosen layer L3are used to compute each of a plurality of chosen element E3iin the chosen layer L3.

For example, one selected element E31is selected from the chosen elements E3iin the chosen layer L3. Due to the size of the filter layer F2, it needs to pick up p*q*C front elements E2iin the front layer L2. Due to the size of the filter layer F1, it needs to pick up (p+m−1)*(q+n−1)*E front elements E1iin front layer L1.

Further, one selected element E32is selected from the chosen elements E3iin the chosen layer L3. Due to the size of the filter layer F2, it needs to pick up p*q*C front elements E2iin the front layer L2. Due to the size of the filter layer F1, it needs to pick up (p+m−1)*(q+n−1)*E front elements E1iin front layer L1.

That is to say, to compute each of the chosen elements E3iin the chosen layer L3, only (p+m−1)*(q+n−1)*E front elements E1iare needed to be stored. On the contrary, if the front layer L1and the front layer L2are computed layer by layer, H*W*E front elements E1iare needed to be stored. It is clear that (p+m−1)*(q+n−1)*E is much less than H*W*E. Comparing to the computation in way of layer by layer, the usage of the memory in the present embodiment using the “multi-layer jump” approach is greatly reduced.

Please refer toFIG.4, which illustrates front computing data of one chosen element. For a chosen element Ej+2 in one chosen layer Lj+2, that might find some related front elements Ej+i in the previous front layer Lj+1 due to the filter and convolution condition. The required front elements Ej+1 in the previous front layer Lj+1 are defined as the front computing data VDj+1 of the chosen element Ej+2.

All front computing data VDj+1, VDj in the front layers Lj+1, Lj in front of the chosen element Ej+2 are defined as the front computing data group Gj+2.

For the more layer difference, the front computing data usually becomes wider for a certain chosen element. For example, the front computing data VDj has larger size than the front computing data VDj+1. In one embodiment, the number of the front layers could be more than two.

The size of the front computing data depends on the neural network configuration, e.g., the filter size, stride step size, layer depth, pooling operation, and layer number, etc.

Please refer toFIG.5, which illustrates the front computing data in one dimension. For a chosen element Ek+6 in one chosen layer Lk+6, the chosen element Ek+6 might be computed through two 3×1 convolutions, one 3×1 convolution with stride at2, one 3×1 convolution, one 2×1 polling, and one 3×1 convolution. The related front computing data group Gk+6 can be found through the neural network structure. The front computing data group Gk+6 of different chosen elements Ek+6 can be calculated in parallel, which might improve the computation performance.

Please refer to table I, which uses the neural network VGG16 as an example. If the data is computed layer by layer, the maximum required intermediate data size is 3136K for the ordinary VGG16 model. If the data is computed via front computing data, for 1×1×256 chosen elements after the “Pooling 3” layer, the required intermediate data size is 396K and the required vision data size is 110.25K. The total data size is 506.25K (396K+110.25K) and it results about 0.16× of reduction of the memory usage.

Please refer to table II, which uses the neural network ResNET18 as an example. If the data is computed layer by layer, the maximum required intermediate data size is 784K for the ordinary ResNET18 model. If the data is computed via front computing data, for 1×1×256 chosen elements after the “Layer 10” layer, the required intermediate data size is 49K and the required front computing data size is 49K. The total data size is 98K (48K+48K) and it results about 0.125× of reduction of the memory usage.

Please refer to table III, which uses the neural network ResNET18 as another example. If the data is computed layer by layer, the maximum required intermediate data size is 784K for the ordinary ResNET18 model. If the data is computed via front computing data, for 1×1×128 chosen elements after the “Layer 6” layer, the required intermediate data size is 98K and the required front computing data size is 7.563K. The total data size is 105.563K (98K+7.563K) and it results about 0.135× of reduction of the memory usage.

According to the examples shown above, the neural network computed via front computing data could result large reduction of the memory usage. Please refer toFIG.6, which show a neural network computing device100according to one embodiment. For computing the data in the neural network via front computing data, the neural network computing device100is provided. The neural network computing device100is, for example, a computer, a chip, a circuit, a circuit board, program codes or a storage device storing program codes. The neural network computing device100includes a deciding unit110, a selecting unit130, a defining unit140, a computing unit150, a determining unit160and a storing unit170. The storing unit170is used for storing data. For example, the storing unit170could be a memory, a disk or a storage cloud. The deciding unit110, the selecting unit130, the defining unit140, the computing unit150and the determining unit160are used to perform various computer operations. For example, the deciding unit110, the selecting unit130, the defining unit140, the computing unit150and the determining unit160could be a chip, a circuit, a circuit board, program codes or a storage device storing program codes. Through those components, the data in the neural network could be computed via front computing data. The operations of those components are illustrated through a flowchart as below.

Please refer toFIG.7, which shows a flowchart of a neural network computing method according to one embodiment. The neural network computing method is used for a Visual Geometry Group (VGG) model, a residual network (ResNET) model, or a Binary neural network model. In step S110, as shown inFIG.3, the deciding unit110decides at least one chosen layer (e.g., the chosen layer L3). In this step, the deciding unit110may decide the chosen layer according to the required front computing data size. To minimum the required front computing data size, more than one chosen layers may be decided. After one chosen layer is computed, the next one chosen layer is computed.

Then, in step S120, as shown inFIG.3, the deciding unit110decides a plurality of front layers (e.g., the front layers L1, L2) previous to the chosen layer (e.g., the chosen layer L3). In this step, the layers between the previous chosen layer (or the input) and this chosen layer are decided as the front layers.

Next, in step S130, as shown inFIG.3, the selecting unit130selects a selected element (e.g., the selected element E31), from a plurality of chosen elements (e.g., the chosen elements E3i) in the chosen layer (e.g., the chosen layer L3). In this step, all of the chosen elements E3jin the chosen layer L3are selected one by one. After one selected element (e.g., the selected element E31) is computed, then another selected element (e.g., the selected element E32) is selected.

Afterwards, in step S140, as shown inFIG.4, the defining unit140defines a front computing data group (e.g., the front computing data group Gj+2) related to the selected element (e.g., the selected element Ej+2). The front computing data group Gj+2 is composed of only part of the front elements Ej+1 in the front layer Lj+1 and only part of the front elements Ej in the front layer Lj.

Next, in step S150, as shown inFIG.4, the computing unit150computes the selected element (e.g., the selected element Ej+2) according to the at least one front computing data group (e.g., the front computing data group Gj+2). In this step, only part of the front elements Ej+1 in the front layer Lj+1 are stored and only part of the front elements Ej in the front layer Lj are stored.

Then, in step S160, as shown inFIG.4, the determining unit160determines whether all of the chosen elements in the chosen layer (e.g., the chosen layer Lj+2) are selected and computed. If all of the chosen elements in the chosen layer are selected and computed, then the process proceeds to the step S170; if not all of the chosen elements in the chosen layer are selected and computed, then the process goes back the step S130. The step S130of selecting the selected element and the step S150of computing the selected element are performed repeatedly until all of the chosen elements in the chosen layer are selected and computed.

In step S170, the determining unit160determines whether all of the layers in the neural network are computed. If all of the layers in the neural network are computed, then the process terminated; if not all of the layers in the neural network are computed, then the process proceeds to step S110. The step S110of deciding the chosen layer and the step S120of deciding the front layers are performed repeatedly until all of the layers in the neural network are computed.

According to the neural network computing method, because only part of the front elements in any one of the front layers are needed to be stored, the memory usage is greatly reduced. Therefore, the memory area and the cost can be decreased. In some applications, such as speech recognition, the computation is not that timing-sensitive. The neural network computing method of the present embodiment is suitable for this application.

Not only the speech recognition application, some object detection applications might not require a high frame rate performance. For example, in the home security monitoring application, it only needs few frames per second to recognize the object and trigger the recording process. The neural network computing method of the present embodiment is suitable for this application.

In the steps S130to S150, only one selected element is computed at one time. In another embodiment, more than one selected elements could be computed in parallel. Please refer toFIG.8, which shows a flowchart of a neural network computing method according to another embodiment. In step S130′, as shown inFIG.9, the selecting unit130selects more than one selected elements (e.g., the selected elements Ej+2, Ej+2′) from a plurality of chosen elements in the chosen layer (e.g., the chosen layer Lj+2).FIG.9, which illustrates front computing data of more than one selected elements. In this step, more than one chosen elements in the chosen layer (e.g., the chosen layer Lj+2) are selected to be the selected elements (e.g., the selected elements Ej+2, Ej+2′). After the selected elements Ej+2, Ej+2′ are computed, then some of the remained chosen elements are selected to be selected elements.

Afterwards, in step S140′, as shown inFIG.9, the defining unit140defines more than one front computing data groups (e.g., the front computing data groups Gj+2, Gj+2′) related to the selected element (e.g., the selected elements Ej+2, Ej+2′). Each of the front computing data groups Gj+2, Gj+2′ is composed of only part of the front elements Ej+1 in the front layer Lj+1 and only part of the front elements Ej in the front layer Lj. The front computing data VDj+1, VDj of the front computing data group Gj+2 and the front computing data VDj+1′, VDj′ of the front computing data group Gj+2′ might be overlapped. The front computing data group Gj+2 and the front computing data group Gj+2′ could be stored in an identical memory, and the overlapped data could be stored at the same location.

Next, in step S150′, as shown inFIG.9, the computing unit150computes the more than one selected elements (e.g., the selected element Ej+2, Ej+2′) according to the more than one front computing data groups (e.g., the front computing data groups Gj+2, Gj+2′). In this step, because the front computing data group Gj+2 and the front computing data group Gj+2′ are overlapped, the computation for the selected elements Ej+2, Ej+2′ could share some of the stored front elements in the front layers. Therefore, the computation performance can be improved.

According to the embodiments described above, the data in the neural network could be computed via front computing data, and only part of the front elements in any one of the front layers are needed to be stored. Therefore, the memory usage is greatly reduced, and the memory area as well as the cost can be decreased.