Patent Publication Number: US-11645365-B2

Title: Dynamic multi-mode CNN accelerator and operating methods

Description:
This application claims the benefit of Taiwan application Serial No. 108140517, filed Nov. 7, 2019, the disclosure of which is incorporated by reference herein in its entirety. 
     TECHNICAL FIELD 
     The disclosure relates in general to a convolutional neural network (CNN) accelerator, and more particularly to a dynamic multi-mode CNN accelerator and operating method. 
     BACKGROUND 
     The CNN algorithm or model has become an important foundation in the development of computer vision. The operation of the CNN model comprises a large volume of convolutional operation and data transfer, and therefore will take a longer operation time. Particularly, the operation of the CNN model has high degrees of both parallelism and dependency. Therefore, it has become a prominent task for the industries to improve the hardware design of the accelerator to speed up the operation of the CNN model. 
     Firstly, the intermediate transient result of the CNN operation has a large data volume and a low re-utilization rate. The input and output of the large volume of data use high memory bandwidth of the computing device. As indicated in Table 1, due to the limited memory bandwidth, the traditional approach of increasing the computing speed of the CNN accelerator by simply increasing the number of convolutional cores, the improvement gained has gradually reached saturation. For example, when the memory bandwidth is limited, and the number of cores is increased to 2048 from 1024, the utilization rate of hardware computing capacity drops to 10% from 20%, and the frame rate CNN accelerator with 2048 cores remains at 10 fps (frames per second). 
     
       
         
           
               
             
               
                 TABLE 1 
               
             
            
               
                   
               
               
                 Less cores, More cores 
               
            
           
           
               
               
               
               
               
               
               
            
               
                   
                 Number of 
                   
                   
                   
                   
                   
               
               
                   
                 cores 
                 64 
                 256 
                 512 
                 1024 
                 2048 
               
               
                   
               
               
                   
                 Reference 
                 1 
                 1.3 
                 2.2 
                 4 
                 5.2 
               
               
                   
                 power 
                   
                   
                   
                   
                   
               
               
                   
                 consumption 
                   
                   
                   
                   
                   
               
            
           
           
               
            
               
                 Memory bandwidth 
               
               
                 BW(1 GB/s) when the memory bandwidth is limited 
               
            
           
           
               
               
               
               
               
               
               
            
               
                 Utilization 
                 Inception 
                 96%, 
                 61%, 
                 38%, 
                 20%, 
                 10%, 
               
               
                 rate and 
                 V4 
                  3 fps 
                  5 fps 
                  9 fps 
                  10 fps 
                  10 fps 
               
               
                 speed 
                 Mobilenet 
                 48%, 
                 13%, 
                 6.5%, 
                 3.2%, 
                 1.6%, 
               
               
                   
                 V2 
                 43 fps 
                  38 fps 
                  46 fps 
                  46 fps 
                  46 fps 
               
            
           
           
               
            
               
                 Memory BW 
               
               
                 (8 GB/s) when the memory bandwidth is sufficient 
               
            
           
           
               
               
               
               
               
               
               
            
               
                 Utilization 
                 Inception 
                 99%, 
                 97%, 
                 95%, 
                 89%, 
                 66%, 
               
               
                 rate and 
                 V4 
                  3 fps 
                  12 fps 
                  23 fps 
                  44 fps 
                  65 fps 
               
               
                 speed 
                 Mobilenet 
                 66%, 
                 53%, 
                 37%, 
                 21%, 
                 12%, 
               
               
                   
                 V2 
                 59 fps 
                 190 fps 
                 265 fps 
                 296 fps 
                 344 fps 
               
               
                   
               
            
           
         
       
     
     Secondly, the convolutional cores used in the CNN models have different lengths and widths, such as 1×7, 7×1, 1×3, 3×1, 3×3, 1×1, and the number of channels gradually increases in a monotonous manner. For example, the mobilenet model has both very deep channels (64˜512) and very shallow channels (the number of channels=1). As indicated in Table 2, even the accelerator architecture has the same number of cores, the accelerator operated in different models will provide different efficiencies and speeds. Therefore, it has become a prominent task for the industries to divide the CNN models and arrange an optimized process based on the number of cores and the operating model to increase the efficiency and computing speed of the accelerator. 
     
       
         
           
               
               
             
               
                   
                 TABLE 2 
               
             
            
               
                   
                   
               
               
                   
                 256 cores 
               
            
           
           
               
               
               
               
               
               
            
               
                   
                 Data 
                 Weight 
                 Volume of 
                 Number of 
                 Utilization 
               
               
                 Layer 
                 dimension 
                 size 
                 operation 
                 cycles 
                 rate 
               
               
                   
               
            
           
           
               
            
               
                 Model 1: Generic CNN Calculation 
               
            
           
           
               
               
               
               
               
               
               
            
               
                 CNN layer A 
                 448 × 448 × 3 
                 0.4 
                 KB 
                 193M 
                 4.39M 
                 8.60%     
               
               
                 CNN layer B 
                 56 × 56 × 64 
                 72 
                 KB 
                 465M 
                 1.04M 
                 87% 
               
               
                 CNN layer C 
                 28 × 28 × 128 
                 288 
                 KB 
                 464M 
                 0.97M 
                 93% 
               
               
                 CNN layer D 
                 14 × 14 × 256 
                 1.1 
                 MB 
                 463M 
                 0.95M 
                 95% 
               
               
                 CNN layer E 
                 7 × 7 × 512 
                 4.5 
                 MB 
                 463M 
                 2.41M 
                 38% 
               
            
           
           
               
            
               
                 Model 2: Image Mode CNN Calculation 
               
            
           
           
               
               
               
               
               
               
               
            
               
                 CNN layer G 
                 448 × 448 × 3 
                 0.4 
                 KB 
                 193M 
                  0.9M 
                 42% 
               
            
           
           
               
            
               
                 Model 3: DW Separable Mode CNN Calculation 
               
            
           
           
               
               
               
               
               
               
               
            
               
                 DW CNN 
                 56 × 56 × 128 
                 1.1 
                 KB 
                  3.6M 
                 113K 
                 13% 
               
               
                 layer H 
               
               
                 PW CNN 
                 56 × 56 × 128 
                 16 
                 KB 
                 51.4M  
                 200K 
                 98% 
               
               
                 layer I 
               
               
                   
               
               
                 DW = depth-wise; 
               
               
                 PW = point-wise 
               
            
           
         
       
     
     SUMMARY 
     The disclosure relates to a convolutional neural network (CNN) accelerator, and more particularly to a CNN accelerator comprising plural sub-accelerators featured by “identical processor structure” and “different numbers of cores”. The accelerator is configured to perform a convolutional neural network operation, which can be divided into plural sub-partial operations comprising a first sub-partial operation and a second sub-partial operation. The CNN accelerator comprises two sub-accelerators comprising a first sub-accelerator and a second sub-accelerator. The first sub-accelerator comprises I units of first CNN processor cores, J units of first element-wise &amp; quantize processors, and K units of first pool and nonlinear function processor. The second sub-accelerator comprises X units of second CNN processor cores, Y units of second element-wise &amp; quantize processors, and Z units of second pool and nonlinear function processor. In the present disclosure, the “identical processor structure” of the sub-accelerator refers to each sub-accelerator comprises a plural of CNN processor cores, at least one element-wise &amp; quantize processors and at least one pool and nonlinear function processor. The above variables I˜K, X˜Z all are natural numbers greater than 0, such that each sub-accelerator can independently perform convolutional neural network operation, reduce the need of data transmission with external memory and reduce the requirement of the memory bandwidth. In the present disclosure, the “different numbers of cores” of the sub-accelerator refers to the first sub-accelerator and the second sub-accelerator satisfying a relation of “different numbers of cores”, which refers to the establishment of at least one of the three relations, namely, “I is different from X”, “J is different from Y”, and “K is different from Z”. 
     According to one embodiment, the at least two sub-accelerators further comprise a third sub-accelerator comprising R units of first CNN processor cores, S units of first element-wise &amp; quantize processors, and T units of first pool and nonlinear function processor. The above variables R˜T all are natural numbers greater than 0. The first sub-accelerator and the third sub-accelerator satisfy a relation of “different numbers of cores”, which refers to the establishment of at least one of the three relations, namely, “I is different from R”, “J is different from S”, and “K is different from T”. The plural sub-partial operations further comprise a third partial convolutional neural network operation, which independently performs the third partial convolutional neural network operation. 
     According to one embodiment, the element-wise &amp; quantize processors are configured to process a scalar operation selected from a group of operations composed of addition, deduction, multiplication, division, batch normalization, quantization, bias and scaling. 
     According to one embodiment, the pool and nonlinear function processor are configured to process a non-linear activation operation selected from a group of operations composed of rectified linear unit (ReLU), Sigmoid function and Tan h function. 
     According to one embodiment, the convolutional neural network operation comprise the operation of T layers, the first sub-partial operation comprises the operation of the first M of the T layers, and the second sub-partial operation comprises the operation of the following N of the T layers. The first sub-accelerator performs the operation of the first M layers and then outputs an intermediate result to the second sub-accelerator, which then performs the operation of the following N layers according to the intermediate result. 
     According to one embodiment, the first sub-accelerator performs the first sub-partial operation for a first time interval T1, the second sub-accelerator performs the second sub-partial operation for a second time interval T2. When the first time interval T1 is substantially equivalent to the second time interval T2, then the time to complete overall CNN operation generally can be reduced. In practice, the first time interval T1 can be 80%˜120% of the second time interval T2. 
     According to one embodiment, the convolutional neural network operation comprises a trunk operation and a first branch operation. The first sub-partial operation comprises a trunk operation, and the second sub-partial operation comprises a first branch operation. The first sub-accelerator performs a trunk operation, and the second sub-accelerator performs a first branch operation. When the convolutional neural network operation finishes the trunk operation and intends to perform the first branch operation, the first sub-accelerator enters the first power saving mode, and the second sub-accelerator exits the second power saving mode. Afterwards, when the convolutional neural network operation finishes the first branch operation and intends to perform the trunk operation, the first sub-accelerator exits the first power saving mode, and the second sub-accelerator enters the second power saving mode. The convolutional neural network operation is a dynamic branch network operation. For example, the trunk operation can be a recognition operation of main image for recognizing a main factory object which is seen more frequently, and the first branch operation can be a recognition operation of scenario image for recognizing a customized object which is seen less frequently. Besides, the trunk operation can be a recognition operation of collision sound, and the branch operation can be an operation for recognizing and analyzing an audio source. 
     According to one embodiment, the convolutional neural network operation comprises a trunk operation, a first branch operation and a second branch operation. The first sub-partial operation comprises a trunk operation, and the second sub-partial operation selectively comprises one of a first branch operation and a second branch operation. The first sub-accelerator performs the trunk operation, and the second sub-accelerator selectively performs one of the first branch operation and the second branch operation. When the convolutional neural network operation intends to perform the first branch operation, the second sub-accelerator loads in a program code corresponding to the first branch operation. When the convolutional neural network operation intends to perform the second branch operation, the second sub-accelerator loads in a program code corresponding to the second branch operation. The convolutional neural network operation can be a dynamic tree network operation, the first branch operation can be a passenger behavior recognition, and the second branch operation can be a vehicle behavior recognition. 
     According to one embodiment, an operating method adaptable to a convolutional neural network operation (CNN) accelerator is provided. The CNN accelerator is configured to perform a convolutional neural network operation, which can be divided into plural sub-partial operations comprising a first sub-partial operation and a second sub-partial operation. The CNN accelerator comprises two sub-accelerators comprising a first sub-accelerator and a second sub-accelerator. The first sub-accelerator comprises I units of first CNN processor cores, J units of first element-wise &amp; quantize processors, and K units of first pool and nonlinear function processor. The second sub-accelerator comprises X units of second CNN processor cores, Y units of second element-wise &amp; quantize processors, and Z units of second pool and nonlinear function processor. The first sub-accelerator and the second sub-accelerator satisfy a relation of different numbers of cores, which refers to the establishment of at least one of the three relations, namely, “I is different from X”, “J is different from Y”, and “K is different from Z”. The operating method comprises the steps of: allocating the first sub-partial operation to the first sub-accelerator, and allocating the second sub-partial operation to the second sub-accelerator, wherein when the convolutional neural network operation finishes the first sub-partial operation, the first sub-accelerator enters a first power saving mode; when the convolutional neural network operation finishes the second sub-partial operation and intends to perform the first sub-partial operation, the first sub-accelerator exits the first power saving mode. 
     According to one embodiment, in the operating method, the first sub-partial operation comprises a trunk operation, and the second sub-partial operation comprises a first branch operation. The operating method further comprises the following steps of: when the convolutional neural network operation finishes the first branch operation and intends to perform the trunk operation, the second sub-accelerator enters a second power saving mode; when the convolutional neural network operation finishes the trunk operation and intends to perform the first branch operation, the second sub-accelerator exits the second power saving mode. 
     According to one embodiment, in the operating method, the first sub-partial operation comprises a trunk operation, and the second sub-partial operation selectively comprises one of a first branch operation and a second branch operation. The operating method further comprises the steps of: when the convolutional neural network operation finishes the trunk operation and intends to perform the first branch operation, the second sub-accelerator loads in a program code corresponding to the first branch operation; when the convolutional neural network operation finishes the trunk operation and intends to perform the second branch operation, the second sub-accelerator loads in a program code corresponding to the second branch operation. 
     According to another embodiment, an allocation method for inputting a convolutional neural network operation to a convolutional neural network operation (CNN) accelerator is provided. The convolutional neural network operation comprises the operation of T layers. The CNN accelerator comprises a first sub-accelerator and second sub-accelerator. The first sub-accelerator comprises I units of first CNN processor cores, J units of first element-wise &amp; quantize processors, and K units of first pool and nonlinear function processor. The second sub-accelerator comprises X units of second CNN processor cores, Y units of second element-wise &amp; quantize processors, and Z units of second pool and nonlinear function processor. The above variables I˜K, X˜Z all are natural numbers greater than 0. At least one of the three relations, namely, “I is different from X”, “J is different from Y”, and “K is different from Z”, is satisfied. The allocation method comprises two allocation arrangements, a first allocation arrangement and a second allocation arrangement. 
     According to the first allocation arrangement: the first sub-accelerator performs the operation of the first M layers and then outputs a first intermediate result to the second sub-accelerator, which then performs the operation of the following N layers, wherein the first sub-accelerator performs the operation of the first M layers for a first time interval T1, and the second sub-accelerator performs the operation of the following N layers for a second time interval T2. According to the second allocation arrangement, the first sub-accelerator performs the operation of the first A layers and then outputs a second intermediate result to the second sub-accelerator, which then performs the operation of the following B layers, that is, A+B=M+N, according to the second intermediate result, wherein the first sub-accelerator performs the operation of the first A layers for a third time interval T3, and the second sub-accelerator performs the operation of the following B layers for a fourth time interval T4. A ratio of the first time interval T1 to the second time interval T2 is compared with a ratio of the third time interval T3 to the fourth time interval T4. If the ratio of the first time interval T1 to the second time interval T2 is closer to 1 than the ratio of the third time interval T3 to the fourth time interval T4, then the first sub-accelerator and the second sub-accelerator perform an operation according to the first allocation arrangement. If the ratio of the third time interval T3 to the fourth time interval T4 is closer to 1 than the ratio of the first time interval T1 to the second time interval T2, then the first sub-accelerator and the second sub-accelerator perform an operation according to the second allocation arrangement. 
     According to another embodiment, an allocation method for inputting a convolutional neural network operation to a convolutional neural network operation (CNN) accelerator is provided. The convolutional neural network operation comprises the operation of T layers. The CNN accelerator comprises a first sub-accelerator and a second sub-accelerator. The first sub-accelerator comprises I units of first CNN processor cores, J units of first element-wise &amp; quantize processors, and K units of first pool and nonlinear function processor. The second sub-accelerator comprises X units of second CNN processor cores, Y units of second element-wise &amp; quantize processors, and Z units of second pool and nonlinear function processor. The above variables I˜K, X˜Z all are natural numbers greater than 0. At least one of the three relations, namely, “I is different from X”, “J is different from Y”, and “K is different from Z”. The allocation method comprises two allocation arrangements, a first allocation arrangement and a second allocation arrangement. According to the first allocation arrangement, the first sub-accelerator performs the operation of the first M layers and then outputs a first intermediate result to the second sub-accelerator, which then performs the operation of the following N layers according to the first intermediate result, wherein the first sub-accelerator consumes a first power E1 for performing the operation of the first M layers, and the second sub-accelerator consumes a second power E2 for performing the operation of the following N layers. According to the second allocation arrangement, the first sub-accelerator performs the operation of the first A layers and then outputs a second intermediate result to the second sub-accelerator, which then performs the operation of the following B layers, that is, A+B=M+N, according to the second intermediate result, wherein the first sub-accelerator consumes a third power E3 for performing the operation of the first A layers, and the second sub-accelerator consumes a fourth power E4 for performing the operation of the following B layers. A sum of the first power E1 plus the second power E2 is compared with a sum of the third power E3 plus the fourth power E4. If the sum of the first power E1 plus the second power E2 is lower than the sum of the third power E3 plus the fourth power E4, then the first sub-accelerator and the second sub-accelerator perform an operation according to the first allocation arrangement. If the sum of the third power E3 plus the fourth power E4 is lower than the sum of the first power E1 plus the second power E2, then the first sub-accelerator and the second sub-accelerator perform an operation according to the second allocation arrangement. 
     The above and other embodiments will become easier understood with regard to the following detailed description of these embodiments. The following description is made with reference to the accompanying drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    A is a hardware architecture diagram of a CNN accelerator. 
         FIG.  1    B is a hardware architecture diagram of the interior of each CNN sub-accelerator. 
         FIG.  2    is a schematic diagram of allocating operation of different layers to different sub-accelerators for performing pipeline parallel processing according to an embodiment of the disclosure. 
         FIG.  3    is a schematic diagram of allocating trunk and branch operations to different sub-accelerators for processing according to another embodiment of the disclosure. 
         FIG.  4    is a schematic diagram of allocating trunk and branch operations to different sub-accelerators for processing according to another embodiment of the disclosure. 
     
    
    
     In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. It will be apparent, however, that one or more embodiments may be practiced without these specific details. In other instances, well-known structures and devices are schematically shown in order to simplify the drawing. 
     DETAILED DESCRIPTION 
       FIG.  1    A is a hardware architecture diagram of a CNN accelerator  12 . The CNN accelerator  12  receives data from a dynamic task scheduler  10  and further works with an external shared memory  20  to perform CNN operation. The CNN accelerator  12  has at least two sub-accelerators disposed therein. As exemplified in  FIG.  1   , the CNN accelerator  12  comprises three CNN sub-accelerators (first CNN sub-accelerator  14 , second CNN sub-accelerator  16 , and third CNN sub-accelerator  18 ), but the number of CNN sub-accelerators is not limited thereto. Each sub-accelerator has a private memory. 
       FIG.  1 B  is a hardware architecture diagram of the interior of each CNN sub-accelerator  14 . The first CNN sub-accelerator  14 , the second sub-accelerator  16  and the third sub-accelerator  18  have an identical processor structure. That is, each of the three sub-accelerators comprises the three types of operation processors, namely, CNN processor cores  14 A, element-wise &amp; quantize processors  14 B and pool and nonlinear function processor  14 C. 
     The three sub-accelerators have different numbers of operation processors. 
     For example, the first sub-accelerator  14  comprises I units of first CNN processor cores, J units of first element-wise &amp; quantize processors, and K units of first pool and nonlinear function processor; the second sub-accelerator  16  comprises X units of second CNN processor cores, Y units of second element-wise &amp; quantize processors, and Z units of second pool and nonlinear function processor; the third sub-accelerator  18  comprises R units of first CNN processor cores, S units of first element-wise &amp; quantize processors, and T units of first pool and nonlinear function processor, wherein the above variables I˜K, R˜T, X˜Z all are natural numbers greater than 0. The feature of the first sub-accelerator  14  and the second sub-accelerator  16  comprising different number of cores refers to the establishment of at least one of the three relations, namely, “I is different from X”, “J is different from Y”, and “K is different from Z”. Similarly, the feature of the first sub-accelerator  14  and the third sub-accelerator  18  comprising different number of cores refers to the establishment of at least one of the three relations, namely, “A is different from R”, “B is different from S”, and “C is different from T”. 
     As indicated in  FIG.  1   , the first sub-accelerator  14  has more CNN processor cores, the second sub-accelerator  16  has less CNN processor cores, and the third sub-accelerator  18  has much less CNN processor cores. 
     The element-wise &amp; quantize processors  14 B are configured to process a scalar operation selected from a group of operations composed of addition, deduction, multiplication, division, batch normalization, quantization, bias and scaling. The pool and nonlinear function processor  14 C are configured to process a non-linear activation operation selected from a group of operations composed of rectified linear unit (ReLU), Sigmoid function and Tan h function. 
     First Embodiment 
       FIG.  2    is a schematic diagram of allocating operation of different layers to four different sub-accelerators for performing parallel processing according to an embodiment of the disclosure. The first sub-accelerator  14  has 64 CNN processor cores, the second sub-accelerator  16  has 256 CNN processor cores, the third sub-accelerator  18  has 256 CNN processor cores, and the fourth sub-accelerator  19  has 64 CNN processor cores. 
     Refer to  FIG.  2   . The convolutional neural network operation comprises the operation of nine layers. According to a first allocation arrangement, the convolutional neural network operation of nine layers is divided into a first partial operation, a second partial operation, a third partial operation and a fourth partial operation. The first partial operation comprises the operation of one layer, that is, the 1 st  layer. The second partial operation comprises the operation of three layers, that is, the 2 nd  layer to the 4 th  layer. The third partial operation comprises the operation of three layers, that is, the 5 th  layer to the 7 th  layer. The fourth partial operation comprises the operation of two layers, that is the 8 th  layer and the 9 th  layer. The first sub-accelerator  14  firstly performs the first partial operation (the operation of the 1 st  layer) and then outputs a first intermediate result R1. The second sub-accelerator  16  receives the first intermediate result R1 and continues to perform the second partial operation (the operation of the 2 nd  layer to the 4 th  layer) and then outputs a second intermediate result R2. The third sub-accelerator  18  receives the second intermediate result R2 and continues to perform the second partial operation (the operation of the 5 th  layer to the 7 th  layer). The fourth sub-accelerator  19  receives the third intermediate result R3 and continues to perform the fourth partial operation (the operation of the 8 th  layer and the 9 th  layer). 
     In view of the timing sequence diagram, (1) within the time interval T1, the first sub-accelerator  14  performs the operation of the 1 st  layer of the first convolution, and the second sub-accelerator  16 , the third sub-accelerator  18  and the fourth sub-accelerator  19  all are waiting; (2) within the time interval T2, the first sub-accelerator  14  performs the operation of the 1 st  layer of the second convolution, the second sub-accelerator  16  performs the operation of the 2 nd  layer to the 4 th  layer of the first convolution, and the third sub-accelerator  18  and the fourth sub-accelerator  19  both are waiting; (3) within the time interval T3, the first sub-accelerator  14  performs the operation of the 1 st  layer of the third convolution, the second sub-accelerator  16  performs the operation of the 2 nd  layer to the 4 th  layer of the second convolution, the third sub-accelerator  18  performs the operation of the 5 th  layer to the 7 th  layer of the first convolution, and the fourth sub-accelerator  19  is waiting; (4) within the time interval T4, the first sub-accelerator  14  performs the operation of the 1 st  layer of the fourth convolution, the second sub-accelerator  16  performs the operation of the 2 nd  layer to the 4 th  layer of the third convolution, the third sub-accelerator  18  performs the operation of the 5 th  layer to the 7 th  layer of the second convolution, and the fourth sub-accelerator  19  performs the operation of the 8 th  layer and the 9 th  layer of the first convolution. From the time interval T5, each unit time interval T will output a result of the convolutional operation of nine layers. From the time interval T4, the four sub-accelerators of the multi-stage pipeline operation architecture start to operate simultaneously and achieve the effect of parallel processing. 
     The unit time interval T is determined according to the longest time among the times required for each sub-accelerator to complete corresponding allocated CNN operation. For example, suppose the first sub-accelerator  14  takes 3.61 nsec to complete the operation of the 1 st  layer of the convolution, the second sub-accelerator  16  takes 3.61 nsec to complete the operation of the 2 nd  layer to the 4 th  layer of the convolution, the third sub-accelerator  18  takes 3.21 nsec to complete the operation of the 5 th  layer to the 7 th  layer of the convolution, and the fourth sub-accelerator  19  takes 4.51 nsec to complete the operation of the 8 th  layer and the 9 th  layer of the convolution, then the unit time interval T will be set to be greater than 4.51 nsec. 
     If the focus is to increase the efficiency of the operation of the multi-stage pipeline architecture, then more attention is paid to the longest time among the times required for each sub-accelerator to perform corresponding CNN operation, so that the length of the unit time interval T for one pipeline stage can be reduced. During the design planning stage, a second allocation arrangement can be tried. For example, under the second allocation arrangement, suppose the first sub-accelerator  14  takes 4.7 nsec to perform the operation of the 1 st  to the 2 nd  layer of the convolution, the second sub-accelerator  16  takes 3.7 nsec to perform the operation of the 3 rd  to the 5 th  layer of the convolution, the third sub-accelerator  18  takes 3.31 nsec to perform the operation of the 6 th  layer to the 8 th  layer of the convolution, and the fourth sub-accelerator  19  takes 4.0 nsec to perform the operation of the 9 th  layer of the convolution, then the longest time is over 4.51 nsec. It shows the first allocation arrangement shown in the  FIG.  2    can achieve a shorter unit time interval T, and the multi-stage pipeline operation can provide a higher efficiency. Thus, the first allocation arrangement shown in the  FIG.  2    is superior to the second allocation arrangement. During the design planning stage, the difference of the longest time interval deducted by the shortest time interval or the ratio of the longest time interval to the shortest time interval can be used as a criterion for selecting the allocation arrangement which achieves a shorter unit time interval T for one pipeline stage. When the first time interval T1 (the longest time interval) is substantially equivalent to the second time interval T2 (the shortest time interval), then the time to complete overall CNN operation can be reduced. In practice, the first time interval T1 can be 80%˜120% of the second time interval T2. 
     If the focus is to decrease the power consumption of the operation of the multi-stage pipeline architecture, then more attention is paid to the sum of the power consumption required for each sub-accelerator to perform the operation of the corresponding layer(s). During the design planning stage, the first allocation arrangement as indicated in  FIG.  2    can be tried first. The simulation result shows that the first sub-accelerator  14  consumes power E1 to perform the operation of the operation of the 1 st  layer of the convolution, the second sub-accelerator  16  consumes power E2 to perform the operation of the 2 nd  layer to the 4 th  layer of the convolution, the third sub-accelerator  18  consumes power E3 to perform the operation of the 5 th  layer to the 7 th  layer of the convolution, and the fourth sub-accelerator  19  consumes power E4 to perform the operation of the 8 th  layer and the 9 th  layer of the convolution. Then, a third allocation arrangement is tried. The simulation result shows that the first sub-accelerator  14  consumes power E5 to perform the operation of the 1 st  layer to the 2 nd  layer of the convolution, the second sub-accelerator  16  consumes power E6 to perform the operation of the 3 rd  layer to the 5 th  layer of the convolution, the third sub-accelerator  18  consumes power E7 to perform the operation of the 6 th  layer to the 8 th  layer of the convolution, and the fourth sub-accelerator  19  consumes power E8 to perform the operation of the 9 th  layer of the convolution. If the sum of the powers E1, E2, E3 and E4 is smaller than the sum of the powers E5, E6, E7 and E8, then CNN operation is performed according to the first allocation arrangement as shown in  FIG.  2   . If the sum of the powers E5, E6, E7 and E8 is smaller than the sum of the powers E1, E2, E3 and E4, then CNN operation is performed according to the second allocation arrangement. During the design planning stage, the sum of power consumption can be used as a criterion for determining the allocation arrangement, so lower total power consumption for all sub-accelerators consumed to complete the CNN operation is achieved. 
     The above disclosure is exemplified by four sub-accelerators. When the number of sub-accelerators changes, the difference of the longest time interval deducted by the shortest time interval, the ratio of the longest time interval to the shortest time interval, or the sum of power consumption still can be used as a criterion for allocating operation to different sub-accelerators, so a shorter time to complete overall CNN operation is achieved. 
     Second Embodiment 
       FIG.  3    is a schematic diagram of allocating trunk and branch operations to different sub-accelerators  14 ,  16  and  18  of a CNN accelerator by a dynamic task scheduler  10  according to another embodiment of the disclosure, wherein the convolutional neural network operation comprises a trunk operation  34 , a first branch operation  36  and a second branch operation  38 . The first sub-accelerator  14  (with more CNN processor cores) performs the network in the trunk operation  34 , and the second sub-accelerator  16  (with less CNN processor cores) and the third sub-accelerator  18  (with much less CNN processor cores) perform the first branch operation  36  and the second branch operation  38 , respectively. 
     To save the power consumption of the entire accelerator, in step  37 , after check point computing, whether the CNN accelerator should enable one of the branch operations  36  and  38  is determined. When the convolutional neural network operation completes the trunk operation  34  and intends to perform the first branch operation  36 , the first sub-accelerator  14  enters the first power saving mode, and the process proceeds to step  39 . In step  39 , the corresponding second sub-accelerator  16  is enabled and exits the second power saving mode. When the convolutional neural network operation finishes the first branch operation  36  and intends to perform the trunk operation  34 , the first sub-accelerator  14  exits the first power saving mode, and the second sub-accelerator  16  enters the second power saving mode. Thus, the effect of dynamic power saving can be achieved. Similarly, when the convolutional neural network operation switches between the trunk operation  34  and the second branch operation  38 , the third sub-accelerator  18  will also selectively exit and enter the third power saving mode. 
     This another embodiment is mainly used in the scenario of dynamic branch network operation. The to-be-recognized images can be divided into a primary category and a secondary category. The trunk operation  34  is a main image recognition which has a higher frequency or requires a longer time of continuous operation. The first branch operation  36  is a scenario image recognition which has a lower frequency or operates occasionally for a shorter time. The trunk operation  34  and the first branch operation share the front-end feature map of the neural network in the trunk operation  34 . For example, the trunk operation can be a recognition operation of main image for recognizing a main factory object which is seen more frequently, and the first branch operation can be a recognition operation of scenario image for recognizing a customized object which is seen less frequently. Besides, the trunk operation can be a recognition operation of collision sound, and the branch operation can be an operation for recognizing and analyzing an audio source for the collision sound. 
     Third Embodiment 
       FIG.  4    is a schematic diagram of allocating trunk and branch operations to different sub-accelerators  14  and  16  of a CNN accelerator by a dynamic task scheduler  10  according to another embodiment of the disclosure, wherein the convolutional neural network operation comprises a trunk operation  44 , a first branch operation  46  and a second branch operation  48 . The first sub-accelerator  14  performs the trunk operation  44 , the second sub-accelerator  16  is left available for further dynamic designation, and the second sub-accelerator  16  selectively performs one of the first branch operation  46  and the second branch operation  48 . When the convolutional neural network operation intends to perform the first branch operation  46 , the second sub-accelerator  16  loads in a program code corresponding to the first branch operation  46 . When the convolutional neural network operation intends to perform the second branch operation  48 , the second sub-accelerator  16  loads in a program code corresponding to the second branch operation  48 . 
     To save the power consumption of the entire accelerator, in step  47 , after the check point computing, whether the CNN accelerator should enable one of the branch operations  46  and  48  is determined. When the convolutional neural network operation finishes the trunk operation  44  and intends to perform the first branch operation  46 , the first sub-accelerator  14  enters the first power saving mode, and the process proceeds to step  49 . In step  49 , the second sub-accelerator  16  is enabled and exits the second power saving mode. When the convolutional neural network operation finishes the first branch operation  46  and intends to perform the trunk operation  44 , the first sub-accelerator  14  exits the first power saving mode, and the second sub-accelerator  16  enters the second power saving mode. Thus, the effect of dynamic power saving can be achieved. Similarly, when the convolutional neural network operation finishes the trunk operation  44  and switches between the second branch operation  48 , the process proceeds to step  51 . In step  51 , the second sub-accelerator  16  is enabled and exits the second power saving mode. 
     The another embodiment of the disclosure is mainly used in an event-triggered dynamic tree network operation. That is, the neural network in the trunk operation  44  performs object detection and roughly divide the detected objects into two types, A and B. When an object A is detected, the neural network in the first branch operation  46  performs fine recognition on the object A. When an object B is detected, the neural network in the second branch operation  48  performs fine recognition on the object B. A specific embodiment could be as follows: the neural network in the trunk operation  44  is used for vehicle detection on autopilot. When a person on the roadside is detected, the second sub-accelerator  16  loads in a program code corresponding to the neural network in the first branch operation  46  to perform a branch operation relevant to human recognition. When an adjacent vehicle is detected, the second sub-accelerator  16  loads in a program code corresponding to the neural network in the second branch operation  48  to perform a branch operation relevant to vehicle recognition. If the neural network in the trunk operation  44  does not detect any person or vehicle, then the neural network in the trunk operation  44  continues and there is no need to trigger the first branch operation  46  or the second branch operation  48 . 
     Refer to Table 3 and Table 4, which show a comparison of operation time and power consumption between the dynamic multi-mode CNN accelerator of the first embodiment as indicated in  FIG.  2    and the single 2048-core CNN accelerator of the prior art for completing a same CNN operation. According to the first embodiment, the dynamic multi-mode CNN accelerator comprises four sub-accelerators featured by “identical processor structure” and “different numbers of cores” and composed of two 64-core sub-accelerators and two 256-core sub-accelerators. The dynamic multi-mode CNN accelerator adopts a pipeline parallel processing architecture and has an operating clock of 600 MHz and a memory transmission bandwidth of 8 GB in total. The four sub-accelerators independently perform a partial convolutional neural network operation but output a complete result of convolutional neural network operation after four stages of pipeline parallel processing. According to the prior art, single 2048-core CNN accelerator, under the condition of the same operating clock of 600 MHz and the same memory transmission bandwidth of 8 GB, independently performs the same complete convolutional neural network operation. A comparison of operation time and simulated power consumption between the two types of CNN accelerators disclosed above is listed in Table 3 and Table 4. 
     As indicated in Table 3: based on the architecture of the single 2048-core accelerator of the prior art, a complete result of convolutional neural network operation is outputted every 4.7 nsec; based on the architecture of the four sub-accelerators CNN accelerators of the disclosure, a complete result of convolutional neural network operation is outputted every 4.51 nsec. As indicated in Table 4: based on the architecture of the single 2048-core accelerator of the prior art, the operation consumes a power of 420 mW; based on the architecture of the four sub-accelerators CNN accelerators of the disclosure, the operation consumes a power of 370 mW. The above comparison shows that the architecture of the dynamic multi-mode CNN accelerator of the disclosure comprising at least two sub-accelerators is superior to the architecture of the single 2048-core CNN accelerator of the prior art in terms of both operation time and power consumption. 
     
       
         
           
               
               
               
               
               
               
             
               
                 TABLE 3 
               
               
                   
               
               
                   
                   
                 Cycle of 
                 Cycle of 
                 Cycle of 
                 Cycle of 
               
               
                   
                   
                 64-Core 
                 256-Core 
                 256-Core 
                 64-Core 
               
               
                   
                 Cycle of 
                 Memory 
                 Memory 
                 Memory 
                 Memory 
               
               
                 CNN convolutional 
                 2048 
                 bandwidth 
                 bandwidth 
                 bandwidth 
                 bandwidth 
               
               
                 model 
                 cores 
                 2 GB/s 
                 1 GB/s 
                 1 GB/s 
                 4 GB/s 
               
               
                   
               
             
            
               
                   
               
            
           
           
               
               
               
               
               
               
            
               
                 1-layer convolution 
                 1.81 
                 3.61 
                   
                   
                   
               
               
                 CONV1 
               
               
                 2-layer convolution 
                 0.45 
                   
                 1.81 
               
               
                 CONV2 
               
               
                 3-layer convolution 
                 0.23 
                   
                 0.90 
               
               
                 CONV3 
               
               
                 4-layer convolution 
                 0.11 
                   
                 0.90 
               
               
                 CONV4 
               
               
                 5-layer convolution 
                 0.11 
                   
                   
                 0.90 
               
               
                 CONV5 
               
               
                 6-layer convolution 
                 0.11 
                   
                   
                 0.90 
               
               
                 CONV6 
               
               
                 7-layer convolution 
                 0.35 
                   
                   
                 1.40 
               
               
                 CONV7 
               
               
                 8-layer convolution 
                 0.18 
                   
                   
                   
                 1.81 
               
               
                 CONV8 
               
               
                 9-layer convolution 
                 1.35 
                   
                   
                   
                 2.70 
               
               
                 CONV9 
                   
                   
                   
                   
                   
               
               
                 Sum 
                 4.70 
                 3.61 
                 3.61 
                 3.21 
                 4.51 
               
               
                   
               
            
           
         
       
     
     
       
         
           
               
               
               
             
               
                   
                 TABLE 4 
               
               
                   
                   
               
               
                   
                 Single 2048-core 
                 Dual 64 and Dual 256 
               
               
                   
                 accelerator 
                 640 cores in total 
               
               
                   
                   
               
             
            
               
                   
               
            
           
           
               
               
               
               
            
               
                 Frequency 
                 600 
                 MHz 
                 All are 600 MHz 
               
               
                 Memory 
                 8 
                 GB/s 
                 Respectively: 1,1,2,4 GB/s 
               
               
                 bandwidth 
                   
                   
                 Sum: 8 GB/s 
               
               
                 Internal memory 
                 512 
                 KB 
                 512 KB (128 KB each) 
               
               
                 Number of 
                 4.7 
                 million 
                 4.5 million 
               
               
                 operating clocks 
               
            
           
           
               
               
               
            
               
                 Simulated power 
                 About 420 mW 
                 About 370 mW 
               
               
                 consumption 
               
               
                 Power saving 
                 Coarse grained 
                 Fine grained 
               
               
                 function 
               
               
                   
               
            
           
         
       
     
     It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed embodiments. It is intended that the specification and examples be considered as exemplary only, with a true scope of the disclosure being indicated by the following claims and their equivalents.