Patent Publication Number: US-10769485-B2

Title: Framebuffer-less system and method of convolutional neural network

Description:
BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The present invention generally relates to a convolutional neural network (CNN), and more particularly to a CNN system without framebuffer. 
     2. Description of Related Art 
     A convolutional neural network (CNN) is a class of artificial neural networks that may be adapted to machine learning. The CNN can be applied to signal processing such as image processing and computer vision. 
       FIG. 1  shows a block diagram illustrating a conventional CNN  900  as disclosed in “A Reconfigurable Streaming Deep Convolutional Neural Network Accelerator for Internet of Things,” entitled to Li Du et al., August 2017, IEEE Transactions on Circuits and Systems I: Regular Papers, the disclosure of which is incorporated herein by reference. The CNN  900  includes a single port static random-access memory (SRAM) as a buffer bank  91  to store intermediate data and exchange data with a dynamic random-access memory (DRAM) (e.g., double data rate synchronous DRAM (DDR SDRAM)) as a framebuffer  92  required to store whole image frame for CNN operation. The buffer bank  91  is separated into two sets: an input layer and an output layer. The CNN  900  includes a column (COL) buffer  93  that is used to remap output of the buffer bank  91  to a convolution unit (CU) engine array  94 . The CU engine array  94  is composed of a plurality of convolution units to enable highly parallel convolution computation. A pre-fetch controller  941  is included inside the CU engine array  94  to periodically fetch parameters from a direct memory access (DMA) controller (not shown) and update weights and bias values in the CU engine array  94 . The CNN  900  also includes an accumulation (ACCU) buffer  95  with scratchpad used to store partial convolution results from the CU engine array  94 . A max pool  951  is included in the ACCU buffer  95  to pool output-layer data. The CNN  900  includes an instruction decoder  96  used to store commands that are pre-stored in the framebuffer  92 . 
     In the conventional CNN system as exemplified in  FIG. 1 , a framebuffer composed of a dynamic random-access memory (DRAM) (e.g., double data rate synchronous DRAM (DDR SDRAM)) is commonly required to store whole image frame for CNN operation. For example, framebuffer may occupy large space of 320×240×8 bits for an image frame with a 320×240 resolution. However, DDR SDRAM is not available for most low-power applications such as wearables or Internet of things (IoT). A need has arisen to propose a novel CNN system that is adaptable to low-power applications. 
     SUMMARY OF THE INVENTION 
     In view of the foregoing, it is an object of the embodiment of the present invention to provide a convolutional neural network (CNN) system without framebuffer. The embodiment is capable of performing CNN operation on high-resolution image frame with low system complexity. 
     According to one embodiment, a framebuffer-less system of convolutional neural network (CNN) includes a region of interest (ROI) unit, a convolutional neural network (CNN) unit and a tracking unit. The ROI unit extracts features, according to which a region of interest in an input image frame is generated. The CNN unit processes the region of interest of the input image frame to detect an object. The tracking unit compares the features extracted at different times, according to which the CNN unit selectively processes the input image frame. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  shows a block diagram illustrating a conventional CNN; 
         FIG. 2A  shows a block diagram illustrating a framebuffer-less system of convolutional neural network (CNN) according to one embodiment of the present invention; 
         FIG. 2B  shows a flow diagram illustrating a framebuffer-less method of convolutional neural network (CNN) according to one embodiment of the present invention; 
         FIG. 3  shows a detailed block diagram of the ROI unit of  FIG. 2A ; 
         FIG. 4A  shows an exemplary decision map composed of 4×6 blocks; 
         FIG. 4B  shows another exemplary decision map updated after that in  FIG. 4A ; 
         FIG. 5  shows a detailed block diagram of the temporary storage of  FIG. 2A ; and 
         FIG. 6  shows a detailed block diagram of the CNN unit of  FIG. 2A . 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       FIG. 2A  shows a block diagram illustrating a framebuffer-less system  100  of convolutional neural network (CNN) according to one embodiment of the present invention, and  FIG. 2B  shows a flow diagram illustrating a framebuffer-less method  200  of convolutional neural network (CNN) according to one embodiment of the present invention. 
     In the embodiment, the system  100  may include a region of interest (ROI) unit  11  configured to generate a region of interest in an input image frame (step  21 ). Specifically, as the system  100  of the embodiment contains no framebuffer, the ROI unit  11  may adopt scan-line based technique and block-based scheme to find the region of interest in the input image frame, which is divided into a plurality of blocks of image arranged in a matrix form composed of, for example, 4×6 blocks of image. 
     In the embodiment, the ROI unit  11  is configured to generate block-based features, according to which decision of whether to perform CNN is made for each block of image.  FIG. 3  shows a detailed block diagram of the ROI unit  11  of  FIG. 2A . In the embodiment, the ROI unit  11  may include a feature extractor  111  configured to extract, for example, shallow features from the input image frame. In one exemplary embodiment, the feature extractor  111  generates (shallow) features of the blocks according to block-based histogram. In another exemplary embodiment, the feature extractor  111  generates (shallow) features of the blocks according to frequency analysis. 
     The ROI unit  11  may also include a classifier  112 , such as support vector machine (SVM), configured to make decision whether to perform CNN for each block of the input image frame. Accordingly, a decision map  12  composed of a plurality of blocks (e.g., arranged in a matrix form) representing the input image frame is generated.  FIG. 4A  shows an exemplary decision map  12  composed of 4×6 blocks, where X indicates that associated block requires no CNN performance, C indicates that associated block requires CNN performance, and D indicates that an object (e.g., a dog) is detected in associated block. Accordingly, the ROI is determined and is thereafter subjected to CNN performance. 
     Referring back to  FIG. 2A , the system  100  may include temporary storage  13  such as static random-access memory (SRAM), which is configured to store the (shallow) features generated by the feature extractor  111  (of the ROI unit  11 ) (Step  22 ).  FIG. 5  shows a detailed block diagram of the temporary storage  13  of  FIG. 2A . In the embodiment, the temporary storage  13  may include two feature maps  131 —first feature map  131 A used to store features of a previous image frame (e.g., at time t−1) and second feature map  131 B used to store features of a current image frame (e.g., at time t). The temporary storage  13  may also include a sliding window  132  of a size, for example, of 40×40×8 bits for storing a block of the input image frame. 
     Referring back to  FIG. 2A , the system  100  of the embodiment may include a convolutional neural network (CNN) unit  14  that operatively receives and processes the generated ROI (from the ROI unit  11 ) of the input image frame to detect an object (step  23 ). Specifically, the CNN unit  13  of the embodiment performs operation only on the generated ROI, instead of entire input image frame as in a conventional system with framebuffer. 
       FIG. 6  shows a detailed block diagram of the CNN unit  14  of  FIG. 2A . Specifically, the CNN unit  14  may include a convolution unit  141  including a plurality of convolution engines configured to perform convolution operation. The CNN unit  14  may include an activation unit  142  configured to perform activation functions when predefined features are detected. The CNN unit  14  may also include a pooling unit  143  configured to perform down-sampling (or pooling) on the input image frame. 
     The system  100  of the embodiment may include a tracking unit  15  configured, in step  24 , to compare the first feature map  131 A (of the previous image frame) and the second feature map  131 B (of the current image frame), followed by updating the decision map  12 . The tracking unit  15  analyzes content variation between the first feature map  131 A and the second feature map  131 B.  FIG. 4B  shows another exemplary decision map  12  updated after that in  FIG. 4A . In this example, the object detected in the blocks located at columns  5 - 6  and row  3  at a previous time (designated D in  FIG. 4A ) disappears in the same blocks at a current time (designated X in  FIG. 4B ). According to feature variation (and constant), the CNN unit  14  need not perform CNN operation on those blocks without feature variation. Alternatively speaking, the CNN unit  14  selectively performs CNN operation only on those blocks with feature variation. Therefore, operation of the system  100  can be substantially accelerated. 
     According to the embodiment proposed above, the amount of CNN operation may be substantially reduced (and thus accelerated) compared with a conventional CNN system. Moreover, as the embodiment of the present invention requires no framebuffer, the embodiment can be preferably adaptable to low-power applications such as wearables or Internet of things (IoT). Regarding an image frame of a 320×240 resolution and a (non-overlap) sliding window of a size of 40×40, the conventional system with framebuffer requires 8×6 sliding window operations for CNN. To the contrary, only a few (e.g., less than ten) sliding window operations for CNN are required in the system  100  of the embodiment. 
     Although specific embodiments have been illustrated and described, it will be appreciated by those skilled in the art that various modifications may be made without departing from the scope of the present invention, which is intended to be limited solely by the appended claims.