# An Energy-Efficient Convolutional Neural Network Processor Architecture Based on a Systolic Array
###### Abstract  
Deep convolutional neural networks (CNNs) have shown strong abilities in the application of artificial intelligence. However, due to their extensive amount of computation, traditional processors have low energy efficiency when executing CNN algorithms, which is unacceptable for portable devices with limited hardware cost and battery capacity, so designing a CNN-specific processor is necessary. In this paper, we propose an energy-efficient CNN processor architecture for lightweight devices with a processing elements (PEs) array consisting of 384 PEs. Using the systolic array-based PE array, it realizes parallel operations between filter rows and between channels of output feature maps, supporting the acceleration of 3D convolution and fully connected computation with various parameters by configuring internal instruction registers. The computing strategy based on the proposed systolic dataflow achieves less hardware overhead compared with other strategies, and the reuse of image values and weight values, which effectively reduce the power of memory access. A memory system with a multi-level storage structure combined with register file (RF) and SRAM is used in the proposed CNN processor, which further reduces the energy overhead of computing. The proposed CNN processor architecture has been verified on a ZC706 FPGA platform using VGG-16 based on the proposed image segmentation method, the evaluation results indicate that the peak throughput achieves 115.2 GOP/s consuming 3.801 W at 150 MHz, energy efficiency and DSP efficiency reaches 30.32 GOP/s/W and 0.26 GOP/s/DSP, respectively. convolutional neural network (CNN); processing elements (PEs); systolic array; multi-level storage; FPGA +
Footnote †: journal: Applied  
proposed for solving these. [PERSON] et al. optimized the storage of the CNN processor [8] to alleviate the problem of limited off-chip storage bandwidth through data reuse and multi-level storage. This design is implemented on a FPGA and increases the operation speed by 11 times compared to the design without data reuse, while consuming the same resources. In order to save the bandwidth and memory access energy, pruning methods using different strategies are used to compress the neural network, such as the filter-level pruning algorithm [9] and the mixed-pruning method [10], which can reduce the operations of the network by over 50% and consume significantly less energy compared to the design with similar recognition accuracy. [PERSON] et al. proposed a K-means-assisted scenario-aware reconfigurable CNN processor, whereby its Lego-like architecture reduces the design complexity [11]. Block convolution is applied to the inference of large-scale CNNs on the FPGA [12], which is realized by splitting a feature map into independent blocks that can completely avoid the off-chip transfer of intermediate feature maps at runtime. Low-precision techniques can effectively reduce the power and bandwidth requirements of CNN inference, but may lead to accuracy degradation. Mixed low-precision can bring advantages of a low precision while maintaining accuracy, such as W8A8 (INT8 weight and INT8 activation) and WTA2 (TERNARY weight and INT2 activation) mixed-precision [13]. Due to the sparsity of CNNs, zero-skipping is used to reduce data access and computing energy [14; 15]. In addition, high-bandwidth memory (HBM2) is also applied to the CNN inference processor [16] for solving the limited bandwidth data movement between the FPGA-based processor and the external memory. Recently, heterogeneous computing has become a new research trend. [PERSON] et al. proposed a CPU-FPGA-based heterogeneous acceleration system and a subgraph segmentation scheme for CNN-RNN hybrid neural networks [17]. Fixed-point quantization and cyclic tiling are used to reduce hardware resource usage and achieve a high degree of parallelization. [PERSON] et al. proposed a heterogeneous accelerator [18] which deploys computationally intensive operations on the FPGA, and the CPU controls the FPGA accelerator to operate through PCI-e.