Sparse convolutional neural network acceleration with lossless input feature map compression for resource‐constrained systems

IET Computers & Digital TechniquesVolume 16, Issue 1 p. 29-43 ORIGINAL RESEARCHOpen Access Sparse convolutional neural network acceleration with lossless input feature map compression for resource-constrained systems Jisu Kwon, Jisu Kwon School of Electronic and Electrical Engineering, Kyungpook National University, Daegu, South KoreaSearch for more papers by this authorJoonho Kong, Corresponding Author Joonho Kong joonho.kong@knu.ac.kr orcid.org/0000-0002-9013-9561 School of Electronic and Electrical Engineering, Kyungpook National University, Daegu, South Korea Correspondence Joonho Kong, School of Electronic and Electrical Engineering, Kyungpook National University, Daegu 41566, South Korea. Email: joonho.kong@knu.ac.krSearch for more papers by this authorArslan Munir, Arslan Munir Department of Computer Science, Kansas State University, Manhattan, Kansas, USASearch for more papers by this author Jisu Kwon, Jisu Kwon School of Electronic and Electrical Engineering, Kyungpook National University, Daegu, South KoreaSearch for more papers by this authorJoonho Kong, Corresponding Author Joonho Kong joonho.kong@knu.ac.kr orcid.org/0000-0002-9013-9561 School of Electronic and Electrical Engineering, Kyungpook National University, Daegu, South Korea Correspondence Joonho Kong, School of Electronic and Electrical Engineering, Kyungpook National University, Daegu 41566, South Korea. Email: joonho.kong@knu.ac.krSearch for more papers by this authorArslan Munir, Arslan Munir Department of Computer Science, Kansas State University, Manhattan, Kansas, USASearch for more papers by this author First published: 29 November 2021 https://doi.org/10.1049/cdt2.12038AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Many recent research efforts have exploited data sparsity for the acceleration of convolutional neural network (CNN) inferences. However, the effects of data transfer between main memory and the CNN accelerator have been largely overlooked. In this work, the authors propose a CNN acceleration technique that leverages hardware/software co-design and exploits the sparsity in input feature maps (IFMs). On the software side, the authors' technique employs a novel lossless compression scheme for IFMs, which are sent to the hardware accelerator via direct memory access. On the hardware side, the authors' technique uses a CNN inference accelerator that performs convolutional layer operations with their compressed data format. With several design optimization techniques, the authors have implemented their technique in a field-programmable gate array (FPGA) system-on-chip platform and evaluated their technique for six different convolutional layers in SqueezeNet. Results reveal that the authors' technique improves the performance by 1.1×–22.6× while reducing energy consumption by 47.7%–97.4% as compared to the CPU-based execution. Furthermore, results indicate that the IFM size and transfer latency are reduced by 34.0%–85.2% and 4.4%–75.7%, respectively, compared to the case without data compression. In addition, the authors' hardware accelerator shows better performance per hardware resource with less than or comparable power consumption to the state-of-the-art FPGA-based designs. 1 INTRODUCTION Convolutional neural networks (CNNs) have been widely deployed in real-world applications such as real-time object detection and image classification [1]. Along with continuous developments and improvements in CNN models, many hardware accelerators for CNNs have been introduced [2] for not only performance improvement but also energy reduction. In the meantime, though huge efforts have been expended to improve computational performance via hardware-based acceleration, there have been few works focussing on the reduction of off-chip data transfer overheads. Although CNN model parameters are continuously being reduced via model compression and quantization [3], the CNN hardware acceleration still entails huge data transfer between the hardware accelerator and memory via direct memory access (DMA). This data transfer overhead aggrandizes latency, memory bandwidth pressure, and energy consumption, which limits the feasibility of many contemporary CNN accelerators for resource-constrained systems such as embedded and Internet-of-things (IoT) devices. Even though many hardware accelerators attempt to reuse data to minimize off-chip data transfer, the limited on-chip memory of CNN hardware accelerators results in large amounts of data transfer between hardware accelerators and main memory. The data transfer overhead in CNNs can be classified into two types: latency overhead and energy overhead. The latency overhead could be hidden by double buffering. However, double buffering requires twice more on-chip memory capacity to support the same processing element (PE) utilizations, thus inhibiting its suitability for resource-constrained systems. Furthermore, in the case of huge amounts of data transfer between the accelerator and main memory, data transfer latency cannot be hidden completely by overlapping computations with data transfer. Moreover, overlapping computations with data transfer do not hide the energy overhead of data transfer. The only way to minimize data transfer energy is to reuse on-chip data as much as possible. However, limited size of on-chip memory in CNN hardware accelerators restricts the amount of the data reuse, thus resulting in non-negligible energy overhead. Though processing in memory (PIM) has been emerged to minimize data transfer, PIM would be hard to be adopted in resource-constrained systems. Hence, it is imperative to consider both computational performance and data transfer between main memory and the CNN accelerator when designing CNN hardware accelerator for resource-constrained systems. To tackle this problem, this paper proposes a novel CNN acceleration technique for resource-constrained systems by leveraging hardware-software (HW/SW) co-design. To reduce data transfer overhead between the hardware accelerator and main memory, it is proposed to use an efficient compression method for input feature map (IFM) data. The proposed scheme exploits the activation sparsity in CNN models, which is attributed to rectified linear unit (ReLU) activation function. The proposed IFM data compression scheme, which is performed on the software side, removes the transfer of zero-valued elements in IFMs. On the hardware side, it is proposed to use an accelerator architecture that efficiently performs convolution operations with our compressed data format. In addition, our proposed hardware accelerator adopts several techniques to further optimize our CNN hardware accelerator. Firstly, it exploits the parallelism that exists in searching and weight matching of non-zero elements (NZEs) from our compressed IFM format. Secondly, it reuses the input data inside of the internal buffer in our accelerator when performing the convolution with N × N kernels (N >1), thus eliminating unnecessary internal data transfer inside of the accelerator. Thirdly, it replaces complex operations required for searching and weight matching of NZEs with simpler operations such as lookup table (LUT) and shift operations. The implementation of our HW/SW co-designed CNN accelerator on a field-programmable gate array-system-on-chip (FPGA-SoC) shows an average of 1.1×–22.6× performance improvement with 47.7%–97.4% energy reduction as compared to the case of using embedded/mobile CPU (ARM Cortex-A53 [4])-based CNN inference across the degree of the IFM sparsity (50%–90%). Note that the proposed technique is based on lossless compression of IFMs, meaning that there is no accuracy loss by adopting the technique. In addition, our hardware accelerator implementation shows better cost efficiency (attainable performance per cost) with less or comparable power consumption as compared to other state-of-the-art accelerators [5-9], which means that the proposed acceleration technique is more suitable for low-power, low-energy, and resource-constrained systems. Our main contributions in this paper can be summarized as follows: Proposal of a software-based novel compression scheme that compresses IFM data to reduce the amount of data transfer between the memory and the accelerator Proposal of a hardware accelerator design that accelerates CNN inference with the proposed compressed data format Introduction of several optimization techniques for hardware accelerator, such as parallel searching and weight matching, input data reuse, and removal of complex operations Implementation of the proposed hardware accelerator in an FPGA-SoC platform with two different versions of precision supports, 32-bit floating-point and 16-bit fixed-point, which shows better cost efficiency with less or comparable power consumption compared to the state-of-the-art FPGA-based accelerator designs Analysis of the proposed HW/SW co-designed IFM compression technique with the accelerator. Results reveal a huge performance improvement of 1.1×–22.6× and energy reduction by 47.7%–97.4% as compared to a software-based CNN inference on an embedded/mobile processor The remainder of this paper is organized as follows. Section 2 presents the relevant works in the literature related to CNN acceleration and data transfer reduction. Section 3 presents the background of CNNs and their input sparsity. The proposed HW/SW co-designed CNN accelerator and its implementation on an FPGA-SoC platform are discussed in Section 4. Section 5 presents the evaluation results. Section 6 discusses several limitations of our work. Finally, Section 7 concludes this paper with future research directions. 2 RELATED WORK There exist many works for sparse CNN accelerator designs. Cnvlutin [10] eliminates multiplication and addition operations with zero values by exploiting the sparsity of the inter-layer data (i.e., caused by activation) in CNNs. Cnvlutin accelerator does not bring zero data from on-chip eDRAM to internal buffers, removing the multiplication and addition with the zero values in the multiply-and-accumulate (MAC) units. Cambricon-X [11] focusses on removing ineffectual operations related to the pruned weights, resulting in the reduction of the required computations. SCNN accelerator [12] exploits both weight and activation sparsity by exploiting Cartesian-product-based internal dataflow architecture. In Ref. [13], to skip multiplications and additions with zero values, a new PE design has been introduced. Sparten [14] introduces an efficient accelerator architecture to find and match NZEs (i.e., inner-join) for efficient vector-vector multiplication with sparse weights and inputs. However, the works introduced above mostly focus only on the accelerator design while the data transfer reduction with the compressed data is ignored. In Ref. [5], Aimar et al. have presented FPGA- and application-specific integrated circuits (ASIC)-based CNN accelerators that exploit the sparsity in feature maps. Although the CNN accelerators presented in [5] attained a performance of hundreds of giga operations per second (GOPs) in certain CNN models, the accelerators only utilized hardware optimizations for acceleration, whereas our proposed technique also introduces a software-based compression technique along with the hardware accelerator. Furthermore, according to the quantitative comparison that will be presented in Section 5.3, our proposed design shows better cost efficiency (i.e., performance per unit cost or performance per hardware resource) as compared to the FPGA-based implementation of [5]. In Ref. [6], Kala et al. have introduced the Winograd-based CNN accelerator design on FPGA. However, the sparsity in IFMs is not considered in Ref. [6] while our proposed technique utilizes HW/SW co-design methodology for sparsity-aware CNN inference acceleration and memory data transfer reduction. In addition, there have been several FPGA-based designs [7-9] for CNN acceleration. However, our proposed design shows better (or comparable) cost efficiency as compared to those designs as illustrated in Section 5.3. Furthermore, our design also contributes to memory energy reduction and effective bandwidth improvement by adopting our two-step IFM compression technique. For data transfer reduction of deep neural networks (DNNs), Chen et al. have introduced the Eyeriss design [15] that exploits activation sparsity with data compression by run-length coding. Though data compression and DRAM access reduction have been considered in [15], the corresponding accelerator design cannot fully eliminate the latency caused by the multiplication operations with zero values while merely power gating the PEs with zero inputs. In Ref. [16], Li et al. have proposed an FPGA-based accelerator with an optimization for efficient off-chip memory bandwidth usage. However, their design only focusses on fully connected layers while our design can be applied to convolution layers that consume the largest execution time among the entire layers in CNNs. In Ref. [17], by exploiting activation sparsity during DNN training, Rhu et al. have introduced data compression methods and a hardware architecture for the DMA engine that enables data compression/decompression. However, their focus is DNN training with GPUs while our focus is CNN inference with hardware accelerator optimized for our novel compressed data format. In Ref. [18], Shen et al. have proposed a CNN accelerator with a flexible buffering scheme that enables a balance between the input and weight transfer to efficiently utilize off-chip data transfer bandwidth. However, the sparsity of data in CNNs has not been considered in their work while our technique reduces both off-chip data transfer via compression and the amount of computations with our novel accelerator design. In Ref. [19], Li et al. have proposed a technique (SmartShuttle) to maximize OPS (operations per second) per DRAM access by layer tiling and scheduling. To find the optimal tiling factor, it elaborates an optimization problem for DRAM access minimization and adaptively applies a scheduling scheme for input-, output-, or weight-reuse while also considering the data compression ratio due to data sparsity. However, SmartShuttle is a software-based scheme to minimize the off-chip data transfer while the amount of the required computations for CNNs is not reduced. In Ref. [20], Zhang et al. have proposed a technique to reduce on-chip SRAM energy in systolic array-based CNN accelerators. By adopting and optimizing the run-length coding, it reduces dynamic energy consumed by accessing the on-chip SRAM memory. In addition, it also employs bank-level low-power modes (drowsy and sleep) to reduce leakage power. However, their proposed technique only focusses on energy reduction while performance improvement by exploiting the data sparsity is overlooked. In Ref. [21], Yuan et al. have proposed a CNN model compression method with the hardware accelerator design. For compression, the method proposed in Ref. [21] classifies the CNN layers into non-pruned (NP) and pruned (P) layers. For error resilience, different compression approaches are employed to NP- and P-layers. For efficient processing of the compressed weights in NP- and P-layers, the proposed hardware architecture introduces a hybrid PE design. For NP-layers and P-layers, finite impulse response-based PEs and shift-accumulator based PEs have been proposed, respectively. However, the compression method proposed in [21] focusses on the weight compression while the IFM compression is overlooked. In addition, due to the data losses during the weight compression, there is an inevitable accuracy loss (0.44% loss for top-5 accuracy in VGG-16 [22]) while ours does not incur any accuracy loss because of the lossless compression of IFMs. 3 BACKGROUND 3.1 Convolutional neural networks CNNs are increasingly being used in many real-world applications. CNN models typically contain many different layers: convolution layers, pooling layers, fully connected layers etc. Typical CNN models have many convolutional layers and there are several pooling layers in-between the convolutional layers. Fully connected layers generally exist in the last layer of the CNNs. This paper focusses on accelerating convolutional (CONV) layers as CNNs spend most of the execution time in CONV layers when performing CNN inferences. As shown in Figure 1, the CONV layer operations are composed of convolution, add (sometimes, and scale) bias, and activation. The input to the CONV layer is IFMs, which are comprised of three-dimensional tensors. There is also a weight input that is composed of multiple three-dimensional filters (thus, four-dimensional tensors). One filter has a format of N × N × M tensor (N × N kernel with the depth of M). The convolution operations are typically implemented with MAC operations. After the convolution operations in CNNs, the bias can be added (and also multiplied for scale). For each filter, one can have different bias values. After the addition of bias, one performs an activation function. Though sigmoid function was used in the early CNN models, ReLU is currently more widely used due to its computation simplicity with negligible accuracy losses. ReLU activation function rectifies negative values while it merely forwards positive values. FIGURE 1Open in figure viewer The convolution layer operations in convolutional neural networks 3.2 Sparsity of CNN IFMs The ReLU activation function results in a huge sparsity in the output feature maps (OFMs) in CNNs. As the output data from the previous layer is used as an input for the next layer in CNNs, there is a huge amount of zero data in the IFMs. It means that there could be huge opportunities for reduction of data size and computation by handling only NZEs in IFMs. As reported in Ref. [14], AlexNet [23] exhibits the input density of 20%–38%, meaning that 62%–80% of IFMs comprise zero elements. In VGGNet [22], it is also reported that the average input density is 13%–57%, which implies there could be a huge opportunity to exploit the input sparsity. 4 CNN ACCELERATION AND DATA TRANSFER REDUCTION FOR SPARSE IFMs 4.1 Overview For CNN acceleration with data transfer overhead reduction, our technique exploits an HW/SW co-design methodology. Figure 2 shows an overall flow of the proposed CNN acceleration technique for sparse IFM. On the software side, the non-compressed IFM is converted into the compressed format before it is sent to the on-chip memory in the hardware accelerator via DMA. The weight data is not compressed and just sent to the hardware accelerator. Though there are also non-negligible zero-valued weights in CNNs when using network pruning [24], it is out-of-scope of this work and left for our future work. On the hardware side, the proposed CNN accelerator performs convolution layer operations with the weight and compressed IFMs. The following subsections present how IFMs are compressed in software (Section 4.2) and convolution operations are performed in the hardware accelerator (Section 4.3). FIGURE 2Open in figure viewer The execution flow of the proposed acceleration technique 4.2 Sparse IFM compression Since there is a huge sparsity in CNN feature maps, the proposed technique only maintains non-zero values (elements) with metadata for a location while zero values are omitted in the IFM. In order to record the location of NZEs in IFMs, the proposed technique also maintains indices of the NZEs in IFMs. Figure 3 shows a format of compressed IFM data. The proposed compression algorithm maintains three entities: non-zero elements (32-bit floating-point for each element11 For explanation of the proposed technique, 32-bit floating-point precision for IFM elements is used. However, the proposed technique can also be applied to the cases of using 16-bit fixed-point and 8-bit integer elements by applying quantization. ), indices (8-bit for each element), and count_table (one element for each chunk22 Although in this work, it is assumed that 256 elements correspond to one chunk, the chunk size depends on the design and can be extended. ). The first part, that is, non-zero element part, stores all NZEs in the IFMs. The second part, indices, indicates the location of NZEs in a chunk of IFM. The third part, count_table, indicates the accumulated number of NZEs from the first chunk to the current one. FIGURE 3Open in figure viewer The proposed two-step data compression technique. IFM, input feature map The proposed compression algorithm works as follows. Firstly, the 3D tensor IFM is converted into the 1D vector format, which is also divided into a granularity of the chunk (256 elements per chunk in this work). It then linearly searches for the NZE from the first chunk. If it finds the NZE in the chunk, it updates the indices by storing the location of the NZE in the chunk (i.e., IFM 1D vector index % 256). It also increases the number of the NZEs of the current chunk in the count_table. Also, it stores the element itself to non-zero elements. After it finishes the NZE search in a chunk, it starts to search for the NZEs in the next chunk and updates the cumulative number of the NZE in the next count_table entry. For example, as shown in Figure 3, since there are three NZEs in the first chunk (IFM[0]-IFM[255]), it records 3 in the count_table[0]. In the next chunk, it found another three NZEs, and thus records 6 (cumulative number) in the count_table[1]. Similarly, it records 8 in the count_table[2] corresponding to the third chunk as it founds two NZEs in the third chunk. Following the same procedure, it searches for the NZEs in all the chunks till it encounters the last element in IFM. Table 1 illustrates a comparison among the proposed two-step compression, conventional one-step compression that maintains NZEs and the corresponding indices (locations), and well-known compressed sparse row (CSR) compression. In Table 1, N ϕ ¯ denotes the number of NZEs and Nc denotes the number of chunks. Table 1 indicates that for the proposed two-step compression scheme, 4B (i.e., 32-bits) are required for storing an NZE, 1B is required for storing the index of an NZE in a chunk (since it performs index%256), and 2B are required to store the cumulative number of NZEs in a chunk. Table 1 also indicates that for the conventional one-step compression schemes, 4B are required for storing an NZE and 2B are required for storing the index of NZEs. The proposed two-step compression scheme leads to a better compression ratio than the conventional one under the condition of N ϕ ¯ > 2 × N c . Since the maximum allowable number of chunks (Nc) is 256 assuming that the maximum number of elements in the IFM is 216, the proposed two-step compression scheme leads to a better compression ratio as long as the sparsity of the IFMs is lower than 99.2% ((65,536−512)/65,536 = 99.2%), which is a common case.33 Since the proposed acceleration technique is geared toward resource-constrained systems, it is assumed that the maximum number of elements in the IFM is 216 (256 KB when using 4B floating-point format for an element). We also compare our proposed compression scheme with the CSR compression scheme. Note that we also assume the 256 × 256 matrix format where 216 elements (at maximum) can exist since the CSR scheme can only be used to compress the 2D matrix format. The CSR format requires a column index for each NZE and row pointers to maintain (in a cumulative manner) how many NZEs exist in each row. Thus, we need N ϕ ¯ × 1B (8-bits) for column indices because we need to represent 0–255 column indices. For the row pointers, we assume the worst-case requirements because the data size for row pointers depends on the sparsity and data distribution pattern in the matrix. Since the row pointer should record the cumulative number of NZEs until the corresponding row (i.e., from the first row to the corresponding row), the maximum value of the row pointer is 216, meaning that we require 16-bits (2B) for each row pointer value (technically, though we need 17-bits for the case where all the elements in the matrix are non-zero value, we exclude this case because it is a very rare case and not desirable to apply CSR compression). In addition, the number of required row pointer elements is the same as the number of rows in the matrix. Thus, 256 (28) row pointer values are required. Thus, for the CSR format, we need N ϕ ¯ × 4B + N ϕ ¯ × 1B + 28 × 2B. As we explained, since the maximum allowable number of chunks (Nc) of our proposed compression technique is 256 assuming that the maximum number of elements in the IFM is 216, our two-step compression technique shows a compression ratio comparable to the CSR format. TABLE 1. Compressed data size comparison between conventional one-step compression, CSR-based compression, and the proposed two-step compression assuming the maximum number of elements in the IFM is 216 Compression scheme Data stored Memory required Conventional one-step compression Data element + indices N ϕ ¯ × 4B + N ϕ ¯ × 2B Conventional CSR compression Data element + column indices + row pointers N ϕ ¯ × 4B + N ϕ ¯ × 1B + 256 × 2B Proposed two-step compression Data element + indices + Count_table N ϕ ¯ × 4B + N ϕ ¯ × 1B + Nc × 2B Note: N ϕ ¯ and Nc denote the number of non-zero elements and chunks, respectively. Abbreviation: CSR, compressed sparse row. On the other hand, the CSR format can only be applied to a 2D sparse matrix because it can only contain the column indices and row pointers, which are two-dimensional positional metadata. On the contrary, our compression technique can also be employed to compress the 3D tensors because we flatten the sparse tensor to a 1D vector before data compression. If we would like to use the CSR compression for a tensor (i.e., IFMs) compression, we can separately compress each 2D IFM and combine the compressed data of each IFM. However, when performing the compression with a large number of input channels, it may seriously increase the metadata (column indices and row pointers) size because the metadata size will be proportional to the number of input channels. In addition, depending on the IFM size (height and width) and the number of input channels, the metadata size will vary. It may hugely increase accelerator hardware complexity due to variable input dimensions. On the contrary, since our compression technique converts the 3D IFM tensor into a 1D vector format, our hardware accelerator operations are not affected by the tensor dimension, relieving the hardware design complexity. Since our acceleration technique is based on HW/SW co-design, we need a tradeoff between the compression ratio and hardware complexity. Even though some other compression techniques may be a little better in terms of the compression ratio, they can increase hardware complexity for decompression and convolution, deteriorating the overall performance. In our compression technique, we remove the zero values to reduce the required storage size. At the same time, the non-zero values still remain in the compressed data with their location information within the tensor. Although we remove the zero values in the compressed data, it is not actually removed from the tensor. In other words, if we restore (i.e., decompress) our compressed data to the original data, we can completely restore the tensor same as before compression (i.e., without any data loss). This is because our compressed data only contains the information on non-zero values (i.e., values and locations), whereas all the remaining spots of the tensor can be filled with zero values, meaning that our compression technique is lossless compression. 4.3 CNN accelerator for compressed IFMs 4.3.1 Hardware design overview Since the proposed acceleration technique compresses the IFMs, it is crucial to efficiently perform convolution operations with the compressed data format. The proposed hardware accelerator performs convolution operations with only NZEs and their location information (extracted from Indices and Count_table) and avoids the MAC operations with the zero operands,44 Though we compute the MAC operations with only NZEs, it does not hurt accuracy as compared to the case without our compression and acceleration technique because MAC operations with the zero values are ineffectual (i.e., N × 0 = 0). eventually resulting in performance and energy benefits proportional to the degree of input sparsity. Figure 4 describes the overall execution flow of the proposed hardware accelerator. FIGURE 4Open in figure viewer Overall architecture of our convolutional neural network accelerator. OFM stands for output feature map The first part of the proposed hardware accelerator is searching and weight matching. It brings the compressed IFMs (non-zero elements) with metadata (indices and count_table) and stores them in on-chip me