Feature Map Transform Coding for Energy-Efficient CNN Inference

F E AT U R E M A P T R A N S F O R M C O D I N G F O R E N E R G Y- E F F I C I E N T C N N I N F E R E N C E Anonymous authors Paper under double-blind review A B S T R A C T Convolutional neural networks ( CNNs ) achieve state-of-the-art accuracy in a variety of tasks in computer vision and beyond . One of the major obstacles hindering the ubiquitous use of CNNs for inference on low-power edge devices is their high computational complexity and memory bandwidth requirements . The latter often dominates the energy footprint on modern hardware . In this paper , we introduce a lossy transform coding approach , inspired by image and video compression , designed to reduce the memory bandwidth due to the storage of intermediate activation calculation results . Our method does not require fine-tuning the network weights and halves the data transfer volumes to the main memory by compressing feature maps , which are highly correlated , with variable length coding . Our method outperform previous approach in term of the number of bits per value with minor accuracy degradation on ResNet-34 and MobileNetV2 . We analyze the performance of our approach on a variety of CNN architectures and demonstrate that FPGA implementation of ResNet-18 with our approach results in a reduction of around 40 % in the memory energy footprint , compared to quantized network , with negligible impact on accuracy . When allowing accuracy degradation of up to 2 % , the reduction of 60 % is achieved . A reference implementation accompanies the paper . 1 Introduction . Deep neural networks have established themselves as the first-choice tool for a wide range of applications . Neural networks have shown phenomenal results in a variety of tasks in a broad range of fields such as computer vision , computational imaging , and image and language processing . Despite deep neural models impressive performance , the computation and computational requirements are substantial for both training and inference phases . So far , this fact has been a major obstacle for the deployment of deep neural models in applications constrained by memory , computational , and energy resources , such as those running on embedded systems . To alleviate the energy cost , custom hardware for neural network inference , including FPGAs and ASICs , is actively being developed in recent years . In addition to providing better energy efficiency per arithmetic operation , custom hardware offers more flexibility in various strategies to reduce the computational and storage complexity of the model inference , for example by means of quantization ( Baskin et al. , 2018 ; Hubara et al. , 2018 ; Jacob et al. , 2018 ) and pruning ( Han et al. , 2016 ; Louizos et al. , 2018 ; Theis et al. , 2018 ) . In particular , quantization to very low precision is especially efficient on custom hardware where arbitrary precision arithmetic operations require proportional resources . To prevent accuracy degradation , many approaches have employed training the model with quantization constraints or modifying the network structure . A recent study ( Yang et al. , 2017 ) has shown that almost 70 % of the energy footprint on such hardware is due to data movement to and from the off-chip memory . Amounts of data typically need to be transferred to and from the RAM and back during the forward pass through each layer , since the local memory is too small to store all the feature maps . By reducing the number of bits representing these data , existing quantization techniques reduce the memory bandwidth considerably . However , to the best of our knowledge , none of these methods exploit the high amount of interdependence between the feature maps and spatial locations of the compute activations . Contributions . In this paper , we propose a novel scheme based on transform-domain quantization of the neural network activations followed by lossless variable length coding . We demonstrate that this approach reduces memory bandwidth by 40 % when applied in the post-training regime ( i.e. , without fine-tuning ) with small computational overhead and no accuracy degradation . Relaxing the accuracy requirements increases bandwidth savings to 60 % . Moreover , we outperform previous methods in term of number bit per value with minor accuracy degradation . A detailed evaluation of various ingredients and parameters of the proposed method is presented . We also demonstrate a reference hardware implementation that confirms a reduction in memory energy consumption during inference . 2 Related work . Quantization . Low-precision representation of the weights and activations is a common means of reducing computational complexity . On appropriate hardware , this typically results in the reduction of the energy footprint as well . It has been demonstrated that in standard architectures quantization down to 16 ( Gupta et al. , 2015 ) or 8 bits ( Jacob et al. , 2018 ; Lee et al. , 2018 ; Yang et al. , 2019 ) per parameter is practically harmless . However , further reduction of bitwidth requires non-trivial techniques ( Mishra et al. , 2018 ; Zhang et al. , 2018 ) , often with adverse impact on training complexity . Lately , the quantization of weights and activations of neural networks to 2 bits or even 1 ( Rastegari et al. , 2016 ; Hubara et al. , 2018 ) has attracted the attention of many researchers . While the performance of binary ( i.e. , 1-bit ) neural networks still lags behind their full-precision counterparts ( Ding et al. , 2019 ; Peng & Chen , 2019 ) , existing quantization methods allow 2-4 bit quantization with a negligible impact on accuracy ( Choi et al. , 2018b ; a ; Dong et al. , 2019 ) . Quantizing the neural network typically requires introducing the quantizer model at the training stage . However , in many applications the network is already trained in full precision , and there is no access to the training set to configure the quantizer . In such a post-training regime , most quantization methods employ statistical clamping , i.e. , the choice of quantization bounds based on the statistics acquired in a small test set . Migacz ( 2017 ) proposed using a small calibration set to gather activation statistics and then randomly searching for a quantized distribution that minimizes the KullbackLeibler divergence to the continuous one . Gong et al . ( 2018 ) , on the other hand , used the L∞ norm of the tensor as a threshold . Lee et al . ( 2018 ) employed channel-wise quantization and constructed a dataset of parametric probability densities with their respective quantized versions ; a simple classifier was trained to select the best fitting density . Banner et al . ( 2018 ) derived an approximation of the optimal threshold under the assumption of Laplacian or Gaussian distribution of the weights , which achieved single-percentage accuracy reduction for 8-bit weights and 4-bit activations . Meller et al . ( 2019 ) showed that the equalization of channels and removal of outliers improved quantization quality . Choukroun et al . ( 2019 ) used one-dimensional line-search to evaluate an optimal quantization threshold , demonstrating state-of-the-art results for 4-bit weight and activation quantization . Influence of memory access on energy consumption . Yang et al . ( 2017 ) studied the breakdown of energy consumption in CNN inference . For example , in GoogLeNet ( Szegedy et al. , 2015 ) arithmetic operations consume only 10 % of the total energy , while feature map transfers to and from an external RAM amount to about 68 % of the energy footprint . However , due to the complexity of real memory systems , not every method that decreases the sheer memory bandwidth will necessarily yield significant improvement in power consumption . In particular , it depends on computational part optimization : while memory performance is mainly defined by the external memory chip , better optimization of computations will lead to higher relative energy consumption of the memory . For example , while Ansari & Ogunfunmi ( 2018 ) reported a 70 % bandwidth reduction , the dynamic power consumption decreased by a mere 2 % . Xiao et al . ( 2017 ) proposed fusing convolutional layers to reduce the transfer of feature maps . In an extreme case , all layers are fused into a single group . A similar approach was adopted by Xing et al . ( 2019 ) , who demonstrated a hardware design that does not write any intermediate results into the off-chip memory . This approach achieved approximately 15 % runtime improvement for ResNet and state-of-the-art throughput . However , the authors did not compare the energy footprint of the design with the baseline . Morcel et al . ( 2019 ) demonstrated that using on-chip cache cuts down the memory bandwidth and thus reduces power consumption by an order of magnitude . In addition , Jouppi et al . ( 2017 ) noted that not only the power consumption but also the speed of DNN accelerators is memoryrather than compute-bound . This is confirmed by Wang et al . ( 2019 ) , who also demonstrated that increasing computation throughput without increasing memory bandwidths barely affects latency . Network compression . Lossless coding and , in particular , variable length coding ( VLC ) , is a way to reduce the memory footprint without compromising performance . In particular , Han et al . ( 2016 ) proposed using Huffman coding to compress network weights , alongside quantization and pruning . Wijayanto et al . ( 2019 ) proposed using the more computationally-demanding algorithm DEFLATE ( LZ77 + Huffman ) to further improve compression rates . Chandra ( 2018 ) used Huffman coding to compress feature maps and thus reduce memory bandwidth . Gudovskiy et al . ( 2018 ) proposed passing only the lower-dimensional feature maps to the memory to reduce the bandwidth . Cavigelli & Benini ( 2019 ) proposed using RLE-based algorithm to compress sparse network activations . 3 Transform-Domain Compression . In what follows , we briefly review the basics of lossy transform coding . For a detailed overview of the subject , we refer the reader to Goyal ( 2001 ) . Let x = ( x1 , . . . , xn ) represent the values of the activations of a NN layer in a block of size n = W × H × C spanning , respectively , the horizontal and the vertical dimensions and the feature channels . Prior to being sent to memory , the activations , x , are encoded by first undergoing an affine transform , y = T x = T ( x− µ ) ; the transform coefficients are quantized by a scalar quantizer , Q∆ , whose strength is controlled by the step size ∆ , and subsequently coded by a lossless variable length coder ( VLC ) . We refer to the length in bits of the resulting code , normalized by n as to the average rate , R∆ . To decode the activation vector , a variable length decoder ( VLD ) is applied first , followed by the inverse quantizer and the inverse transform . The resulting decoded activation , x̂ = T −1Q−1∆ ( Q∆ ( T x ) ) , typically differs from x ; the discrepancy is quantified by a distortion , D∆ . The functional relation between the rate and the distortion is controlled by the quantization strength , ∆ , and is called rate-distortion curve . Classical rate-distortion analysis in information theory assumes the MSE distortion , D = 1n‖x− x̂‖ 2 2 . While in our case the measure of distortion is the impact on the task-specific loss , we adopt the Euclidean distortion for two reasons : firstly , it is well-studied and leads to simple expressions for the quantizer ; and , secondly , computing loss requires access to the training data , which are unavailable in the post-training regime . The derivation of an optimal rate is provided in Appendix D. A crucial observation justifying transform coding is the fact that significant statistical dependence usually exists between the xi ( Cogswell et al. , 2016 ) . We model this fact by asserting that the activations are jointly Gaussian , x ∼ N ( µ , Σ ) , with the covariance matrix Σ , whose diagonal elements are denoted by σ2i . Statistical dependence corresponds to non-zero off-diagonal elements in Σ . The affinely transformed y = T ( x − µ ) is also Gaussian with the covariance matrix Σ′ = TTΣT . The distortion ( D.2 ) is minimized over orthonormal matrices by T = Σ−1/2 diagonalizing the covariance ( Goyal , 2001 ) . The latter is usually referred to as the Karhunen-Loeve transform ( KLT ) or principal component analysis ( PCA ) . The corresponding minimum distortion is D∗ ( R ) = πe 6 det ( Σ ) 1/n2−2R . Since the covariance matrix is symmetric , T is orthonormal , implying T−1 = TT . In Fig . 1 , a visualization of 2D vector quantization is shown . For correlated channels ( Fig . 1a ) , many 2D quantization bins are not used since they contain no values . Linear transformation ( Fig . 1b ) provides improved quantization error for correlated channels by getting rid of those empty bins .

Summary: A lossy transform coding approach was proposed to reduce the memory bandwidth of edge devices deploying CNNs. For this purpose, the proposed method compresses highly correlated feature maps using variable length coding. In the experimental analyses, the proposed method outperforms some of the previous work in terms of the compression ratio and accuracy for training ResNet-34 and MobileNetV2.