Energy-Constrained Compression for Deep Neural Networks via Weighted Sparse Projection and Layer Input Masking

1 INTRODUCTION . Deep Neural Networks ( DNNs ) have become the fundamental building blocks of many emerging application domains such as computer vision ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2014 ) , speech recognition ( Hinton et al. , 2012 ) , and natural language processing ( Goldberg , 2016 ) . Many of these applications have to operate in highly energy-constrained environments . For instance , autonomous drones have to continuously perform computer vision tasks ( e.g. , object detection ) without a constant power supply . Designing DNNs that can meet severe energy budgets has increasingly become a major design objective . The state-of-the-art model compression algorithms adopt indirect techniques to restrict the energy consumption , such as pruning ( or sparsification ) ( He et al. , 2018 ; Han et al. , 2015a ; Liu et al. , 2015 ; Zhou et al. , 2016 ; Li et al. , 2016 ; Wen et al. , 2016 ) and quantization ( Gong et al. , 2014 ; Wu et al. , 2016 ; Han et al. , 2015a ; Courbariaux et al. , 2015 ; Rastegari et al. , 2016 ) . These techniques are agonistic to energy consumption ; rather they are designed to reduce the amount of computations and the amount of model parameters in a DNN , which do not truly reflect the energy consumption of a DNN . As a result , these indirect approaches only indirectly reduce the total energy consumption . Recently , Energy-Aware Pruning ( EAP ) ( Yang et al. , 2017 ) proposes a more direct manner to reduce the energy consumption of DNN inferences by guiding weight pruning using DNN energy estimation , which achieves higher energy savings compared to the indirect techniques . However , a fundamental limitation of all existing methods is that they do not provide quantitative energy guarantees , i.e. , ensuring that the energy consumption is below a user-specified energy budget . In this paper , we aspire to answer the following key question : how to design DNN models that satisfy a given energy budget while maximizing the accuracy ? This work provides a solution to this question through an end-to-end training framework . By end-to-end , we refer to an approach that directly meets the energy budget without relying heuristics such as selectively restoring pruned weights and layer by layer fine-tuning ( Han et al. , 2015b ; Yang et al. , 2017 ) . These heuristics are effective in practice but also have many hyper-parameters that must be carefully tuned . Our learning algorithm directly trains a DNN model that meets a given energy budget while maximiz- ing model accuracy without incremental hyper-parameter tuning . The key idea is to formulate the DNN training process as an optimization problem in which the energy budget imposes a previously unconsidered optimization constraint . We integrate the quantitative DNN energy estimation into the DNN training process to assist the constrained optimization . In this way , a DNN model , once is trained , by design meets the energy budget while maximizing the accuracy . Without losing generality , we model the DNN energy consumption after the popular systolic array hardware architecture ( Kung , 1982 ) that is increasingly adopted in today ’ s DNN hardware chips such as Google ’ s Tensor Processing Unit ( TPU ) ( Jouppi et al. , 2017 ) , NVidia ’ s Tensor Cores , and ARM ’ s ML Processor . The systolic array architecture embodies key design principles of DNN hardware that is already available in today ’ s consumer devices . We specifically focus on pruning , i.e. , controlling the DNN sparsity , as the main energy reduction technique . Overall , the energy model models the DNN inference energy as a function of the sparsity of the layer parameters and the layer input . Given the DNN energy estimation , we formulate DNN training as an optimization problem that minimizes the accuracy loss under the constraint of a certain energy budget . The key difference between our optimization formulation and the formulation in a conventional DNN training is two-fold . First , our optimization problem considers the energy constraint , which is not present in conventional training . Second , layer inputs are non-trainable parameters in conventional DNN training since they depend on the initial network input . We introduce a new concept , called input mask , that enables the input sparsity to be controlled by a trainable parameter , and thus increases the energy reduction opportunities . This lets us further reduce energy in scenarios with known input data pattern . We propose an iterative algorithm to solve the above optimization problem . A key step in optimization is the projection operation onto the energy constraint , i.e. , finding a model which is closest to the given ( dense ) model and satisfies the energy constraint . We prove that this projection can be casted into a 0/1 knapsack problem and show that it can be solved very efficiently . Evaluation results show that our proposed training framework can achieve higher accuracy under the same or lower energy compared to the state-of-the-art energy-saving methods . In summary , we make the following contributions in this paper : • To the best of our knowledge , this is the first end-to-end DNN training framework that provides quantitative energy guarantees ; • We propose a quantitative model to estimate the energy consumption of DNN inference on TPU-like hardware . The model can be extended to model other forms of DNN hardware ; • We formulate a new optimization problem for energy-constrained DNN training and present a general optimization algorithm that solves the problem . 2 RELATED WORK . Energy-Agnostic Optimizations Most existing DNN optimizations indirectly optimize DNN energy through reducing the model complexity . They are agonistic to the energy consumption , and therefore can not provide any quantitative energy guarantees . Pruning , otherwise known as sparsification , is perhaps the most widely used technique to reduce DNN model complexity by reducing computation as well as hardware memory access . It is based on the intuition that DNN model parameters that have low-magnitude have little impact on the final prediction , and thus can be zeroed-out . The classic magnitude-based pruning ( Han et al. , 2015b ) removes weights whose magnitudes are lower than a threshold . Subsequent work guides pruning using special structures ( Liu et al. , 2015 ; Zhou et al. , 2016 ; Li et al. , 2016 ; Wen et al. , 2016 ; He et al. , 2017 ) , such as removing an entire channel , to better retain accuracy after pruning . Quantization reduces the number of bits used to encode model parameters , and thus reducing computation energy and data access energy ( Gong et al. , 2014 ; Wu et al. , 2016 ; Han et al. , 2015a ) . The extreme case of quantization is using 1-bit to represent model parameters ( Courbariaux et al. , 2015 ; Rastegari et al. , 2016 ) . Such binary quantization methods are usually trained from scratch instead of quantizing a pre-trained DNN . Energy-Aware Optimizations Recently , energy-aware pruning ( EAP ) ( Yang et al. , 2017 ) proposes to use a quantitative energy model to guide model pruning . Different from pure magnitude-based pruning methods , EAP selectively prunes the DNN layer that contributes the most to the total energy consumption . It then applies a sequence of fine-tuning techniques to retain model accuracy . The pruning step and fine-tuning step are alternated until the accuracy loss exceeds a given threshold . Although EAP a promising first-step toward energy-aware optimizations , its key limitation is that it does not provide quantitative energy guarantees because it does not explicitly consider energy budget as a constraint . Our work integrates the energy budget as an optimization constraint in model training . Latency-Guaranteed Compression Lately , model compression research has started providing guarantees in execution ( inference ) latency , which theoretically could be extended to providing energy guarantees as well . However , these methods are primarily search-based through either reinforcement learning ( He et al. , 2018 ) or greedy-search ( Yang et al. , 2018 ) . They search the sparsity setting for every single layer to meet the given budget . Thus , they may require a large number of trials to achieve a good performance , and may not ensure that the resulting model accuracy is maximized . 3 MODELING DNN INFERENCE ENERGY CONSUMPTION . This section introduces the model of estimating energy consumption of a single DNN inference . We consider the widely-used feed-forward DNNs . Note that our proposed methodology can be easily extended to other network architectures as well . In this section , we first provide an overview of our energy modeling methodology ( Section 3.1 ) . We then present the detailed per-layer energy modeling ( Section 3.2 and Section 3.3 ) , which allow us to then derive the overall DNN energy consumption ( Section 3.4 ) . Our energy modeling results are validated against the industry-strength DNN hardware simulator ScaleSim ( Samajdar et al. , 2018 ) . DNN model sparsity ( via pruning ) is well recognized to significantly affect the execution efficiency and thus affect the energy consumption of a DNN model ( He et al. , 2018 ; Yang et al. , 2017 ; Han et al. , 2015a ; Liu et al. , 2015 ; Zhou et al. , 2016 ) . We thus use pruning as the mechanism to reduce energy consumption1 . Note , however , that model sparsity is not the end goal of our paper ; rather we focus on reducing the energy consumption directly . Many dedicated DNN hardware chips ( a.k.a. , Neural Processing Units , NPUs ) ( Jouppi et al. , 2017 ; Chen et al. , 2016 ; Han et al. , 2016 ; Parashar et al. , 2017 ) have been developed to directly benefit from model sparsity , and are already widely available in today ’ s consumer devices such as Apple iPhoneX , Huawei Mate 10 , and Microsoft HoloLens . Our paper focuses on this general class of popular , widely-used DNN chips . 3.1 ENERGY MODELING OVERVIEW . A DNN typically consists of a sequence of convolution ( CONV ) layers and fully connected ( FC ) layers interleaved with a few other layer types such as Rectified Linear Unit ( ReLU ) and batch normalization . We focus mainly on modeling the energy consumption of the CONV and FC layers . This is because CONV and FC layers comprise more than 90 % of the total execution time during a DNN inference ( Chen et al. , 2016 ) and are the major energy consumers ( Han et al. , 2015a ; Yang et al. , 2017 ) . Energy consumed by other layer types is insignificant and can be taken away from the energy budget as a constant factor . A DNN inference ’ s energy consumption is tied to the underlying hardware that performs the inference . In particular , we assume a systolic-array-based DNN hardware architecture . Systolic array ( Kung , 1982 ) has long been know as an effective approach for matrix multiplication . Many DNN hardware architectures adopt the systolic array , most notably the Google Tensor Processing Unit ( TPU ) ( Jouppi 1Quantization is another useful mechanism to reduce energy consumption . It is orthogonal to the pruning mechanism and they could be combined . This paper specifically focuses on the pruning mechanism . et al. , 2017 ) , Nvidia ’ s Tensor Cores in their most recent Volta GPUs , and ARM ’ s ML Processor . Targeting systolic-array-based DNN hardware ensures that our approach has a wide applicability . However , our modeling and training strategies can generally be applied to other DNN architectures . Figure 1 shows the overall hardware architecture . The systolic array comprises of several compute units that perform the Multiply-and-Accumulate ( MAC ) operation , which conducts the following computation : a← a+ ( b× c ) , where b and c are the two scalar inputs and a is the scalar intermediate result called “ partial sum. ” MAC operation is the building block for matrix multiplication . The MAC units are organized in a 2-D fashion . The data is fed from the edges , both horizontally and vertically , which then propagate to the MAC units within the same row and columns . We decompose the energy cost into two parts : computation energy Ecomp and data access energy Edata . Ecomp denotes the energy consumed by computation units , and Edata denotes the energy consumed when accessing data from the hardware memory . Since we mainly use pruning as the energy reduction technique , we now model how Ecomp and Edata are affected by DNN sparsity .

Summary: The paper proposes a method for neural network training under a hard energy constraint (i.e. the method guarantees the energy consumption to be upper bounded). Based on a systolic array hardware architecture the authors model the energy consumption of transferring the weights and activations into different levels of memory (DRAM, Cache, register file) during inference. The energy consumption is therefore determined by the number of nonzero elements in the weight and activation tensors. To minimize the network loss under an energy constraint, the authors develop a training framework including a novel greedy algorithm to compute the projection of the weight tensors to the energy constraint.