Integral Pruning on Activations and Weights for Efficient Neural Networks

With the rapidly scaling up of deep neural networks ( DNNs ) , extensive research studies on network model compression such as weight pruning have been performed for efficient deployment . This work aims to advance the compression beyond the weights to the activations of DNNs . We propose the Integral Pruning ( IP ) technique which integrates the activation pruning with the weight pruning . Through the learning on the different importance of neuron responses and connections , the generated network , namely IPnet , balances the sparsity between activations and weights and therefore further improves execution efficiency . The feasibility and effectiveness of IPnet are thoroughly evaluated through various network models with different activation functions and on different datasets . With < 0.5 % disturbance on the testing accuracy , IPnet saves 71.1 % ∼ 96.35 % of computation cost , compared to the original dense models with up to 5.8× and 10× reductions in activation and weight numbers , respectively . The source codes are available at ( omitted for blind review ) . 1 INTRODUCTION . Deep neural networks ( DNNs ) have demonstrated significant advantages in many real-world applications , such as image classification , object detection and speech recognition ( He et al. , 2016 ; Redmon et al. , 2016 ; Sainath & Parada , 2015 ) . On the one hand , DNNs are developed for improving performance in these applications , which leads to intensive demands in data storage , communication and processing . On the other hand , the ubiquitous intelligence promotes the deployment of DNNs in light-weight embedded systems that are equipped with only limited memory and computation resource . To reduce the model size while ensuring the performance quality , DNN pruning is widely explored . Redundant weight parameters are removed by zeroing-out those in small values ( Han et al. , 2015 ; Park et al. , 2016 ) . Utilizing the zero-skipping technique ( Han et al. , 2016 ) on sparse weight parameters can further save the computation cost . In addition , many specific DNN accelerator designs ( Albericio et al. , 2016 ; Reagen et al. , 2016 ) leveraged the intrinsic zero-activation pattern of the rectified linear unit ( ReLU ) to realize the activation sparsity . The approach , however , can not be directly extended to other activation functions , e.g. , leaky ReLU . Although these techniques achieved tremendous success , pruning only the weights or activations can not lead to the best inference speed , which is a crucial metric in DNN deployment , for the following reasons . First , the existing weight pruning methods mainly focus on the model size reduction . However , the most essential challenge of speeding up DNNs is to minimize the computation cost , such as the intensive multiple-and-accumulate operations ( MACs ) . Particularly , the convolution ( conv ) layers account for most of the computation cost and dominate the inference time in DNNs ( Park et al. , 2016 ) . Because weights are shared in convolution , the execution speed of conv layers is usually bounded by computation instead of memory accesses ( Jouppi et al. , 2017 ; Zhang et al. , 2015 ) . Second , the activation in DNNs is not strictly limited with ReLU . The intrinsic zeroactivation patterns do not exist in non-ReLU activation functions , such as leaky ReLU and sigmoid . Third , the weights and activations of a network together determine the network performance . Our experiment shows that the zero-activation percentage obtained by ReLU decreases after applying the weight pruning ( Han et al. , 2016 ) . Such a deterioration in activation sparsity could potentially eliminate the advantage of the aforementioned accelerator designs . In this work , we propose the integral pruning ( IP ) technique to minimize the computation cost of DNNs by pruning both weights and activations . As the pruning processes for weights and activations are correlated , IP learns dynamic activation masks by attaching activation pruning to weight pruning after static weight masks are well trained . Through the learning on the different importance of neuron responses and connections , the generated network , namely IPnet , balances the sparsity between activations and weights and therefore further improves execution efficiency . Moreover , our method not only stretches the intrinsic activation sparsity of ReLU , but also targets as a general approach for other activation functions , such as leaky ReLU . Our experiments on various network models with different activation functions and on different datasets show substantial reduction in MACs by the proposed IPnet . Compared to the original dense models , IPnet can obtain up to 5.8× activation compression rate , 10× weight compression rate and eliminate 71.1 % ∼ 96.35 % of MACs . Compared to state-of-the-art weight pruning technique ( Han et al. , 2015 ) , IPnet can further reduce the computation cost 1.2× ∼ 2.7× . 2 RELATED WORKS . Weight Pruning : The weight pruning emerges as an effective compression technique in reducing the model size and computation cost of neural networks . A common approach of pruning the redundant weights in DNN training is to include an extra regularization term ( e.g. , the ` 1-normalization ) in the loss function ( Liu et al. , 2015 ; Park et al. , 2016 ) to constrain the weight distribution . Then the weights below a heuristic threshold will be pruned . Afterwards , a certain number of finetuning epochs will be applied for recovering the accuracy loss due to the pruning . In practice , the directpruning and finetuning stages can be carried out iteratively to gradually achieve the optimal trade-off between the model compression rate and accuracy . Such a weight pruning approach demonstrated very high effectiveness , especially for fully-connected ( fc ) layers ( Han et al. , 2015 ) . For conv layers , removing the redundant weights in structured forms , e.g. , the filters and filter channels , has been widely investigated . For example , Wen et al . ( 2016 ) proposed to apply group Lasso regularization on weight groups in a variety of self-defined sizes and shapes to remove redundant groups . Molchanov et al . ( 2016 ) used the first-order Taylor series expansion of the loss function on feature maps to determine the rankings of filters and those in low ranking will be removed . The filter ranking can also be represented by the root mean square or the sum of absolute values of filter weights ( Mao et al. , 2017 ; Yu et al. , 2017 ) . Activation Sparsity : The activation sparsity has been widely utilized in various DNN accelerator designs . Chen et al . ( 2016 ) , Albericio et al . ( 2016 ) and Reagen et al . ( 2016 ) accelerated the DNN inference with reduced off-chip memory access and computation cost benefiting from the sparse activations originated from ReLU . A simple technique to improve activation sparsity by zeroing out small activations was also explored ( Albericio et al. , 2016 ) . However , the increment of activation sparsity is still limited without accuracy loss . The biggest issue in the aforementioned works is that they heavily relied on ReLU . However , zero activations do not exist in non-ReLU activation function . To regulate and stretch the activation sparsity , many dropout-based methods are proposed . Adaptive dropout ( Ba & Frey , 2013 ) , for instance , developed a binary belief network overlaid on the original network . The neurons with larger activation magnitude incur higher probability to be activated . Although this method achieved a better regularization on DNNs , the inclusion of belief network complicated the training and had no help on inference speedup . The winners-take-all ( WTA ) autoencoder was built with a regularization based on activation magnitude to learn deep sparse representations from various datasets ( Makhzani & Frey ( 2015 ) ) . As can be seen that the model size compression is the main focus of weight pruning , while the use of activation sparsification focuses more on the intrinsic activation sparsity by ReLU or exploring the virtue of sparse activation in the DNN training for better model generalization . In contrast , our proposed IP aims for reducing the DNN computation cost and therefore accelerating the inference by integrating and optimizing both weight pruning and activation sparsification . 3 APPROACH . As depicted in Figure 1 , the proposed IP consists of two steps by concatenating the activation pruning to the weight pruning . Both stages seek for unimportant information ( weights and activations , respectively ) and mask them off . We aim to keep only the important connections and activations to minimize the computation cost . In this section , we will first explain the integration of the two steps . The technical details in model quality ( e.g. , accuracy ) control will then be introduced . The prediction method for deriving activation masks is also proposed to speed up the inference of IPnets . At last , the appropriate settings of dropout layers and training optimizers are discussed . 3.1 INTEGRATION OF WEIGHT PRUNING AND ACTIVATION PRUNING . Weight pruning . In the weight pruning stage , weight parameters with magnitude under a threshold are masked out , and weight masks will be passed to the following finetuning process . After the model is finetuned for certain epochs to recover accuracy loss , weight masks need to be updated for the next finetuning round . There are two crucial techniques to help weight pruning . 1 ) The threshold used to build weight masks are determined based on the weight distribution of each layer . Because of different sensitivity for weight pruning , each layer owns a specific weight sparsity pattern . Basically , the leading several conv layers are more vulnerable to weight pruning . 2 ) The whole weight pruning stage needs multiple pruning-finetuning recursions to search an optimal weight sparsity . Weight masks are progressively updated to increase pruning strength . Activation pruning . While weak connections between layers are learned to be pruned , activaitons with small magnitude are taken as unimportant and can be masked out to further minimize interlayer connections , and hence to reduce computation cost . Notice that , neurons in DNNs are trained to be activated in various patterns according to different input classes , thus dynamic masks should be learned in the activation pruning stage , which are different from the static masks in the weight pruning stage . The selected activations by the dynamic mask are denoted as winners , and the winner rate is defined as : Winner rate = Swinner Stotal , ( 1 ) where Swinner and Stotal denote the number of winners and total activation number . The winner rate per layer is determined by the analysis of activation pruning sensitivity layer-wise on the models obtained after weight pruning . The winner activation after the pruning mask , Am , obeys the rule : Am = { Aorig , if |Aorig| > θ 0 , otherwise ( 2 ) where θ is the threshold derived at run-time from the activation winner rate for each layer , andAorig is the result from original activation function . Same with weight pruning , the model with dynamic activation masks is finetuned to recover accuracy drop . No iterative procedure of mask updating and finetuning is required in our activaiton pruning method . 3.2 WINNER RATE SETTINGS . Not all layers share the same winner rate . Similar to the trend in weight pruning , deeper layers tolerate larger activation pruning strength . To analyze the activation pruning sensitivity , the model with activation masks is tested on a validation set sampled from the training images with the same size as the testing set . Accuracy drops are taken as the indicator of pruning sensitivity for different winner rate settings . Before finetuning , the activation winner rate per layer is set empirically to keep accuracy drop less than 2 % . For the circumstances that model accuracy is resistant to be tuned back , winner rates in the leading several layers should be set smaller . Examples of sensitivity analysis will be given and discussed in Section 5 .

Summary: This article presents a novel approach called Integral Pruning (IP) to reduce the computation cost of Deep neural networks (DNN) by integrating activation pruning along with weight pruning. The authors show that common techniques of exclusive weight pruning does compress the model size, but increases the number of non-zero activations after ReLU. This would counteract the advantage of DNN accelerator designs (Albericio et al., 2016; Reagen et al., 2016) that leverage activation sparsity to speed up the computations. IP starts with pruning the weights using an existing technique to mask out weights under a threshold and then fine-tune the network in an iterative fashion to maintain the accuracy. After weight pruning, IP further masks out the activations with smaller magnitude to reduce the computation cost. Unlike weight pruning techniques that use static masks, the authors propose to use dynamic activation masks for activation sparsity in order to account for various patterns that are being activated in DNN for different input samples. In order to do this, the 'winner rate' measure for every layer (or for a group of layers in deep networks like ResNet32) is defined, to dynamically set the threshold for the generation of activation masks which eventually controls the amount of non-zero activation entries. The article empirically analyzes the sensitivity of activation pruning on validation data by setting different winner rates at every layer in DNN and decides upon a set of winner rates accordingly followed by an iteration of fine-tuning the network to maintain its performance. The authors show that their technique produced lower number of non-zero activations in comparison with the intrinsic sparse ReLU activations and weight pruning techniques.