Title: INTEGRAL PRUNING ON ACTIVATIONS AND WEIGHTS FOR EFFICIENT NEURAL NETWORKS

Abstract: With the rapidly scaling up of deep neural networks (DNNs), extensive research studies on network model compression such as weight pruning have been performed for efficient deployment. This work aims to advance the compression beyond the weights to the activations of DNNs. We propose the Integral Pruning (IP) technique which integrates the activation pruning with the weight pruning. Through the learning on the different importance of neuron responses and connections, the generated network, namely IPnet, balances the sparsity between activations and weights and therefore further improves execution efficiency. The feasibility and effectiveness of IPnet are thoroughly evaluated through various network models with different activation functions and on different datasets. With < 0.5% disturbance on the testing accuracy, IPnet saves 71.1% ∼ 96.35% of computation cost, compared to the original dense models with up to 5.8× and 10× reductions in activation and weight numbers, respectively. The source codes are available at (omitted for blind review).

1. INTRODUCTION:

Deep neural networks (DNNs) have demonstrated significant advantages in many real-world applications, such as image classification, object detection and speech recognition (He et al., 2016; Redmon et al., 2016; Sainath & Parada, 2015) . On the one hand, DNNs are developed for improving performance in these applications, which leads to intensive demands in data storage, communication and processing. On the other hand, the ubiquitous intelligence promotes the deployment of DNNs in light-weight embedded systems that are equipped with only limited memory and computation resource. To reduce the model size while ensuring the performance quality, DNN pruning is widely explored. Redundant weight parameters are removed by zeroing-out those in small values (Han et al., 2015; Park et al., 2016) . Utilizing the zero-skipping technique (Han et al., 2016) on sparse weight parameters can further save the computation cost. In addition, many specific DNN accelerator designs (Albericio et al., 2016; Reagen et al., 2016) leveraged the intrinsic zero-activation pattern of the rectified linear unit (ReLU) to realize the activation sparsity. The approach, however, cannot be directly extended to other activation functions, e.g., leaky ReLU.

Although these techniques achieved tremendous success, pruning only the weights or activations cannot lead to the best inference speed, which is a crucial metric in DNN deployment, for the following reasons. First, the existing weight pruning methods mainly focus on the model size reduction. However, the most essential challenge of speeding up DNNs is to minimize the computation cost, such as the intensive multiple-and-accumulate operations (MACs). Particularly, the convolution (conv) layers account for most of the computation cost and dominate the inference time in DNNs (Park et al., 2016) . Because weights are shared in convolution, the execution speed of conv layers is usually bounded by computation instead of memory accesses (Jouppi et al., 2017; Zhang et al., 2015) . Second, the activation in DNNs is not strictly limited with ReLU. The intrinsic zeroactivation patterns do not exist in non-ReLU activation functions, such as leaky ReLU and sigmoid. Third, the weights and activations of a network together determine the network performance. Our experiment shows that the zero-activation percentage obtained by ReLU decreases after applying the weight pruning (Han et al., 2016) . Such a deterioration in activation sparsity could potentially eliminate the advantage of the aforementioned accelerator designs.

In this work, we propose the integral pruning (IP) technique to minimize the computation cost of DNNs by pruning both weights and activations. As the pruning processes for weights and activations are correlated, IP learns dynamic activation masks by attaching activation pruning to weight pruning after static weight masks are well trained. Through the learning on the different importance of neuron responses and connections, the generated network, namely IPnet, balances the sparsity between activations and weights and therefore further improves execution efficiency. Moreover, our method not only stretches the intrinsic activation sparsity of ReLU, but also targets as a general approach for other activation functions, such as leaky ReLU. Our experiments on various network models with different activation functions and on different datasets show substantial reduction in MACs by the proposed IPnet. Compared to the original dense models, IPnet can obtain up to 5.8× activation compression rate, 10× weight compression rate and eliminate 71.1% ∼ 96.35% of MACs. Compared to state-of-the-art weight pruning technique (Han et al., 2015) , IPnet can further reduce the computation cost 1.2× ∼ 2.7×.

2. RELATED WORKS:

Weight Pruning: The weight pruning emerges as an effective compression technique in reducing the model size and computation cost of neural networks. A common approach of pruning the redundant weights in DNN training is to include an extra regularization term (e.g., the 1 -normalization) in the loss function (Liu et al., 2015; Park et al., 2016) to constrain the weight distribution. Then the weights below a heuristic threshold will be pruned. Afterwards, a certain number of finetuning epochs will be applied for recovering the accuracy loss due to the pruning. In practice, the directpruning and finetuning stages can be carried out iteratively to gradually achieve the optimal trade-off between the model compression rate and accuracy. Such a weight pruning approach demonstrated very high effectiveness, especially for fully-connected (fc) layers (Han et al., 2015) . For conv layers, removing the redundant weights in structured forms, e.g., the filters and filter channels, has been widely investigated. For example, Wen et al. (2016) proposed to apply group Lasso regularization on weight groups in a variety of self-defined sizes and shapes to remove redundant groups. Molchanov et al. (2016) used the first-order Taylor series expansion of the loss function on feature maps to determine the rankings of filters and those in low ranking will be removed. The filter ranking can also be represented by the root mean square or the sum of absolute values of filter weights (Mao et al., 2017; Yu et al., 2017) .

Activation Sparsity: The activation sparsity has been widely utilized in various DNN accelerator designs. Chen et al. (2016) , Albericio et al. (2016) and Reagen et al. (2016) accelerated the DNN inference with reduced off-chip memory access and computation cost benefiting from the sparse activations originated from ReLU. A simple technique to improve activation sparsity by zeroing out small activations was also explored (Albericio et al., 2016) . However, the increment of activation sparsity is still limited without accuracy loss. The biggest issue in the aforementioned works is that they heavily relied on ReLU. However, zero activations do not exist in non-ReLU activation function. To regulate and stretch the activation sparsity, many dropout-based methods are proposed. Adaptive dropout (Ba & Frey, 2013) , for instance, developed a binary belief network overlaid on the original network. The neurons with larger activation magnitude incur higher probability to be activated. Although this method achieved a better regularization on DNNs, the inclusion of belief network complicated the training and had no help on inference speedup. The winners-take-all (WTA) autoencoder was built with a regularization based on activation magnitude to learn deep sparse representations from various datasets (Makhzani & Frey (2015) ).

As can be seen that the model size compression is the main focus of weight pruning, while the use of activation sparsification focuses more on the intrinsic activation sparsity by ReLU or exploring the virtue of sparse activation in the DNN training for better model generalization. In contrast, our proposed IP aims for reducing the DNN computation cost and therefore accelerating the inference by integrating and optimizing both weight pruning and activation sparsification.

3. APPROACH:

As depicted in Figure 1 , the proposed IP consists of two steps by concatenating the activation pruning to the weight pruning. Both stages seek for unimportant information (weights and activations, respectively) and mask them off. We aim to keep only the important connections and activations to minimize the computation cost. In this section, we will first explain the integration of the two steps. The technical details in model quality (e.g., accuracy) control will then be introduced. The prediction method for deriving activation masks is also proposed to speed up the inference of IPnets. At last, the appropriate settings of dropout layers and training optimizers are discussed.

3.1. INTEGRATION OF WEIGHT PRUNING AND ACTIVATION PRUNING:

Weight pruning. In the weight pruning stage, weight parameters with magnitude under a threshold are masked out, and weight masks will be passed to the following finetuning process. After the model is finetuned for certain epochs to recover accuracy loss, weight masks need to be updated for the next finetuning round. There are two crucial techniques to help weight pruning. 1) The threshold used to build weight masks are determined based on the weight distribution of each layer. Because of different sensitivity for weight pruning, each layer owns a specific weight sparsity pattern. Basically, the leading several conv layers are more vulnerable to weight pruning. 2) The whole weight pruning stage needs multiple pruning-finetuning recursions to search an optimal weight sparsity. Weight masks are progressively updated to increase pruning strength. Activation pruning. While weak connections between layers are learned to be pruned, activaitons with small magnitude are taken as unimportant and can be masked out to further minimize interlayer connections, and hence to reduce computation cost. Notice that, neurons in DNNs are trained to be activated in various patterns according to different input classes, thus dynamic masks should be learned in the activation pruning stage, which are different from the static masks in the weight pruning stage. The selected activations by the dynamic mask are denoted as winners, and the winner rate is defined as:

Weight pruning:

EQUATION

where S winner and S total denote the number of winners and total activation number. The winner rate per layer is determined by the analysis of activation pruning sensitivity layer-wise on the models obtained after weight pruning. The winner activation after the pruning mask, A m , obeys the rule:

EQUATION

where θ is the threshold derived at run-time from the activation winner rate for each layer, and A orig is the result from original activation function. Same with weight pruning, the model with dynamic activation masks is finetuned to recover accuracy drop. No iterative procedure of mask updating and finetuning is required in our activaiton pruning method.

3.2. WINNER RATE SETTINGS:

Not all layers share the same winner rate. Similar to the trend in weight pruning, deeper layers tolerate larger activation pruning strength. To analyze the activation pruning sensitivity, the model with activation masks is tested on a validation set sampled from the training images with the same size as the testing set. Accuracy drops are taken as the indicator of pruning sensitivity for different winner rate settings. Before finetuning, the activation winner rate per layer is set empirically to keep accuracy drop less than 2%. For the circumstances that model accuracy is resistant to be tuned back, winner rates in the leading several layers should be set smaller. Examples of sensitivity analysis will be given and discussed in Section 5.

3.3. THRESHOLD PREDICTION IN ACTIVATION PRUNING:

The dynamic activation pruning method increases the activation sparsity and maintains the model accuracy as well. The solution of determining threshold θ in Equation ( 2) for activation masks is actually a canonical argpartion problem to find top-k arguments in an array. According to the Master Theorem (Bentley et al., 1980) , argpartition can be fast solved in linear time O(N ) through recursive algorithms, where N is the number of elements to be partitioned. To further speed up, threshold prediction can be applied on the down-sampled activation set. An alternate threshold θ is predicted by selecting top-αk elements from the down-sampled activation set comprising αN elements with α as the down-sampling rate. θ is applied for the original activation set afterwards.

3.4. DROPOUT LAYER WITH ACTIVATION PRUNING:

For DNN training, dropout layer is commonly added after large fc layers to avoid over-fitting problem. The neuron activations are randomly chosen in the feed-forward phase, and weights updates will be only applied on the neurons associated with the selected activations in the back-propagation phase. Thus, a random partition of weight parameters are updated in each training iteration. Although the activation mask only selects a small portion of activated neurons, dropout layer is still needed, for the selected neurons with winner activations are always kept and updated, which makes over-fitting prone to happen. In fc layers, the remaining activated neurons are reduced to S winner from S total neurons as defined in Equation ( 1). The dropout layer connected after the activation mask is suggested to be modified with the setting:

Dropout rate = 0.5 S winner S total = 0.5

EQUATION

where 0.5 is the conventionally chosen dropout rate in the training process for original models, and the activation winner rate is introduced to regulate the dropout strength for balancing over-fitting and under-fitting. The dropout layers will be directly removed in the inference stage.

3.5. OPTIMIZER AND LEARNING RATE:

We find different optimizer requirements for weight pruning and activation pruning. In the weight pruning stage, it's recommended to adopt the same optimizer used for training the original model. The learning rate should be properly reduced to 0.1× ∼ 0.01× of the original learning rate. In the activation pruning stage, our experiments show that Adadelta (Zeiler, 2012) usually brings the best performance. Adadelta adapts the learning rate for each individual weight parameter. Smaller updates are performed on neurons associated with more frequently occurring activations, whereas larger updates will be applied for infrequent activated neurons. Hence, Adadelta is beneficial for sparse weight updates, which is exactly the common situation in our activation pruning. During finetuning, only a small portion of weight parameters are updated because of the combination of sparse patterns in weights and activations. The learning rate for Adadelta is also reduced 0.1× ∼ 0.01× compared to that used in training the original model.

4. EXPERIMENTS:

All of our models and evaluations are implemented in TensorFlow. IPnets are verified on various models ranging from simple multi-layer perceptron (MLP) to deep convolution neural networks (CNNs) on three datasets, MNIST, CIFAR-10 and ImageNet as in Table 1 . For AlexNet (Krizhevsky et al., 2012) and ResNet-32 (Zagoruyko & Komodakis, 2016) , we focus on conv layers because conv layers account for more than 90% computation cost in these two models.

The compression results of IPnets on activations, weights and MACs are summarized in Table 1 compared to the original dense models. IPnets achieve a 2.3× ∼ 5.8× activation compression rate and a 2.5× ∼ 10× weight compression rate. Benefiting from sparse weights and activations, IPnets only need 3.65% ∼ 28.9% of MACs required in dense models. The accuracy drop is kept less than 0.5%, and for some cases, e.g., MLP-3 and AlexNet in Table 1 , the IPnets achieve a better accuracy.

Table 1 shows that our method can learn both sparser activations and sparse weights and thus save computation. More importantly, in Figure 2 , we will show that our approach is superior to ap-proaches which explore intrinsic sparse ReLU activations and state-of-the-art weight pruning. The ReLU function brings intrinsic zero activations for MLP-3, ConvNet-5 and AlexNet in our experiments. However, the non-zero activation percentage increases in weight-pruned (WP) models as depicted in Figure 2 (a). The increment of non-zero activations undermines the effort from weight pruning. The activation pruning can remedy the activation sparsity loss and prune 7.7% -18.5% more activations even compared to the original dense models. 

4.1. MLP-3 ON MNIST:

The MLP-3 on MNIST has two hidden layers with 300 and 100 neurons respectively, and the model configuration details are summarized in Table 2 . The amount of MACs is calculated with batch size as 1, and the non-zero activation percentage at the output per layer is averaged from random 1000 samples from the training dataset. The following discussions on other models obey the same statistics setting. The model size of MLP-3 is firstly compressed 10× through weight pruning.

IP further reduces the total number of MACs to 3.65% by keeping only 17.1% activations. The accuracy of the priginal dense model is 98.41% on MNIST, and the aggressive reduction of MACs (27.4×) doesn't decrease the accuracy. For digit images in MNIST dataset have specific sparse features, the results on small-footprint MLP-3 are very promising. IP is further applied for a 5-layers CNN, ConvNet-5, on a more complicated dataset, CIFAR-10. With two conv layers and three fc layers, the original model has an 86% accuracy. As shown in Table 3 , the IPnet for ConvNet-5 only needs 27.7% of total MACs compared to the dense model through pruning 59.6% of weights and 56.4% of activations at the same time.

The accuracy only has a marginal 0.06% drop. The dominant computation cost is from conv layers accounting for more than 4/5 of total MACs for inference. Although fc layers can generally be pruned in larger strength than conv layers, the computation cost reduction of IPnet is dominated by the pruning results in conv layers. long tails towards both positive and negative directions. The activation distribution after IP are shown in Figure 3 (b). Activations near zero are pruned out, and the major contribution comes from removing small negative values. In addition, the kept activations are trained to be stronger with larger magnitude, which is consistent with the phenomenon that the non-zero activation percentage increases after weight pruning when using ReLU as illustrated in Figure 2 (a) . 

5. DISCUSSION:

The static activation pruning approach has been widely adopted in efficient DNN accelerator designs (Albericio et al., 2016; Reagen et al., 2016) . By selecting a proper static threshold θ in Equation (2), more activations can be pruned with little impact on model accuracy. For the activation pruning in IP, the threshold is dynamically set according to the winner rate and activation distribution layer-wise. The comparison between static and dynamic pruning is conducted on ResNet-32 for CIFAR-10 dataset. For the static pruning setup, the θ for leaky ReLU is assigned in the range of [0.07, 0.14], which brings different activation sparsity patterns. As the result of leaky ReLU with static threshold shown in Figure 4 , the accuracy starts to drop rapidly when non-zero activation percentage is less than 58.6% (θ = 0.08). Using dynamic threshold settings according to winner rates, a better accuracy can be obtained under the same activation sparsity constraint. Finetuning the model using dynamic activation masks will dramatically recover

:



Figures and Tables:

Figure 1: Working flow of integral pruning.

Figure 2: Comparison between WP models and IPnets.

Figure 3: Activation distribution of ResNet-32 (500 image samples).

Figure 4: Comparison to static activation pruning.

The largest gain from IP exits in ResNet-32 which uses leaky ReLU as activation function. Leaky ReLU generates dense activations in the original and WP models. The IPnet for ResNet-32 realizes a 61.4% activation reduction. At last, IPnets reduce 4.4% ∼ 22.7% more MACs compared to WP models as depicted in Figure2 (b), which means a 1.2× ∼ 2.7× improvement. More details on model configuration and analysis are discussed as follows.

Table content: <table><tr><td/><td/><td/><td colspan="3">Table 1: Summary of IPnets</td></tr><tr><td/><td>Network</td><td/><td colspan="2">MLP-3 ConvNet-5</td><td>AlexNet</td><td>ResNet-32</td></tr><tr><td/><td>Dataset</td><td/><td colspan="3">MNIST CIFAR-10 ImageNet</td><td>CIFAR-10</td></tr><tr><td/><td colspan="2">Orig Acti Function</td><td>ReLU</td><td>ReLU</td><td>ReLU Leaky ReLU</td></tr><tr><td/><td colspan="3">Accuracy Baseline 98.41%</td><td>86%</td><td>57.22%</td><td>95.01%</td></tr><tr><td/><td>Accuracy IP</td><td/><td>98.42%</td><td>85.94%</td><td>57.26%</td><td>94.58%</td></tr><tr><td/><td>Activation %</td><td/><td>17.1%</td><td>43.6%</td><td>44.2%</td><td>38.6%</td></tr><tr><td/><td>Weight %</td><td/><td>10%</td><td>40.4%</td><td>38.8%</td><td>32.4%</td></tr><tr><td/><td>MAC %</td><td/><td>3.65%</td><td>27.7%</td><td>28.9%</td><td>13.7%</td></tr><tr><td/><td>Dense</td><td>WP</td><td>IP</td><td/></tr><tr><td/><td>100%</td><td/><td/><td/></tr><tr><td>Activation %</td><td>40% 60% 80%</td><td/><td/><td/></tr><tr><td/><td>20%</td><td/><td/><td/></tr><tr><td/><td colspan="3">MLP-3 ConvNet-5 Alexnet ResNet-32</td><td/></tr></table>



Table content: <table><tr><td/><td/><td/><td colspan="2">: MLP-3 on MNIST</td><td/><td/></tr><tr><td>Layer</td><td>Shape</td><td colspan="5">Weight # MAC # Acti % Weight % MAC %</td></tr><tr><td>fc1</td><td>784×300</td><td>235.2K</td><td>235.2K</td><td>12%</td><td>10%</td><td>3.77%</td></tr><tr><td>fc2</td><td>300×100</td><td>30K</td><td>30K</td><td>24%</td><td>10%</td><td>2.62%</td></tr><tr><td>fc3</td><td>100×10</td><td>1K</td><td>1K</td><td>100%</td><td>20%</td><td>6.81%</td></tr><tr><td/><td>Total</td><td>266.2K</td><td colspan="2">266.2K 17.1%</td><td>10%</td><td>3.65%</td></tr></table>

We push IP onto AlexNet for ImageNet ILSVRC-2012 dataset which consists of about 1.2M training images and 50K validating images. The ALexNet comprises 5 conv layers and 3 fc layers and achieves 57.22% top-1 accuracy on the validation set. Similar to ConvNet-5, the computation bottleneck of AlexNet exits in conv layers by consuming more than 9/10 of total MACs. We focus on conv layers here. As shown in Table4, deeper layers have larger pruning strength on weights and activations because of the sparse high-level feature abstraction of input images. For example, the MACs of layer conv5 can be reduced 10×, while only a 1.2× reduction rate is realized in layer conv1. In total, the needed MACs are reduced 3.5× using IP with 38.8% weights and 44.2% activations. models are getting deeper with tens to hundreds of conv layers. We verify the IP method on ResNet-32 as shown in Table5. The ResNet-32 consists of 1 conv layer, 3 stacked residual units and 1 fc layer. Each residual unit contains 5 consecutive residual blocks. The filter numbers in residual units increase rapidly, and same for weight amount. An average pooling layer is connected before the last fc layer to reduce feature dimension. Compared to conv layers, the last fc layer can be neglected in terms of weight volume and computation cost. The original model has a 95.01% accuracy on CIFAR-10 dataset with 7.34G MACs per image. Weight and activation pruning strength is designed unit-wise to reduce the exploration space of hyperparameters, i.e., threshold settings. Notice that leaky ReLU is used as the activation function, thus zero activations are extremely hard to occur in the original and WP model. Only with IP, the activation percentage can be reduced down to 38.6%. As shown in Table5, the model size is compressed 3.1×, and the final gain is that 86.3% of MACs can be avoided while keeping the accuracy drop less than 0.5%.

Table content: <table><tr><td/><td/><td colspan="3">: ConvNet-5 on CIFAR-10</td><td/><td/></tr><tr><td>Layer</td><td>Shape</td><td colspan="5">Weight # MAC # Acti % Weight % MAC %</td></tr><tr><td>conv1</td><td>5×5, 64</td><td>4.8K</td><td colspan="2">0.69M 50.6%</td><td>70%</td><td>70%</td></tr><tr><td>conv2</td><td>5×5, 64</td><td>102.4K</td><td colspan="2">3.68M 17.3%</td><td>50%</td><td>25.3%</td></tr><tr><td>fc1</td><td>2304×384</td><td>884.7K</td><td>884.7K</td><td>9.9%</td><td>40%</td><td>6.92%</td></tr><tr><td>fc2</td><td>384×192</td><td>73.7K</td><td>73.7K</td><td>44.8%</td><td>30%</td><td>3%</td></tr><tr><td>fc3</td><td>192×10</td><td>1.92K</td><td>1.92K</td><td>100%</td><td>50%</td><td>22.4%</td></tr><tr><td/><td>Total</td><td>1.07M</td><td colspan="2">5.34M 43.6%</td><td>40.4%</td><td>27.7%</td></tr><tr><td colspan="2">4.3 ALEXNET ON IMAGENET</td><td/><td/><td/><td/><td/></tr><tr><td/><td/><td colspan="3">Table 4: AlexNet on ImageNet</td><td/><td/></tr><tr><td>Layer</td><td>Shape</td><td colspan="5">Weight # MAC # Acti % Weight % MAC %</td></tr><tr><td colspan="2">conv1 11×11, 96</td><td>34.85K</td><td colspan="2">112.2M 68.7%</td><td>85%</td><td>85%</td></tr><tr><td colspan="2">conv2 5×5, 256</td><td>307.2K</td><td colspan="2">240.8M 35.8%</td><td>40%</td><td>27.5%</td></tr><tr><td colspan="2">conv3 3×3, 384</td><td>884.7K</td><td>149.5M</td><td>25%</td><td>35%</td><td>12.6%</td></tr><tr><td colspan="2">conv4 3×3, 384</td><td>663.5K</td><td>112.1M</td><td>25%</td><td>40%</td><td>10%</td></tr><tr><td colspan="2">conv5 3×3, 256</td><td>442.4K</td><td>74.8M</td><td>27.7%</td><td>40%</td><td>10%</td></tr><tr><td/><td>Total</td><td>2.33M</td><td colspan="2">689.5M 44.2%</td><td>38.8%</td><td>28.9%</td></tr><tr><td colspan="2">4.4 GOING DEEPER</td><td/><td/><td/><td/><td/></tr></table>



Table content: <table><tr><td>Layer</td><td>Shape</td><td/><td colspan="5">Weight # MAC # Acti % Weight % MAC %</td></tr><tr><td>conv1</td><td>3×3, 16</td><td/><td>0.43K</td><td>0.44M</td><td>40%</td><td>40%</td><td>40%</td></tr><tr><td>unit2 {</td><td>3×3, 160 3×3, 160</td><td>} × 5</td><td>2.1M</td><td>2.15G</td><td>40%</td><td>40%</td><td>16%</td></tr><tr><td>unit3 {</td><td>3×3, 320 3×3, 320</td><td>} × 5</td><td>8.76M</td><td>2.6G</td><td>40%</td><td>40%</td><td>16%</td></tr><tr><td>unit4 {</td><td>3×3, 640 3×3, 640</td><td colspan="2">} × 5 35.02M</td><td>2.6G</td><td>30%</td><td>30%</td><td>9.5%</td></tr><tr><td/><td>Total</td><td/><td>45.87M</td><td>7.34G</td><td>38.6%</td><td>32.4%</td><td>13.7%</td></tr></table>

Peer review (iclr2019):
 This article presents a novel approach called Integral Pruning (IP) to reduce the computation cost of Deep neural networks (DNN) by integrating activation pruning along with weight pruning. The authors show that common techniques of exclusive weight pruning does compress the model size, but increases the number of non-zero activations after ReLU. This would counteract the advantage of DNN accelerator designs (Albericio et al., 2016; Reagen et al., 2016) that leverage activation sparsity to speed up the computations. IP starts with pruning the weights using an existing technique to mask out weights under a threshold and then fine-tune the network in an iterative fashion to maintain the accuracy. After weight pruning, IP further masks out the activations with smaller magnitude to reduce the computation cost. Unlike weight pruning techniques that use static masks, the authors propose to use dynamic activation masks for activation sparsity in order to account for various patterns that are being activated in DNN for different input samples. In order to do this, the *winner rate* measure for every layer (or for a group of layers in deep networks like ResNet32) is defined, to dynamically set the threshold for the generation of activation masks which eventually controls the amount of non-zero activation entries. The article empirically analyzes the sensitivity of activation pruning on validation data by setting different winner rates at every layer in DNN and decides upon a set of winner rates accordingly followed by an iteration of fine-tuning the network to maintain its performance. The authors show that their technique produced lower number of non-zero activations in comparison with the intrinsic sparse ReLU activations and weight pruning techniques. The topic of reducing network complexity for embedded implementations of DNNs is highly relevant, in particular for the ICLR community. The IP technique yields significantly reduced number of multiply-accumulate operations (MACs) across different models like MLP-3, ConvNet-5, ResNet32 and AlexNet and on different datasets like MNIST, CIFAR10 and ImageNet. They also depicted that pruning the activations with dynamic activation masks followed by fine-tuning the network yields more sparse activations and negligible loss in accuracy when compared against using static activation masks. Strengths of the paper: - The motivation to extend compression beyond the weights to activations in order to support the DNN accelerator designs and the technical details are clearly explained. - The proposed technique indeed produces sparser activations than intrinsic ReLu sparse activations and can also applied to any network regardless of the choice of activation function. - The proposed technique is evaluated across different network architectures and datasets. - The advantage of adapting dynamic activation masks over static ones is clearly demonstrated. Weaknesses of the paper: - The originality of the approach is limited because it is a relatively straightforward combination of existing techniques for weight and activation pruning. - The "winner rate" measure is defined for every layer and should be explored over different values in order to find the equilibrium to reduce the number of non-zero activations and maintain the accuracy. This search of winner rates will become inefficient as the depth of the network increases. However, the authors used a single winner rate for a group of layers in case of ResNet-32 to reduce the exploration of search space but this choice might lead to suboptimal results. - The authors compare the resultant number of MAC operations against numbers from the weight pruning technique. However, there also exist different works on group pruning techniques like Liu et al. (2017), Huang & Wang (2017), Ye et al. (2018) to prune entire channels / feature maps and thus yield more compact networks. Since these approaches prune the channels, they show a direct impact on the computation complexity and greatly reduce the computation time. A proper and fair comparison would be to compare the numbers of IP against such group pruning techniques. This comparison is highly important to highlight the significance of the approach on speeding up the DNNs and it is missing from the paper. - At several locations in Section 4, e.g. Sec. 4.1, 4.3, and 4.4. there is no precise statement about the incurred accuracy loss (or no statement at all). The connection to Figures 4 and 5 is not immediately clear and should be made explicit. References: - Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. - Jianbo Ye, Xin Lu, Zhe Lin, and James Z Wang. Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers - Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. Overall Evaluation: The authors integrate activation pruning along with the weight pruning and show that the number of MAC operations are greatly reduced by their technique when compared to the numbers of weight pruning alone. However, I am not convinced regarding the reported number of MAC operations since the number of MAC operations of sparse weight matrices and activations would remain the same as the original models unless some of the filters/activation maps are pruned from the network. On the other hand, comparisons against group pruning techniques are highly necessary to evaluate the potential impact of the approach on speeding up of DNNs. My preliminary rating is a weak reject but I am open to revise my rating based on the authors response to the above stated major weaknesses. Minor comments: - Caption of Fig. 4 should mention the task on which the results were obtained. - There are occasional grammar errors and typos that should be corrected.