DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures
Authors: Huanrui Yang
Wei Wen
Hai Li
Venue:  (2020)
Keywords: Deep neural network
Sparsity inducing regularizer
Model compression

Abstract: In seeking for sparse and efficient neural network models, many previous works investigated on enforcing L1 or L0 regularizers to encourage weight sparsity during training. The L0 regularizer measures the parameter sparsity directly and is invariant to the scaling of parameter values. But it cannot provide useful gradients and therefore requires complex optimization techniques. The L1 regularizer is almost everywhere differentiable and can be easily optimized with gradient descent. Yet it is not scale-invariant and causes the same shrinking rate to all parameters, which is inefficient in increasing sparsity. Inspired by the Hoyer measure (the ratio between L1 and L2 norms) used in traditional compressed sensing problems, we present DeepHoyer, a set of sparsity-inducing regularizers that are both differentiable almost everywhere and scale-invariant. Our experiments show that enforcing DeepHoyer regularizers can produce even sparser neural network models than previous works, under the same accuracy level. We also show that DeepHoyer can be applied to both element-wise and structural pruning.

Published as a conference paper at ICLR 2020
DEEPHOYER: LEARNING SPARSER NEURAL NETWORK WITH DIFFERENTIABLE SCALE-INVARIANT
SPARSITY MEASURES
Huanrui Yang, Wei Wen, Hai Li
Department of Electrical and Computer Engineering, Duke University, Durham, NC 27708 yauanrui.yang, wei.wen, hai.li @duke.edu
ABSTRACT
In seeking for sparse and efficient neural network models, many previous works investigated enforcing ε or to regularizers encourage weight sparsity during training. The lg regularizer measures the parameter sparsity directly and is invariant the scaling of parameter values. But it cannot provide useful gradients and therefore requires complex optimization techniques. The & regularizer is almost everywhere differentiable and can be easily optimized with gradient descent. Yet it is not scale-invariant and causes the same shrinking rate to all parameters, which is inefficient in increasing sparsity. Inspired by the Hoyer measure (the ratio between L and & norms) used in traditional compressed sensing problems, we present DeepHoyer, a set of -inuuiig regularizers that are both differentiable almost everywhere and scale-invarian
Our experiments show that enforcing DeepHoyer regularizers can produce even sparser neural network models than previous works, under the same accuracy level. We also show that DeepHoyer can be applied to both element- wise and structural pruning. The codes are available at https: /tt:://gitttu com wtt.oo///aagchheee http///aaaahp/////yyeeeee
INTRODUCTION
The use of deep neural network (DNN) models has been expanded from handwritten digit recogni- tion (LeCun et al. 1998) to real-world applications, such as large-scale image classification (Simonyan
& Zisserman 2014). self driving (Makantasis et al. 2015) and complex control problems (Mnih
et al. 2013). However, modern DNN model like AlexNet (Krizhevsky et al.) (012) or ResNet (He
al. 2016) often introduces a large number of parameters and computation load, which makes the deployment and real-time processing embedded and edge devices extremely difficult (Han
ctt al. 2015bJa; Wen et al., 2016). Thus, model compression techniques, especially pruning methods that increase the sparsity of weight matrices, have been extensively studied to reduce the memory
consumption and computation cost of DNNs (Han et al. 2015b a Wen ct al. 2016 Guo et al. 2016
Louizos et al. 2017b Luo al. 2017 Zhang al. 2018: Liu al. 2015).
Most of the previous works utilize some form of pparsity-induciie regularizer in searching for sparse neural networks. The l regularizer, originally proposed by Tibshirani (1996), can be easily optimized through gradient descent for its convex and almost everywhere differentiable property. Therefore it is widely used in DNN pruning: Liu al. (2055) directly apply E regularization all the weights of a DNN to achieve element-wise sparsity; Wen et al.] (2016) 2017) present structural sparsity via group lasso, which applies an & regularization over the Eg norms of different groups of parameters. However, it has been noted that the value of the E regularizer is proportional to the scaling of parameters (i.e. |||]|| |a - [[]]), so "scales down" all the elements the weight matrices with the same speed. This is not efficient in finding sparsity and may sacrifice the flexibility of the trained model. On the other hand, the & regularizer directly reflects the real sparsity of weights and is scale invariant (i.e. ||a || Ya 7 0), yet the En norm cannot provide useful gradients. Han et al. (2015b) enforce an element-wise fy constraint by iterative pruning a fixed percentage of smallest weight elements, which is heuristic method and therefore can hardly achieve optimal compression rate. Some recent works mitigate the lack of gradient information by integrating to regularization with stochastic approximation (Louizos et al. 2017b) more complex optimization methods (e.g.

---

Published as a conference paper at ICLR 2020
ADMM) Zhang al. 2018). These additional measures brought overheads to the optimization process, making the use of these methods on larger networks difficult. To achieve even sparser neural networks. argue to move beyond En and & regularizers and seek for a aparsity-inducinng regularizer that is both almost everywhere differentiable (like (1) and scale-invariant (like Eo)-
Beyond the & regularizer, plenty of non-convex sparsity measurements have been used in the field of feature selection and compressed sensing (Hurley & Rickard 2209;; Wen et al.) 2018 Some popular regularizers like SCAD (Fan & Li 2001), MDP (Zhang et al. 2010) and Trimmed & (Yun) et al. 2019) use a piece-wise formulation to mitigate the proportional scaling problem of é. The piece-wiss wise formulation protects larger elements by having zero penalty to elements greater than a predefined threshold. However, it is extremely costly to manually seek for the optimal trimming threshold, so is hard to obtain optimal result in DNN pruning by using these regularizers. The transformed l regularizer formulated as >>=1
(a+1)]wal
manages to smoothly interpolate between e+ [ey
l and to by tuning the hyperparameter a (Ma et al. 2019). However, such an approximation is close to la only when approaches infinity, so the practical formulation of the transformed & (i.e. a = 1) is still not scale-invariant.
Particularly, we are interested in the Hoyer regularizer (Hoyer) 2004), which estimates the sparsity of a vector with the ratio between its & and l2 norms. Comparing to other sparsity-inducing regularizers, Hoyer regularizer achieves superior performance in the fields of non-negative matrix factorization (Hoyer, 2004), sparse reconstruction (Esser et al., 2013) Tran et al., 2018) and blend deconvolution (Krishnan et al. 2011 Repetti et al. 2015). We note that Hoyer regularizer is both almost everywhere differentiable and scale invariant, satisfying the desired property of a sparsity- inducing regularizer. We therefore propose DeepHoyer, which is the first Hoyer-inspired regularizers for DNN sparsification. Specifically, the contributions of this work include:
Hoyer-Square (HS) regularizer for element-wic wise sparsity: We enhance the original Hoyer regularizer the HS regularizer and achieve element-wise sparsity by applying it in the training of DNNs. The HS regularizer is both almost everywhere differentiable and scale invariant. It has the same range and minima structure as the Eg norm. Thus, the HS regularizer presents the ability of turning small weights to zero while protecting and maintaining those weights that are larger than an induced, gradually adaptive threshold:
Group-Hi HS regularizer for structural sparsity, which is extended from the HS regularizer;
Generating sparser DNN models: Our experiments show that the proposed regularizers beat state-of-the-arts in both element-wise and structural weight pruning of modern DNNs.
2 RELATED WORK ON DNN PRUNING
It is well known that high redundancy pervasively exists in DNNs. Consequently, pruning methods have been extensively investigated to identify and remove unimportant weights. Some heuristic pruning methods (Han et al.) 2015b Guo al., 2016) simply remove weights in small values to generate sparse models. These methods usually require long training time without ensuring the optimality, due the lack of theoretical understanding and well-forrullated optimization /Zhng et al. 2018). Some works formulate the problem as a sparsity-inducing optimization prohlem, such as é regularization (Liu al. 2015 Park al. 2016) that can be optimized using standard gradient- based algorithms, or to regulan legularization (Louizos et al., 2017b; Zhang et al.] 2018) which requires stochastic approximation
special optimization techniques. We propose DeepHoyer regularizers in this work, which belong to the line of sparsity- -rnduiiy-nnducing optimization research. More specific, the proposed Hoyer-Squaa regularizer for element-wiss wise pruning is icale-invariant and can serve as an differentiable approximation to the lo norm. Furthermore, it can be optimized by wradient-based optimization methods in the same way as the l regularization. With these properties, the Hoyer- Square regularizer achieves further 38% and 63% sparsity improvement LeNet-300-11 100 model and LeNet-5 model respectively comparing to previous state-of-th tfth---rsss- and achieves the highest sparsity on AlexNet without accuracy loss.
Structurally sparse DNNs attempt to create regular sparse patterns that are friendly for hardware execution. To achieve the goal, Li et al. (2016) propose to remove filters with small norms; Wen jet al. (2016) apply group Lasso regularization based methods to remove various structures (e.g.. filters, channels, layers) in DNNs and the similar approaches used to remove neurons (Alvarez &
2

---

Published as a conference paper at ICLR 2020
1 [
47 [141
0.5
0.5
0.5
0.5
455
Figure I: Comparing the E and the Hoyer regularizer of 2-D vector. Their contours are shown in the left 2 subplots (darker color corresponds to lower value). The right 2 subplots compare their negative gradients.
Salzmann 2016); Liu et al. (2017) and Gordon et al. (2018) (MorphNet) enforce sparsity-inducing regularization on the scaling parameters within Batch Normalization layers to remove the corre- sponding channels in DNNs; ThiNet (Luo et al. (017) removes unimportant filters by minimizing the reconstruction error of feature maps; and He et al. (2017) incorporate both Lasso regression and reconstruction error into the optimization problem. Bayesian optimization methods have also been applied for neuron pruning (Louizos et al. 2017a| Neklyudov et al.) (017). yet these methods are not applicable in large-scale problems like ImageNet. We further advance the DeepHoyer to learn structured sparsity (such as reducing filters and channels) with the newly proposed "Group-HS" regularization. The Group-HS regularizer further improves the computation reduction of the LeNet-5 model by 8.8% from the & based method (Wen et al., (016), and by 110.6% from the Eo based method (Louizos et al., 2017b). Experiments on ResNet models reveal that the accuracy-speedug tradeoff' induced by Group-HS constantly stays above the Pareto frontier of previous methods. More detailed results can found in Section
3 MEASURING SPARSITY WITH THE HOYER MEASURE
Sparsity measures provide tractable sparsity constraints for enforcement during problem solving and therefore have been extensively studied in the compressed sensing society. early non-negan -egatvve matrix factorization (NMF) research, a consensus was that sparsity measure should map a n- dimensional vector to a real number 5 E [0, 1]. such that the possible sparsest vectors with only one nonzero element has S = 1, and vector with all equal elements has S 0 (Hoyer 2004).
Unders the assumption, the Hoyer measure was proposed as follows
S(X) = -ŭ- (E,t,l//VE
(1)
Vπ- 1-
It can be seen that
Edeed Vn, EEER".
1 1
(2)
VI, x
Thus, the normalization in Equation 1 fits the measure S(X) into the [0, 1] interval. According to the survey by Hurley & Rickard (2009). among the six desired heuristic criteria of sparsity measures, the Hoyer measure satisfies five, more than all other commonly applied sparsity measures. Given its success as sparsity measure in NMF, the Hoyer measure has been applied as a -parsity-inducing regularizer in optimization problems such blind deconvolution (Repetti al. (0155) and image deblurring (Krishnan et al. 2011. Without the range constraint, the Hoyer regularizer in these works directly, as the ratio of the and lg norms.
adopts the form R(X)
2, 4
Figure I compares the Hoyer regularizer and the L regularizer. Unlike the the C, norm with a single minimum at the origin, the Hoyer regularizer has minima along axes, the structure of which is very
similar to the t, norm's. Moreover, the Hoyer regularizer is scale-invariant, i.e. R(aX)= = R(X). because both the & norm and the Eg norm proportional to the scale of X. The gradients of the Hoyer regularizer are purely radial, leading to "rotations" " towards the nearest axis. These features make the Hoyer regularizer outperform the l regularizer various tasks (Esser al. 2013
Tran
3

---

Published as a conference paper at ICLR 2020
al.
2018 Krishnan et al. 2011 Repetti et al. 2015). The theoretical analysis by Yin et al. (2014)
also proves that the Hoyer regulanzer has a better guarantee than the & norm on recovering sparse solutions from coherent and redundant representations.
4 MODEL COMPRESSION WITH DEEPHOYER REGULARIZERS
Inspired by the Hoyer regularizer, we propose two types of DeepHoyer regularizers: the Hoyer-Square regularizer (HS) for element-wise pruning and the Group-HS regularizer for structural pruning.
4.1 HOYER-SQUARE REGULARIZER FOR ELEMENT- WISE PRUNING
Since the process of the element-wise pruning is equivalent to regularizing each layer's weight with the Ea norm, is intuitive to configure the sparsity-inducing regularizer to have similar behavior as the to norm. As shown in Inequality (2), the value of the original Hoyer regularizer of a N- dimensional nonzero vector lies between I and vN, while its lo norm is within the range of [1, N). Thus we propose to apply the square of Hoyer regularizer, namely Hoyer-Square (HS), to the weights W of a layer, like
Hs(W) [ w Σ, mu?
(3)
The proposed HS regularizer behaves differentiable approximation to the lu norm. First, both regularizers now have the same range of [1, NJ. Second, Hs is scale invariant as Hs(aW)= = (W) holds for Ya * O, as the lo norm. Moreover, the squaring operator monotonously increases in the range of [1, VN]. the Hoyer-Square regularizer's minima remain along the regularizer's s do (see Figure []. In other words, they have similar minima structure as the lo norm. At last, the Hoyer-Square regularizer is also almost everywhere differentiable and Equation formulates the gradient of Hs w.r.t. an element wi in the weight matrix W:
the Hoyer
E.wuyl
Ow, Hs( 2sign(w: w1))
|w5]
Σ /w:l).
(4)
(E.
Very importantly, this formulation induces a trimming effect: when Hs(W) is being minimized
through gradient descent, wy moves towards if |wyl<< [.,T otherwise moves away from 0. In other words, unlike the l regularizer which tends to shrink all elements, our Hoyer-Square regularizer will turn weights in small value to zero meanwhile protecting large weights. Traditional trimmed regularizers (Fan & Li 2001 Zhang et al. [210 Yun et al. (2019) usually define trimming threshold as fixed value or percentage. Instead, the HS regularizer can gradually extend the scope pruning as more weights coming close to zero. This behavior can be observed in the gradient descent path
shown in Figure 2
4.2 GROUP-HS REGULARIZER FOR STRUCTURAL PRUNING
Beyond element-wise pruning, structural pruning is often more preferred because it can construct the sparsity in a structured way and therefore achieve higher computation speed-up on general computation platforms (Wen et al., 2016). The structural pruming is previously empowered by the group lasso (Yuan & Lin 2006) Wen et al. 2016), which the sum (i.e. l norm) of the & norms of all the groups within weight matrix like
(W)W) Σ [ ||,
where |||== Vz, /i represents the norm, (() is group of elements in the weight matrix W which consists of G such groups.
Following the same approach in Section 4.1 use the Hoyer-Square regularizer to replace the f
regularizer in the group lasso formulation and define the Croup-HS (G,r) regularizer in Equation 66:
(2 1 1|2) (_
Gn(W)=
(6)
1=1 (w()) 113
||
4

---

Published as a conference paper at ICLR 2020
8
100
30
50
70
100
Steps
Figure Minimization path of Hoyer-Square regularizer during gradient descent, with W 20 initialized as i.i.d (0,, 1). The figure shows the path of each element w; during the minimization, with the black dash line showing the induced trimming threshold.
Note that the second equality holds when and only when the groups cover all the elements of W without overlapping with each other. Our experiments this paper satisfy this requirement. However, the Group- HS regularizer can always be used in the form of the first equality when overlapping exists across groups. The gradient and the descent path of the Group-HS regularizer are very similar to those of the Hoyer-Square regularizer, and therefore we omit the detailed discussion here. The derivation the Group-HS regularizer's gradient shall be found in Appendix A
4.3 APPLY DEEPHOYER REGULARIZERS IN DNN TRAINING
The deployment of the DeepHoyer regularizers in DNN training follows the common layer-based regularization approach (Wen et al. 2016 Liu et al, 2015). For element-wise pruning, we apply
the Hoyer-Square regularizer to layer weight matrix W for all L layers, and directly minimize alongside DNN's original training objective L(W(1=L)) The & regularizer can also be added to the objective if needed. Equation (7) presents the training objective with Hs defined in Equation (3) Here, a and 3 are pre-selected weight decay parameters for the regularizers.
min ((W(1:)) W(LL)
Σ (aHs(W(") + (l)w()))2)
For structural pruning, we mainly focus on pruning the columns and rows of fully connected layers and the filters and channels of convolutional layers. More specific, we group layer in filter-wise and channel-we -wise fashion as proposed by Wen et al. (2016) and then apply the Group-HS regularizer to layer. The resulted optimization objective formulated Equation (8)
1 + ag (2C B (2..)2)1)) + (l(W(()))) (8)
min ((W(LA)) W(0))
Σ (On
|
||
Here Nj the number of filters and C the number channels the 7th layer if a convolutional layer. If the 7th layer is fully connected, then N; and Ct is the number of rows and columns respectively. on a. and are pre-selected weight decay parameters for the regularizers.
The recent advance in stochastic gradient descent (SGD) method provides satisfying results under large-scale non-convex settings (Sutskever er al. 2013 Kingma & Bal 2014), including DNNs with non-convex objectives (Auer et 1996). So we directly optimize the DeepHoyer regularizers with the same SGD optimizer used for the original DNN training objective, despite their nonconvex formulations. Our experiments show that the tiny-bit nonconvexity induced by DeepHoyer does not affect the performance of DNNs.
The pruning conducted by following the common stage operations: (1) train the DNN with the DeepHoyer regularizer, (2) prune all the weight elements smaller than a predefined small threshold, and (3) finetune the model by fixing all the zero elements and removing the DeepHoyer regularizer.
5

---

Published as a conference paper at ICLR 2020
Table I: Element-wiss pruning results on LeNet-300-100 model accuracy 98.4%
Nonzero wights left after pruning
Method
Total
FC1
FC2
FC3
Orig
266.2k
235.2k
30k
(Han et al. 2015b)
21.8k (8%)
18.8k (8%)
2.7k (9%)
260 (26%)
Zhang et al. 2018
11.6k (4.37%)
9.4k (4%)
2.1k (7%)
120 (12%)
Lee ct al. 2019)
13.3k (5.0%)
Not reported in (Lee et al.
2019)
Ma et al. 2019
6.4k (2.40%)
5.0k (2.11%)
1.2k (4.09%)
209 (20.90%
Hoyer Hoyer-Square
6.0k (2.27%)
5.3k (2.25%)
672 (2.24%)
82 (8.20 %)
4.6k (1.74%)
3.7k (1.57% %)
768 (2.56%)
159 (15.90 (0.
Table 2: Element-wise pruning results on LeNet-5 model @ accuracy 99.2%
Nonzero wights left after pruning
Method
Total
CONVI
CONV2
FCI
FC2
Orig
430.5k
500
25k
400k
5k
HHan et al.
2015b
36k (8%)
330 (66%)
3k (12%)
32k (8%)
950 (19%)
Zhang et al.
20188
6.1k (1.4%)
100 (20%)
Not reported in 2k (8%)
3.6k (0.9%)
350 (7%)
Lee ct al.
2019)
8.6k (2.0%)
(Lee et al. 20199
Ma et al.
2019)
5.4k (1.3%)
100 (20%)
690 (2.8%)
4.4k (1.1 %))
203 (4.1%)
Hoyer
4.0k (0.9%)
53 (10.6%) %
613 )
3.2k (0.8%)
136 (2.7%)
Hoyer-Square
3.5k (0.8%)
67 (13.4%)
848 (3.4%)
2.4k (0.6%)
234 (4.7%)
5 EXPERIMENT RESULT
The proposed DeepHoyer regularizers are first tested on the MNIST benchmark using the LeNet-
300-100 fully connected model and the LeNet-5 CNN model (LeCun al.) 1998). We also conduct
tests on the CIFAR-10 dataset (Krizhevsky & Hinton 2009) with ResNet models (He et al., 2016) in various depths, and on ImageNet ILSVRC-2012 benchmark (Russakovsky al.] 2015)) with the AlexNet model (Krizhevsky et al. 2012) and the ResNet-50 model (He et al. 2016). All the models are implemented and trained in the PyTorch deep learning framework (Paszke al. 2017 where we match the model structure and the benchmark performance with those of previous works for the fairness of comparison. The experiment results presented the rest of this section show that proposed DeepHoyer regularizers consistently outperform previous works in both element- and structural pruning. Detailed information on the experiment setups and the parameter choices of our reported results can be found in Appendix B
5.1
ELEMENT-WISE PRUNING
Table and Table 2 summarize the performance of the proposed Hoyer-square regularizer on the MNIST benchmark, with comparisons against state of the art (SOTA) element-wise pruning methods. Without losing the testing accuracy, training with the Hoyer-Square regularizer reduces the number of nonzero weights by 54.5x on the LeNet-300-100 model and by 122x on the LeNet-5 model. Among all the methods, ours achieves the highest sparsity: it is 38% improvement on the LeNet-300- model and a 63% improvement on the LeNet-5 model comparing to the best available methods. Additional results in Appendix C.1 further illustrates the effect of the Hoyer-Square regularizer each layer's weight distribution during the training process.
The element-wise pruning performance the AlexNet model testing on the ImageNet benchmark is presented in Table 3] Without losing the testing accuracy, the Hoyer-Square regularizer improves
We implement the transformed & regularizer in Ma et al. 2019 ourselves because the experiments in the original paper are under different settings. Implementation details be found in Appendix B

---

Published as a conference paper at ICLR 2020
Table Element-wise pruning results on AlexNet model.
Method
Top-5 error increase
#Parameters
Percentage left
Orig
+0.0%
60.9M
100%
Han ct al.
2015b
-0.1%
6.7M
11.0%
Guo ct
2016
+0.2%
3.45M
5.67%
Dai et al.
2017
-0.1%
3.1M
6.40%
Ma et al.
2019
+0.0%
3.05M
5.01%
Zhang al. 2018
+0.0%
2.9M
4.76%
Hoyer Hoyer-Square
+0.0%
3,62M
5.94%
+0.0%
2.85M
4.69%
Table 4: Structural pruning results on LeNet-300-100 model
Method
Accuracy
#FLOPs
Pruned structure
Orig
98.4%
266.2k
784-300-100
Sparse VD Molchanov et al.
2017
98.2%
67 .3.3k (25, (.5.28%)
512-114-72
BC-GNJ Louizos et al. 20
98.2%
28 6k (10. (10.66%)
278-98-13
BC-GHS (Luuizos et al.
2017a
98.2%
28.1k (10.55%)
311-86-14
lon.
(Louizos et al. 20176)
98.2%
26.6k (10.01%)
266-88-33
Bayes 1errm Yun al.
2019
98.3%
20.5k (7.70%)
245-75-25
Group-HS
98.2%
16.5k (6.19%)
353-45-11
the compression rate by 21.3x. This result is the highest among all methods, even better than the ADMM method (Zhang al. 2018) which requires two additional Lagrange multipliers and involves the optimization of two objectives. Considering that the optimization of the Hoyer-Square regularizer can be directly realized Hoyer-Square regularizer can achieve sparse DNN model with much lower cost. A more detailed layer-by-layer sparsity comparison the compressed model can be found in Appendix C.2
conclude that the single objective without additional variables.
We perform the ablation study for performance comparison between the Hoyer-Square regularizer and the original Hoyer regularizer. The results in Tables 1 a all show that the Hoyer-Square regularizer always achieves a higher compression rate than the original Hoyer regularizer. The layer-wise compression results show that the Hoyer-Square regularizer emphasizes more on the layers with more parameters (i.e. FCI for the MNIST models). This corresponds the fact that the value of the Hoyer-Square regularizer is proportional the number of non-zero elements in the weight. These observations validate our choice to the Hoyer- -quere regularizer for DNN compression.
5.2 STRUCTURAL PRUNING
This section reports the effectiveness of the Group-HS regularizer in structural pruning tasks. Here we mainly focus on the number of remaining neurons (output channels for convolution layers and rows for fully connected layers) after removing the all-zero channels rows in the weight matrices. The comparison is then made based on the required float-point operations (FLOPs) to inference with the remaining neurons, which indeed represents the potential inference speed of the pruned model. As shown in Table 4 training with the Group-HS regularizer can reduce the number of FLOPs by 16.2x for the LeNet-300 300- model with a slight accuracy drop. This is the highest speedup among all existing methods achieving the same testing accuracy. Table 5 shows that the Group-HS regularizer can reduce the number FLOPs of the LeNet-5 model by 12.4x, which outperforms most of the existing work-an 8.8% increase from the E based method (Wen et al. (016)) and a 110.6% increase from the Ey based method (Louizos et al., 2017b). Only the Bayesian compression (BC) method with the croup-horsesiie the LeNet-5 model. However, the complexity of high dimensional Bayesian inference limits BC's capability. is difficult to apply BC ImageNet-level problems and large DNN models like ResNet.
prior (BC-GHS) (Louizos al., (017a) achieves slightly higher speedup
7

---

Published as a conference paper at ICLR 2020
Table 5: Structural pruning result on LeNet-5 model
Method
Accuracy
#FLOPs
Pruned structure
Orig
99.2%
2293k
20-50-800-500
Sparse VD
Molchanov et al.
2017
99.0%
660.2k (28.77 79%)
14-19-242-133
GL
Wen et al. 2016)
99.0%
201.8k (8.80%
3-12-192-500
SBP (Neklyudov et al.
2017)
99.1%
212.8k (9.28%)
3-18-284-283
BC-GNI (Louzzos et
2017a
99.0%
282.9k (12.34%)
8-13-88-13
BC-GHS (Louizos el al.
2017a)
99.0%
153.4k (6.69 )
5-10-76-16
Lone
(Louizos et al. 20T7b)
99.0%
390.7k (17.04%)
9-18-26-25
Bayes 11rem
Yun et al.
2019)
99.0%
334.0k (14.57%)
8-17-53-19
Group-HS
99.0%
169.9k (7.41 %)
5-12-139-13
ResNet-50 on ImageNet
ResNet-56 on CIFAR-10
ResNet-110 on CIFAR-10
(x) Ceccccpoe
Previous works
#F FLOPPS Cepaadi XX
Previous works
#FLOPs reduction
45-
DeepHoyer
DeepHoyer
Previous works
25
Deepkoyer
opps
#
12
93
0.5
0.6
0.8
0.6
-22
02
34
00
Top-5 accuracy (%)
Top-1 accuracy gain (%)
Top-1 accuracy gain (%)
Figure 3: Comparisons of accuracy tradeoff on ImageNet and CIFAR-10, black dash lines mark the Pareto frontiers. The exact data for the points are listed Appendix
In contrast, the effectiveness of the Group-Ht HS regularizer can be easily extended to deeper models and larger datasets, which is demonstrated by our experiments. We apply the Group-HS regularizer to ResNet models (He et al. 2016) on the CIFAR-10 and the ImageNet datasets. Pruning ResNet has long been considered difficult due to the compact structure of the ResNet model. Since previous works usually report the compression rate at different accuracy, we use the "accuracy##FLOOP to represent the tradeoff. The tradeoff between the accuracy and the FLOPs are explored in this work by changing the strength of the Group-HS regularizer used in training. Figure 3 shows the performance of DeepHoyer constantly stays above the Pareto frontier of previous methods.
plot
6 CONCLUSIONS
In this work, we propose DeepHoyer, a set of sparsity-inducing regularizers that are both scale- invariant and almost everywhere differentiable. We show that the proposed regularizers have similar range and minima structure as the t, norm, so it can effectively measure and regularize the sparsity of the weight matrices of DNN models. Meanwhile, the differentiable property enables the proposed regularizers to be simply optimized with standard gradient-based methods, in the same way as the é regularizer is. In the element-wise pruning experiment, the proposed Hoyer-Square regularizer achieves a 38% sparsity increase on the LeNer-300-11 LeNet-5 model without accuracy loss comparing to the state-of-the-ae rate is achieved on AlexNet, which also surpass all previous methods. In the structural pruning experiment, the proposed Group-HS regularizer further reduces the computation load by 24.4% from the state-of-the-art on LeNet-300- 100 model. It also achieves 8.8% increase from the & based method and 10.6% increase from the &o based method of the computation reduction rate on the LeNet-5 model. For 10 and ImageNet dataset, the accuracy-FLOP. tradeoff achieved by training ResNet models with various strengths of the Group-HS regularizer constantly stays above the Pareto frontier of previous methods. These results prove that the DeepHoyer regularizers are effective in achieving both element-wise and structural sparsity in deep neural networks, and can produce sparser DNN models than previous works.
model and a 63% sparsity increase on the A 21.3x model compression
8

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
The authors would like thank Feng Yan for his help on computation resources throughout this project. Our work was supported in part by NSF SPX-1725456 and NSF CNS-1822085.
REFERENCES
Jose M Alvarez and Mathieu Salzmann. Learning the number of neurons in deep networks. In Advances in Neural Information Processing Systems, pp. 2270-2278, 2016.
Peter Auer, Mark Herbster, and Manfred K Warmuth. Exponentially many local minima for single neurons. In Advances in neural information processing systems, pp. 316-322, 1996.
Xiaoliang Dai, Hongxu Yin, and Niraj K. Jha. Nest: A neural network synthesis tool based on now-annd-rune paradigm. CoRR, abs/1711 1.00077, 2017. httphhttp:/aaxxv.orrr abs/
1711. 02017]
Xiaohan Ding, Guiguang Ding, Yuchen Guo, and Jungong Han. Centripetal sgd for pruning very deep convolutional networks with complicated structure. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 199334995,, 2019.
Ernie Esser, Yifei Lou, and Jack Xin. A method for finding structured sparse solutions to nonnegative least squares problems with applications. SIAM Journal on Imaging Sciences, 6(4):2010-200 2013.
Jianqing Fan and Runze Li. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American statistical Association, 96(456): 1348-1360, 2001.
Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang, and Edward Choi Morphnet: Fast & simple essource-constraincd structure learning deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1586-1599 2018.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dons. In Advances In Neural Information Processing Systems, pp. 1379-1387, 2016.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.001 201 Sa.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv: 06866, 2018a.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397, 2017.
Yihui He, Lin, Zhijian Liu, Hanrui Wang. Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings of the European Conference Computer Vision (ECCV), pp. 784-800, 2018b.
Patrik O Hoyer. Non-negative matrix factorization with sparseness constraints. Journal of machine learning research, 1457-1464, 2004.
Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks, In Proceedings of the European Conference on Computer Vision (ECCV), pp. 304-320, 2018.
Niall Hurley and Scott Rickard. Comparing measures of sparsity. IEEE Transactions on Information Theory, 55(10):4723 4741, 2009.

---

Published as a conference paper at ICLR 2020
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Dilip Krishnan, Terence Tay, and Rob Fergus. Blind deconvolution using a normalized sparsity measure. CVPR 20/1, pp. 233-240. IEEE, 2011.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):227--2322, 1998.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: SINGLE-SHOT NETWORK
PRUNING BASED ON CONNECTION SENSITIVITY. In International Conference on Learning
Representations, 2019, URL htthtts:: /tttps::/opennevieew http:/iewwneeeeeeeee (otoorummmd-BB12
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 160.v:168..08710, 2016.
Tuanhui Li, Baoyuan Wu, Yujiu Yang, Yanbo Fan, Yong Zhang, and Wei Liu. Compressing convolutional neural networks via factorized convolutional filters. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3977-3986, 2019.
Shaohui Lin, Rongrong Ji, Yuchao Li, Yongjian Wu, Feiyue Huang, and Baochang Zhang. Accel- erating convolutional networks via global & dynamic filter pruning. In IJCAI, pp. 2425-2432, 2018.
Shaohui Lin, Rongrong Ji, Yuchao Li, Cheng Deng, and Xuelong Li. Toward compact convnets via structure-sparsity regularized filter pruning. IEEE transactions on neural networks and learning systems, 2019.
Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convolu- tional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 806-81 814, 2015.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2736-2744, 2017.
Christos Louizos, Karen Ullrich, and Max Welling, Bayesian compression for deep learning. In Advances in Neural Information Processing Systems, pp. 3288-3298, 2017a.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_0 regularization. arXiv preprint arXiv: 1712.0132, 2017b.
Jian-Hao Luo and Jianxin Wu. Autopruner: An end-to-end trainable filter pruning method for efficient deep model inference. arXiv preprint arXiv:1805.08941 2018.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 5058-5066, 2017.
Rongrong Ma, Jianyu Miao, Lingfeng Niu, and Peng Zhang. Transformed E regularization for learning sparse deep neural networks. Neural Networks, 2019.
Konstantinos Makantasis, Konstantinos Karantzalos, Anastasios Doulamis, and Nikolaos Doulamis. Deep supervised learning for hyperspectral data classification through convolutional neural net- works. In 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pp. 4962. IEEE, 2015.
10

---

Published as a conference paper at ICLR 2020
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In Proceedings of the 34th International Conference on Machine Learning- Soinning-Vooumme Volume 70, pp. 2498-2507. JMLR. org, 2017.
Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, and Dmitry P Vetrov, Structured bayesian pruning via log-normal multiplicative noise. In Advances in Neural Information Processing Systems, pp. 6775-6784 2017.
Jongsoo Park, Sheng Li, Wei Wen, Ping Tak Peter Tang, Hai Li, Yiran Chen, and Pradeep Dubey. Faster cnns with direct sparse convolutions and guided pruning. arXiv preprint arXiv: 00409, 2016.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Dito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, W, 2017.
Audrey Repetti, Mai Quyen Pham, Laurent Duval, Emilie Chouzenoux, and Jean- Chanshhhitttphee Pesquet. Euclid in a taxicab: Sparse blind deconvolution with smoothed 2,/22 regularization. IEEE signal processing letters, 22(5):539-543, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-2522 2015. doi: 10.1007/s1 1263-015-0816-y:
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Pravendra Singh, Vinay Kumar Verma, Piyush Rai, and Vinay P Namboodiri. Leveraging filter correlations for deep model compression. arXiv preprint 181. 10559, 2018.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pp. 1139-1147, 2013.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series (Methodological), 58(1):667-288 1996.
Anmenak Petrosyan Tran, Clayton Webster, et al. Reconstruction of jointly sparse vectors via manifold optimization. arXiv preprint
2018.
Fei Wen, Lei Chu, Peilin Liu, and Robert C Qiu. A survey on nonconvex regularization bassed sparse and low-rank recovery in signal processing, statistics, and machine learning. IEEE Access, 6: 69883-69906, 2018.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pp. 2074-2082, 2016.
Wei Wen, Yuxiong He, Samyam Rajbhandari, Minjia Zhang, Wenhan Wang, Fang Liu, Bin Hu, Yiran Chen, and Hai Li. Learning intrinsic sparse structures within long short-term memory. arXiv preprint
2017.
Penghang Yin, Ernie Esser, and Jack Xin. Ratio and difference of 11 and 12 norms and sparse representation with coherent dictionaries. Commun. Inform. Systems, 14(2):87-109, 2014.
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad Morariu, Xintong Han, Mingfei Gao, Ching Yung Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9194-9203, 2018.
11

---

Published as a conference paper at ICLR 2020
Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(1):49 67, 2006.
Jihun Yun, Peng Zheng, Eunho Yang, Aurelie Lozano, and Aleksandr Aravkin. Trimming the & regularizer: Statistical analysis, optimization, and applications deep learning. In International Conference on Machine Learning, pp. 7242-7251, 2019.
Cun-Hui Zhang et al. Nearly unbiased variable selection under minimax concave penalty. The Annals of statistics, 38(2):894 942, 2010.
Tianyun Zhang. Shaokai Ye, Kaiqi Zhang, Jian Tang, Wujie Wen, Makan Fardad, and Yanzhi Wang. A systematic dnn weight pruning framework using alternttng direction method of multipliers. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 184-199, 2018.
Zhuangwei Zhuang. Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrimination-awen channel pruning for deep neural networks. In Advances in Neural Information Processing Systems, pp. 875-886, 2018.
12

---

Published as a conference paper at ICLR 2020
A DERIVATION OF DEEPHOYER REGULARIZERS' GRADIENTS
In this section we provide detailed derivation of the gradient of the Hoyer-Square regularizer and the
Group-Gs Soup-GS regularizer w.r.t. an element wy in the weight matrix W.
The gradient of the Hoyer-Square regularizer is shown in Equation Equation (4) achieved at the end of the derivation.
( The formulation shown in
Ows Hs(W)= = 10w, (E [ 1))] Σ, w 10u,
EE, w ) mx/l)
210 wr (1ee [J] Σ; |w;
> u 2w |E
12
Enboyl
(sign(wy)
Σ w sign(wy)) (ay) Duas))
[E w9)2
Σ; |wal L,pl
2sign(w,)
w
|wg)
we l).
E i
The gradient of the Group-HS regularizer shown Equation (T). For simplicity we use the form shown in the second equality of Equation (o). where there is no overlapping between the groups. Here we assume that wy belongs to group w(1)
[Eg= (1 12)
Ow, (n) ow;
Σ, wi
1O, (12 6
(15g= (( [2) E, w. Ow, (E, w g= 1 12)
(E, "x?
2(0w, (1/2r(00)
[|2)]
G
||w(())
11 E, wp 2w (E g=1 |w ||2)
g=1l
C w?? 2,
(10)
2 >g ||206))/12
Wj
Σ w/ w
(o) (12)
E w2)²
(wr(( |1
= 2
Wy
E
|wal
||
112
|2 12)
(3)
w
9=1
B DETAILED EXPERIMENT SETUP
B. MNIST EXPERIMENTS
The MNIST dataset (LeCun et al. 1998) is well known handwritten digit dataset consists of grey- scale images with the size of 28 x 28 pixels. We use the dataset API provided in the "torchvision" python package to access the dataset. In our experiments we use the whole 60,000 training set images for the training and the whole 10,000 testing set images for the evaluation. All the accuracy results reported in the paper are evaluated on the testing set. Both the training set and the testing set are normalized have zero mean and variance one, Adam optimizer (Kingma Bal (0144) with learning rate 0.001 is used throughout the training process. All the MNIST experiments are done with single TITAN XP GPU.
Both the LeNet-300-100 model and the LeNet-5 model are firstly pretrained without the sparsity- inducing regularizer, where they achieve the testing accuracy of 98.4% and 99.2% respectively. Then the models are further trained for 250 epochs with the DeepHoyer regularizers applied in the objective. The weight decay parameters (as in Equation 7 and (8) are picked by hand to reach the best result. In the last step, prune the weight of each layer with threshold proportional to the standard derivation of each layer's weight. The threshold/std ratio is chosen to achieve the highest
sparsity without accuracy loss. All weight elements with absolute value smaller than the threshold
13

---

Published as a conference paper at ICLR 2020
Table 6: Hyper parameter used for MNIST benchmarks
Model
LeNet-300-100
LeNet-5
Regularizer
Decay
Threshold/std
Decay
Threshold/std
Hoyer
0.02
0.05
0.01
0.08
Hoyer-Square
0.0002
0.03
0.0001
0.03
Group-HS
0.002
0.8
0.1
0.008
Transformed &
2c-5
0.3
2c-5
0.6
set to zero and is fixed during the final finetuning. The pruned model is finetuned for another 100 steps without DeepHoyer regularizers and the best testing accuracy achieved is reported. Detailed parameter choices used in achieving the reported results are listed in Table
B.2 IMAGENET AND CIFAR-10 10 EXPERIMENTS
The ImageNet dataset is a large-scale color-image dataset containing 1.2 million images of 1000 categories (Russakovsky et al. 2015), which has long been utilized as an important bench- mark on image classification problems. In this paper, we use the "ILSVRC2012" version of the dataset, which can be found at http: www. imagemmaeenn net org cha Lwngee//sggsse 2012 / /oopunbp down Sowwlooee Loads We use all the data in the provided training set to train our model, and use the provided validation set to evaluate our model and report the testing accuracy. We follow the data reading and preprocessing pipeline suggested by the official Py PyTorch ImageNet example (https:: //gtthb.. com/ hwtp//yyorcch/ccc examples /ttteteeeeee mwsttmtttee/..m/eam. imagenet). For training images, we first randomly crop the training images to desired input size, then apply random horizontal flipping and finally normalize them before feeding them into the network. Validation images are first resized to 256 x 256 pixels, then center cropped to desired input size and normalized in the end. We use input size 227 x 227 pixels for experiments on the AlexNet, and input size 224 x 224 for experiments on the ResNet-50. All the models are optimized with the SGD optimizer Sutskever et al.. (2013), and the batch size is chosen as 256 for all the experiments. Two TITAN XP GPUs are used in parallel for the AlexNet training and four are used for the ResNet-50 training.
One thing worth noticing is that the AlexNet model provided in the "torchvision" package is not the ordinary version used in previous works Haan et al. (2015b); WWen et al.] (2016);; ( /Zhang al. (2018). Therefore we reimplement the AlexNet model in PyTorch for fair comparison. We pretrain the implemented model for 90 epochs and achieve 19.8 % top-5 error, which is the same as reported in previous works. In the AlexNet experiment, the reported result in Table 3 is achieved by applying the Hoyer-Square regularizer with decay parameter 1e-6. Before the pruning, the model is firstly train from the pretrained model with the Hoyer-Squan regularizer for 90 epochs, where initial learning rate 0.001 is used. An Eg regularization with le-4 decay is also applied. We then prune the convolution layers with threshold 1e-4 and the FC layers with threshold equal 0.4x of their standard derivations. The model is then finetuned until the best accuracy is reached. The learning rate decayed by 0.1 for every 30 epochs of training. The training process with the Hoyer regularizer and the T', regularizer (Ma et al. 2019) is the same as the HS regularizer. For the reported result, we use decay le-3 and FC threshold 0.8x std for the Hoyer regularizer, and use decay 2e-5 and FC threshold 1.0x std for the T&, regularizer.
For the Re experiments on ImageNet, the model architecture and pretrained model provided in the "torchvision" package is directly utilized, which achieves 23.85% top-1 error and 7.13% top-5 error. All the reported results Figure 3 and Table are achieved with 90 epochs of training with the Group-HS regularizer from the pretrained model using initial learning rate O. All the models are pruned with lc-4 as threshold and finetuned to the best accuracy. We only tune the decay parameter of the Group-HS regularizer to explore the accuracy-FLOPs tradeoff. The exact decay parameter used for each result is specified in Table g
We also use the CIFAR-10 dataset (Krizhevsky & Hinton| 2009) evaluate the structural pruning performance on ResNet-56 and ResNet- models. The CIFAR-10 dataset can be directly accessed through the dataset API provided in the "torchvision" python package. Standard preprocessing,
14

---

Published as a conference paper at ICLR 2020
x10
Pretrained
Trained with Hg. regularizer
Pruned and finetuned
1.5
300
Count
Count
Count
200
0.5
0
0
0
Weight value
Weight
Weight
x10
3000
1.5
Count
Count
Count
1
40
1000
0.5
0
0
-1
Weight value
Weight value
Weight value
600
Count
Count
400
Count
40
200
20
-0.5
0
-0.5
Weight value
Weight value
Weight value
Figure 4: Histogram of nonzero weight elements of each layer in the LeNet-300- 100 model. From top to bottom corresponds to layer FC1, FC2, FC3 respectively. The original pretrained model is shown in column 1, column 2 shows the model achieved after Hs regularization, column shows the final model after pruning and finetuning.
including random crop, horizontal flip and normalization is used on the training set to train the model. We implemented the ResNet models for CIFAR-10 following the description in (He et al.) 2016)). and pretrain the models for 164 epochs. Learning rate is set to 0.1 initially, and decayed by 0.1 at epoch 81 and epoch 122. The pretrained ResNet-56 model reaches the testing accuracy of 93.14 % while the ResNet-1 10 model reaches 93.62 %. Similar the ResNet-50 experiment, the pretrained models and train with the Group-HS regularizer. Same learning rate scheduling is used for both pretraining and training with Group-HS. All the models are pruned with le-4 as threshold and finetuned to the best accuracy. The decay parameters of the Group-HS regularizer used to get the result Figure is specified in Table Table 10
start with
C ADDITIONAL EXPERIMENT RESULTS
C.I WEIGHT DISTRIBUTION AT DIFFERENT STAGES
Here we demonstrate how will the weight distribution change in each layer at different stages of our element-wine pruning process. Since most of the weight elements will be zero in the end, we only plot the histogram of nonzero weight elements for better observation. The histogram of each layer of the L00-1-300-100 model and the LeNet-5 model are visualized in Figure f and Figure 5 respectively. It can be seen that majority of the weights will be concentrated near zero after applying the Hs regularizer during training, while rest of the weight elements will spread out in a wide range. The weights close to are then set be exactly zero, and the model is finetuned with zero weights fixed. The resulted histogram shows that most of the weights are pruned away, only a small amount of nonzero weights remaining in the model.
15

---

Published as a conference paper at ICLR 2020
Pretrained
Trained with Hg regularizer
Pruned and finetuned
30
200
Count
20
Count
Count
100
-0.5
0.5
-1
-0.5
0
0.5
0
Weight valuee
Weight value
Weight value
2000
40
10000
Count
Count
Count
1000
5000
0
-0.5
0
0.5
-1
1
0
Weight value
Weight value
Weight value
x10
x10
100
Count
Count
Count
2
-0.5
0
0.5
O
Weight value
Weight value
Weight value
600
4000
Count
400
Count
Count
2000
-0.5
0.5
-0.5
0.5
Weight value
Weight value
Weight value
Figure 5: Histogram of nonzero weight elements of each layer in the LeNet-5 model. From top to bottom corresponds to layer CONV1, CONV2, FC1, FC2 respectively. The original pretrained model shown in column column shows the model achieved after Iis regularization, column shows the final model after pruning and finetuning.
C.2 LAYER-BY-LAYER COMPARISON OF ELEMENT-WISS PRUNING RESULT OF ALEXNET
Table compares the element-wir wise pruming result of the Hoyer-Square Square regularizer AlexNet with other methods in a layer-by-layer fashion. It can be seen that the Hoyer-Square regularizer achieves high pruning rates the largest layers (i.e. FC1-3). This observation is consistent with the observation made on the element-wiss pruning performance of models on the MNIST dataset.
C.3 DETAILED RESULTS OF THE RESNET EXPERIMENTS
In this section we list the data used to plot Figure Table & shows the result of pruning ResNet-50 model ImageNet, Table 9 shows the result of pruning ResNet-56 model on CIFAR- 10 and Table I0 shows the result of pruning ResNet-1ll model on CIFAR 10. For all the tables, the results of previous works are listed on the top, and are ordered based publication year. Results achieved with the Group-HS regularizer are listed below, marked with the regularization strength used for the training.
16

---

Published as a conference paper at ICLR 2020
Table 7: Element-wise pruning results on AlexNet without accuracy loss. Refer Table for the full reference of the mentined methods.
Layer
Nonzero wights left after pruning
Baseline
Han et al.
Zhang et al.
Ma et al.
Hoyer
HS
CONVI
34.8K
29.3K
28.2K
24.2K
21.3K
31.6K
CONV2
307.2K
116.7K
61.4K
109.9K
77.2K
148.4K
CONV3
884.7K
309.7K
168.1K
241.2K
192.0K
299.3K
CONV4
663.5K
245.5K
132.7K
207.4K
182.6K
275.6K
CONV5
442.2K
163.7K
88.5K
134.7K
116.6K
197.1K
FCI
37.7M
3,40M
1.06M
0.763M
1.566M
0.781M
FC2
16.8M
1,51M
0.99M
1.070M
0.974M
0.650M
FC3
4.10M
1.02M
0.38M
0.505M
0.490M
0.472M
Total
60.9M
6.8M
2.9M
3.05M
3.62M
2.85M
Table 8: Structural pruning result of the ResNet-50 model on imageNet.
Model
Top-1 acc
Top-5 acc
#FLOPs reduction
Orig
76.15%
92.87%
1.00x
Channel pruning (He
al.
2017)
N/A
90.80%
2.00x
ThiNet-70
Luo et al.
2017)
72.04%
90.67%
1.58x
ThiNet-50
Luo et
2017
71.01%
90.02%
2.26x
ThiNet-30
[uo et al.
2017)
68.42%
88.30%
3.51
SSS (Huang
Wang
2018)
74.18%
91.91% %
1.45x
SFP (He et al. 201 8a)
74.61%
92.06%
1.72x
CFP (Singh al. 2018
73.4%
91.4%
1.98x
Autopruner (Luo & Wu
20188)
74.76%
92.15%
2.05x
GDP (Lin et al.
2018)
71.89%
90.71%
2.05
DCP (Zhang et
2018
74.95%
92.32%
2.26x
SSR- (Lin et al.
2019
71.47%
90.19%
2.26x
C-SGD-70 [Din
2019
75.27%
92.46%
1.58x
C-SGD-50 (Ding et al.
2019
74.93%
92.27%
1.86x
C-SGD-30 (Ding et
2019
74.54%
92.09%
2.26x
CNN-FCF-A [Li et al.
2019
76.50%
93.13%
1.41x
CNN-FCF-B
al. 2019
75.68%
92.68%
1.85
CNN-FCF-C
Li et al.
2019
74.55%
92.18%
2.33x
CNN-FCF-D
al.
2019
73.54%
91.50%
2.96x
Group-Hi HS le-5
76.43%
93.07%
1.89x
Group-HS 2e-5
75.20%
92.52%
3.09
Group-HS 3e-5
73.19%
91.36%
4.68
Group-HS 4e-5
71.08%
90.21%
5.48x
17

---

Published as a conference paper at ICLR 2020
Table 9: Structural pruning result of the ResNet-56 model CIFAR-10.
Model
Base acc
Acc gain
#FLOPs reduction
Pruning-A (Li al.
2016
93.04%
+0.06%
1.12x
Pruning-B L1 et al. 2016
93.04%
+0.02%
1.38x
Channel pruning (He al. 2017)
92.8%
-1.0%
2.00
NISP-56 Yu et al. 2018
N/A
-0.03%
1.77
SFP (He et al.) 2018a)
93.59%
+0.19%
1.70x
AMC (ee ct al. (018b)
92.8%
-0.9%
2.00x
C-SGD-5A 5/8
(Ding al. 2019
93.39%
+0.23%
2.55x
CNN-FCF-A Li et al. 2019
93.14%
+0.24%
1.75x
CNN-FCF-B et al.
2019
93.14%
-1.22%
3.44x
Group- HS 2e-4
93.14%
+0.44%
2.38x
Group- HS 2.5e-4
93.14%
+0.31%
3.07
Group-HS 3e-4
93.14%
-0.24%
3.52x
Group-HS 5e-4
93.14%
-0.91%
5.63x
Table 10: Structural pruning result the ResNet- 110 model on CIFAR-10.
Model
Base ace
Ace gain
#FL.Oss reduction
Pruning-A (Li et al.
2016
93.53%
-0.02%
1.19x
Pruning-B (Li
2016
93.53%
-0.23%
1.62
NISP-110 Yu et al.
2018)
N/A
-0.18%
1.78x
SFP He et
93.68%
+0.18%
1.69x
C-SGD-578 Ding el al. 2019
94.38%
+0.03%
2.56x
CNN-FCF-
2019
93.58%
+0.09%
1.76x
CNN-FCF-B
Li al.
2019
93.58%
-0.62%
3.42x
Group-HS 7e-5
93.62%
+0.44%
2.30x
Group-HS lc-4
93.62%
+0.18%
3.09
Group-HS HS 1.5c-4
93.62%
-0.08%
4.38x
Group-HS 2e-4
93.62%
-0.65%
5.84x
18


Reviews:
[{"review_id": "HygbT3sqKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "To enforce sparsity in neural networks, the paper proposes a scale-invariant regularizer (DeepHoyer) inspired by the Hoyer measure. It is simply the ratio between l1 and l2 norm, which is almost everywhere differentiable, and enforces element-wise sparsity. It further proposes the Hoyer measure to quantify sparsity and applies the DeepHoyer in DNN training to train pruned models. The extension of Hoyer-Square is also straightforward. \n\nI generally enjoy simple yet effective ideas. The idea is very straightforward and well intuitive. The paper is well written and easy the follow. The discussion on the Hoyer measure is inspiring and the empirical studies on various different network architecture/datasets compared to several competitive baselines verify the effectiveness of the DeepHoyer model. \n\nTherefore I'm leaning to accept it. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HkllzwGI5B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper focuses on sparse neural networks. Typically, l1 regularization is the go-to strategy, however, it is not scale invariant. That is, all weights are affected by the regularization, not only those that are being driven to 0. l0 regularization is theoretically optimal, however, it is not smooth and has no gradients almost everywhere, so it cannot be used for training. As a compromise the paper proposes Hoyer regularization, that is the l1/l2 ratio. The Hoyer regularization has the same minima structure and leads to sparse solutions while being scale invariant, that is it does not affect all weights in the process. Additionally, the paper proposes structured Hoyer regularization. Last, it employs the said regularizations in deep networks: LeNet, AlexNet and ResNet on several datasets: MNIST, CIFAR, ImageNet.\n\nStrengths:\n+ The described method is simple, intuitive and straightforward. By applying the said regularization (~ Σ |w_i|/sqrt(Σ w_i^2)), one arrives at seemingly sparser solutions, which is verified in practice.\n\n+ The experiments are extensive and convincing. I particularly like that the authors have used their method with complex and deep models like ResNets, on large scale datasets like ImageNet.\n\n+ The presentation is generally clear and one can understand the paper straightaway.\n\nWeaknesses:\n+ The contributions of the paper are rather on the thin side. At the end of the day, Hoyer regularization is taken from another field (compressed sensing) and applied on deep networks. This is also witnessed by some moderate repetition in the writing, e.g., between the introduction and the related work.\n\n+ There are some points where the paper becomes unclear. For instance, in Figure 3 what are the \"other methods\"?\n\n+ In Figure 1 it is explained that the Hoyer regularization leads to minima along the axis. The gradients then push the models \"rotationally\". Could this lead to bad multiple local optimal problems? Is there any guarantee that any particular axis will generate better solutions than the other?\n\nAll in all, I would recommend for now weak accept. I find the work interesting and solid, although not that exciting.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Byx6q3uqtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is written very nicely and the experiments are convincing (though you can always show more)\n\nIn terms of novelty, I shamelessly can say the idea is very simple and the basic form, the Hoyer regularization, was known. That said, I am all in for simple and efficient solutions so I am giving this paper a weak accept for now.\n\nThere is not much to ask here (unless I say I want to see how this would work on other backbones and problems). nevertheless, to improve this work, I think the authors need to compare which solution (referring to algorithms in Table1, 2 etc.) is faster/more well-behaved given the combo explained at the bottom of page 5 . This is basically my question/request for the rebuttal.  ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a scale-invariant sparsity measure for deep networks. The experiments are extensive and convincing, according to reviewers. I recommend acceptance.
Decision: Accept (Poster)