DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures

1 INTRODUCTION . The use of deep neural network ( DNN ) models has been expanded from handwritten digit recognition ( LeCun et al. , 1998 ) to real-world applications , such as large-scale image classification ( Simonyan & Zisserman , 2014 ) , self driving ( Makantasis et al. , 2015 ) and complex control problems ( Mnih et al. , 2013 ) . However , a modern DNN model like AlexNet ( Krizhevsky et al. , 2012 ) or ResNet ( He et al. , 2016 ) often introduces a large number of parameters and computation load , which makes the deployment and real-time processing on embedded and edge devices extremely difficult ( Han et al. , 2015b ; a ; Wen et al. , 2016 ) . Thus , model compression techniques , especially pruning methods that increase the sparsity of weight matrices , have been extensively studied to reduce the memory consumption and computation cost of DNNs ( Han et al. , 2015b ; a ; Wen et al. , 2016 ; Guo et al. , 2016 ; Louizos et al. , 2017b ; Luo et al. , 2017 ; Zhang et al. , 2018 ; Liu et al. , 2015 ) . Most of the previous works utilize some form of sparsity-inducing regularizer in searching for sparse neural networks . The ` 1 regularizer , originally proposed by Tibshirani ( 1996 ) , can be easily optimized through gradient descent for its convex and almost everywhere differentiable property . Therefore it is widely used in DNN pruning : Liu et al . ( 2015 ) directly apply ` 1 regularization to all the weights of a DNN to achieve element-wise sparsity ; Wen et al . ( 2016 ; 2017 ) present structural sparsity via group lasso , which applies an ` 1 regularization over the ` 2 norms of different groups of parameters . However , it has been noted that the value of the ` 1 regularizer is proportional to the scaling of parameters ( i.e . ||αW ||1= |α|·||W ||1 ) , so it “ scales down ” all the elements in the weight matrices with the same speed . This is not efficient in finding sparsity and may sacrifice the flexibility of the trained model . On the other hand , the ` 0 regularizer directly reflects the real sparsity of weights and is scale invariant ( i.e . ||αW ||0= ||W ||0 , ∀α 6= 0 ) , yet the ` 0 norm can not provide useful gradients . Han et al . ( 2015b ) enforce an element-wise ` 0 constraint by iterative pruning a fixed percentage of smallest weight elements , which is a heuristic method and therefore can hardly achieve optimal compression rate . Some recent works mitigate the lack of gradient information by integrating ` 0 regularization with stochastic approximation ( Louizos et al. , 2017b ) or more complex optimization methods ( e.g . ADMM ) ( Zhang et al. , 2018 ) . These additional measures brought overheads to the optimization process , making the use of these methods on larger networks difficult . To achieve even sparser neural networks , we argue to move beyond ` 0 and ` 1 regularizers and seek for a sparsity-inducing regularizer that is both almost everywhere differentiable ( like ` 1 ) and scale-invariant ( like ` 0 ) . Beyond the ` 1 regularizer , plenty of non-convex sparsity measurements have been used in the field of feature selection and compressed sensing ( Hurley & Rickard , 2009 ; Wen et al. , 2018 ) . Some popular regularizers like SCAD ( Fan & Li , 2001 ) , MDP ( Zhang et al. , 2010 ) and Trimmed ` 1 ( Yun et al. , 2019 ) use a piece-wise formulation to mitigate the proportional scaling problem of ` 1 . The piece-wise formulation protects larger elements by having zero penalty to elements greater than a predefined threshold . However , it is extremely costly to manually seek for the optimal trimming threshold , so it is hard to obtain optimal result in DNN pruning by using these regularizers . The transformed ` 1 regularizer formulated as ∑N i=1 ( a+1 ) |wi| a+|wi| manages to smoothly interpolate between ` 1 and ` 0 by tuning the hyperparameter a ( Ma et al. , 2019 ) . However , such an approximation is close to ` 0 only when a approaches infinity , so the practical formulation of the transformed ` 1 ( i.e . a = 1 ) is still not scale-invariant . Particularly , we are interested in the Hoyer regularizer ( Hoyer , 2004 ) , which estimates the sparsity of a vector with the ratio between its ` 1 and ` 2 norms . Comparing to other sparsity-inducing regularizers , Hoyer regularizer achieves superior performance in the fields of non-negative matrix factorization ( Hoyer , 2004 ) , sparse reconstruction ( Esser et al. , 2013 ; Tran et al. , 2018 ) and blend deconvolution ( Krishnan et al. , 2011 ; Repetti et al. , 2015 ) . We note that Hoyer regularizer is both almost everywhere differentiable and scale invariant , satisfying the desired property of a sparsityinducing regularizer . We therefore propose DeepHoyer , which is the first Hoyer-inspired regularizers for DNN sparsification . Specifically , the contributions of this work include : • Hoyer-Square ( HS ) regularizer for element-wise sparsity : We enhance the original Hoyer regularizer to the HS regularizer and achieve element-wise sparsity by applying it in the training of DNNs . The HS regularizer is both almost everywhere differentiable and scale invariant . It has the same range and minima structure as the ` 0 norm . Thus , the HS regularizer presents the ability of turning small weights to zero while protecting and maintaining those weights that are larger than an induced , gradually adaptive threshold ; • Group-HS regularizer for structural sparsity , which is extended from the HS regularizer ; • Generating sparser DNN models : Our experiments show that the proposed regularizers beat state-of-the-arts in both element-wise and structural weight pruning of modern DNNs . 2 RELATED WORK ON DNN PRUNING . It is well known that high redundancy pervasively exists in DNNs . Consequently , pruning methods have been extensively investigated to identify and remove unimportant weights . Some heuristic pruning methods ( Han et al. , 2015b ; Guo et al. , 2016 ) simply remove weights in small values to generate sparse models . These methods usually require long training time without ensuring the optimality , due to the lack of theoretical understanding and well-formulated optimization ( Zhang et al. , 2018 ) . Some works formulate the problem as a sparsity-inducing optimization problem , such as ` 1 regularization ( Liu et al. , 2015 ; Park et al. , 2016 ) that can be optimized using standard gradientbased algorithms , or ` 0 regularization ( Louizos et al. , 2017b ; Zhang et al. , 2018 ) which requires stochastic approximation or special optimization techniques . We propose DeepHoyer regularizers in this work , which belong to the line of sparsity-inducing optimization research . More specific , the proposed Hoyer-Square regularizer for element-wise pruning is scale-invariant and can serve as an differentiable approximation to the ` 0 norm . Furthermore , it can be optimized by gradient-based optimization methods in the same way as the ` 1 regularization . With these properties , the HoyerSquare regularizer achieves a further 38 % and 63 % sparsity improvement on LeNet-300-100 model and LeNet-5 model respectively comparing to previous state-of-the-arts , and achieves the highest sparsity on AlexNet without accuracy loss . Structurally sparse DNNs attempt to create regular sparse patterns that are friendly for hardware execution . To achieve the goal , Li et al . ( 2016 ) propose to remove filters with small norms ; Wen et al . ( 2016 ) apply group Lasso regularization based methods to remove various structures ( e.g. , filters , channels , layers ) in DNNs and the similar approaches are used to remove neurons ( Alvarez & Salzmann , 2016 ) ; Liu et al . ( 2017 ) and Gordon et al . ( 2018 ) ( MorphNet ) enforce sparsity-inducing regularization on the scaling parameters within Batch Normalization layers to remove the corresponding channels in DNNs ; ThiNet ( Luo et al. , 2017 ) removes unimportant filters by minimizing the reconstruction error of feature maps ; and He et al . ( 2017 ) incorporate both Lasso regression and reconstruction error into the optimization problem . Bayesian optimization methods have also been applied for neuron pruning ( Louizos et al. , 2017a ; Neklyudov et al. , 2017 ) , yet these methods are not applicable in large-scale problems like ImageNet . We further advance the DeepHoyer to learn structured sparsity ( such as reducing filters and channels ) with the newly proposed “ Group-HS ” regularization . The Group-HS regularizer further improves the computation reduction of the LeNet-5 model by 8.8 % from the ` 1 based method ( Wen et al. , 2016 ) , and by 110.6 % from the ` 0 based method ( Louizos et al. , 2017b ) . Experiments on ResNet models reveal that the accuracy-speedup tradeoff induced by Group-HS constantly stays above the Pareto frontier of previous methods . More detailed results can be found in Section 5 . 3 MEASURING SPARSITY WITH THE HOYER MEASURE . Sparsity measures provide tractable sparsity constraints for enforcement during problem solving and therefore have been extensively studied in the compressed sensing society . In early non-negative matrix factorization ( NMF ) research , a consensus was that a sparsity measure should map a ndimensional vector X to a real number S ∈ [ 0 , 1 ] , such that the possible sparsest vectors with only one nonzero element has S = 1 , and a vector with all equal elements has S = 0 ( Hoyer , 2004 ) . Unders the assumption , the Hoyer measure was proposed as follows S ( X ) = √ n− ( ∑ i|xi| ) / √∑ i x 2 i√ n− 1 . ( 1 ) It can be seen that 1 ≤ ∑ i|xi|√∑ i x 2 i ≤ √ n , ∀X ∈ Rn . ( 2 ) Thus , the normalization in Equation ( 1 ) fits the measure S ( X ) into the [ 0 , 1 ] interval . According to the survey by Hurley & Rickard ( 2009 ) , among the six desired heuristic criteria of sparsity measures , the Hoyer measure satisfies five , more than all other commonly applied sparsity measures . Given its success as a sparsity measure in NMF , the Hoyer measure has been applied as a sparsity-inducing regularizer in optimization problems such as blind deconvolution ( Repetti et al. , 2015 ) and image deblurring ( Krishnan et al. , 2011 ) . Without the range constraint , the Hoyer regularizer in these works adopts the form R ( X ) = ∑ i|xi|√∑ i x 2 i directly , as the ratio of the ` 1 and ` 2 norms . Figure 1 compares the Hoyer regularizer and the ` 1 regularizer . Unlike the the ` 1 norm with a single minimum at the origin , the Hoyer regularizer has minima along axes , the structure of which is very similar to the ` 0 norm ’ s . Moreover , the Hoyer regularizer is scale-invariant , i.e . R ( αX ) = R ( X ) , because both the ` 1 norm and the ` 2 norm are proportional to the scale of X . The gradients of the Hoyer regularizer are purely radial , leading to “ rotations ” towards the nearest axis . These features make the Hoyer regularizer outperform the ` 1 regularizer on various tasks ( Esser et al. , 2013 ; Tran et al. , 2018 ; Krishnan et al. , 2011 ; Repetti et al. , 2015 ) . The theoretical analysis by Yin et al . ( 2014 ) also proves that the Hoyer regularizer has a better guarantee than the ` 1 norm on recovering sparse solutions from coherent and redundant representations .

Summary: The paper focuses on sparse neural networks. Typically, l1 regularization is the go-to strategy, however, it is not scale invariant. That is, all weights are affected by the regularization, not only those that are being driven to 0. l0 regularization is theoretically optimal, however, it is not smooth and has no gradients almost everywhere, so it cannot be used for training. As a compromise the paper proposes Hoyer regularization, that is the l1/l2 ratio. The Hoyer regularization has the same minima structure and leads to sparse solutions while being scale invariant, that is it does not affect all weights in the process. Additionally, the paper proposes structured Hoyer regularization. Last, it employs the said regularizations in deep networks: LeNet, AlexNet and ResNet on several datasets: MNIST, CIFAR, ImageNet.