DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures

1 INTRODUCTION . The use of deep neural network ( DNN ) models has been expanded from handwritten digit recognition ( LeCun et al. , 1998 ) to real-world applications , such as large-scale image classification ( Simonyan & Zisserman , 2014 ) , self driving ( Makantasis et al. , 2015 ) and complex control problems ( Mnih et al. , 2013 ) . However , a modern DNN model like AlexNet ( Krizhevsky et al. , 2012 ) or ResNet ( He et al. , 2016 ) often introduces a large number of parameters and computation load , which makes the deployment and real-time processing on embedded and edge devices extremely difficult ( Han et al. , 2015b ; a ; Wen et al. , 2016 ) . Thus , model compression techniques , especially pruning methods that increase the sparsity of weight matrices , have been extensively studied to reduce the memory consumption and computation cost of DNNs ( Han et al. , 2015b ; a ; Wen et al. , 2016 ; Guo et al. , 2016 ; Louizos et al. , 2017b ; Luo et al. , 2017 ; Zhang et al. , 2018 ; Liu et al. , 2015 ) . Most of the previous works utilize some form of sparsity-inducing regularizer in searching for sparse neural networks . The ` 1 regularizer , originally proposed by Tibshirani ( 1996 ) , can be easily optimized through gradient descent for its convex and almost everywhere differentiable property . Therefore it is widely used in DNN pruning : Liu et al . ( 2015 ) directly apply ` 1 regularization to all the weights of a DNN to achieve element-wise sparsity ; Wen et al . ( 2016 ; 2017 ) present structural sparsity via group lasso , which applies an ` 1 regularization over the ` 2 norms of different groups of parameters . However , it has been noted that the value of the ` 1 regularizer is proportional to the scaling of parameters ( i.e . ||αW ||1= |α|·||W ||1 ) , so it “ scales down ” all the elements in the weight matrices with the same speed . This is not efficient in finding sparsity and may sacrifice the flexibility of the trained model . On the other hand , the ` 0 regularizer directly reflects the real sparsity of weights and is scale invariant ( i.e . ||αW ||0= ||W ||0 , ∀α 6= 0 ) , yet the ` 0 norm can not provide useful gradients . Han et al . ( 2015b ) enforce an element-wise ` 0 constraint by iterative pruning a fixed percentage of smallest weight elements , which is a heuristic method and therefore can hardly achieve optimal compression rate . Some recent works mitigate the lack of gradient information by integrating ` 0 regularization with stochastic approximation ( Louizos et al. , 2017b ) or more complex optimization methods ( e.g . ADMM ) ( Zhang et al. , 2018 ) . These additional measures brought overheads to the optimization process , making the use of these methods on larger networks difficult . To achieve even sparser neural networks , we argue to move beyond ` 0 and ` 1 regularizers and seek for a sparsity-inducing regularizer that is both almost everywhere differentiable ( like ` 1 ) and scale-invariant ( like ` 0 ) . Beyond the ` 1 regularizer , plenty of non-convex sparsity measurements have been used in the field of feature selection and compressed sensing ( Hurley & Rickard , 2009 ; Wen et al. , 2018 ) . Some popular regularizers like SCAD ( Fan & Li , 2001 ) , MDP ( Zhang et al. , 2010 ) and Trimmed ` 1 ( Yun et al. , 2019 ) use a piece-wise formulation to mitigate the proportional scaling problem of ` 1 . The piece-wise formulation protects larger elements by having zero penalty to elements greater than a predefined threshold . However , it is extremely costly to manually seek for the optimal trimming threshold , so it is hard to obtain optimal result in DNN pruning by using these regularizers . The transformed ` 1 regularizer formulated as ∑N i=1 ( a+1 ) |wi| a+|wi| manages to smoothly interpolate between ` 1 and ` 0 by tuning the hyperparameter a ( Ma et al. , 2019 ) . However , such an approximation is close to ` 0 only when a approaches infinity , so the practical formulation of the transformed ` 1 ( i.e . a = 1 ) is still not scale-invariant . Particularly , we are interested in the Hoyer regularizer ( Hoyer , 2004 ) , which estimates the sparsity of a vector with the ratio between its ` 1 and ` 2 norms . Comparing to other sparsity-inducing regularizers , Hoyer regularizer achieves superior performance in the fields of non-negative matrix factorization ( Hoyer , 2004 ) , sparse reconstruction ( Esser et al. , 2013 ; Tran et al. , 2018 ) and blend deconvolution ( Krishnan et al. , 2011 ; Repetti et al. , 2015 ) . We note that Hoyer regularizer is both almost everywhere differentiable and scale invariant , satisfying the desired property of a sparsityinducing regularizer . We therefore propose DeepHoyer , which is the first Hoyer-inspired regularizers for DNN sparsification . Specifically , the contributions of this work include : • Hoyer-Square ( HS ) regularizer for element-wise sparsity : We enhance the original Hoyer regularizer to the HS regularizer and achieve element-wise sparsity by applying it in the training of DNNs . The HS regularizer is both almost everywhere differentiable and scale invariant . It has the same range and minima structure as the ` 0 norm . Thus , the HS regularizer presents the ability of turning small weights to zero while protecting and maintaining those weights that are larger than an induced , gradually adaptive threshold ; • Group-HS regularizer for structural sparsity , which is extended from the HS regularizer ; • Generating sparser DNN models : Our experiments show that the proposed regularizers beat state-of-the-arts in both element-wise and structural weight pruning of modern DNNs . 2 RELATED WORK ON DNN PRUNING . It is well known that high redundancy pervasively exists in DNNs . Consequently , pruning methods have been extensively investigated to identify and remove unimportant weights . Some heuristic pruning methods ( Han et al. , 2015b ; Guo et al. , 2016 ) simply remove weights in small values to generate sparse models . These methods usually require long training time without ensuring the optimality , due to the lack of theoretical understanding and well-formulated optimization ( Zhang et al. , 2018 ) . Some works formulate the problem as a sparsity-inducing optimization problem , such as ` 1 regularization ( Liu et al. , 2015 ; Park et al. , 2016 ) that can be optimized using standard gradientbased algorithms , or ` 0 regularization ( Louizos et al. , 2017b ; Zhang et al. , 2018 ) which requires stochastic approximation or special optimization techniques . We propose DeepHoyer regularizers in this work , which belong to the line of sparsity-inducing optimization research . More specific , the proposed Hoyer-Square regularizer for element-wise pruning is scale-invariant and can serve as an differentiable approximation to the ` 0 norm . Furthermore , it can be optimized by gradient-based optimization methods in the same way as the ` 1 regularization . With these properties , the HoyerSquare regularizer achieves a further 38 % and 63 % sparsity improvement on LeNet-300-100 model and LeNet-5 model respectively comparing to previous state-of-the-arts , and achieves the highest sparsity on AlexNet without accuracy loss . Structurally sparse DNNs attempt to create regular sparse patterns that are friendly for hardware execution . To achieve the goal , Li et al . ( 2016 ) propose to remove filters with small norms ; Wen et al . ( 2016 ) apply group Lasso regularization based methods to remove various structures ( e.g. , filters , channels , layers ) in DNNs and the similar approaches are used to remove neurons ( Alvarez & Salzmann , 2016 ) ; Liu et al . ( 2017 ) and Gordon et al . ( 2018 ) ( MorphNet ) enforce sparsity-inducing regularization on the scaling parameters within Batch Normalization layers to remove the corresponding channels in DNNs ; ThiNet ( Luo et al. , 2017 ) removes unimportant filters by minimizing the reconstruction error of feature maps ; and He et al . ( 2017 ) incorporate both Lasso regression and reconstruction error into the optimization problem . Bayesian optimization methods have also been applied for neuron pruning ( Louizos et al. , 2017a ; Neklyudov et al. , 2017 ) , yet these methods are not applicable in large-scale problems like ImageNet . We further advance the DeepHoyer to learn structured sparsity ( such as reducing filters and channels ) with the newly proposed “ Group-HS ” regularization . The Group-HS regularizer further improves the computation reduction of the LeNet-5 model by 8.8 % from the ` 1 based method ( Wen et al. , 2016 ) , and by 110.6 % from the ` 0 based method ( Louizos et al. , 2017b ) . Experiments on ResNet models reveal that the accuracy-speedup tradeoff induced by Group-HS constantly stays above the Pareto frontier of previous methods . More detailed results can be found in Section 5 . 3 MEASURING SPARSITY WITH THE HOYER MEASURE . Sparsity measures provide tractable sparsity constraints for enforcement during problem solving and therefore have been extensively studied in the compressed sensing society . In early non-negative matrix factorization ( NMF ) research , a consensus was that a sparsity measure should map a ndimensional vector X to a real number S ∈ [ 0 , 1 ] , such that the possible sparsest vectors with only one nonzero element has S = 1 , and a vector with all equal elements has S = 0 ( Hoyer , 2004 ) . Unders the assumption , the Hoyer measure was proposed as follows S ( X ) = √ n− ( ∑ i|xi| ) / √∑ i x 2 i√ n− 1 . ( 1 ) It can be seen that 1 ≤ ∑ i|xi|√∑ i x 2 i ≤ √ n , ∀X ∈ Rn . ( 2 ) Thus , the normalization in Equation ( 1 ) fits the measure S ( X ) into the [ 0 , 1 ] interval . According to the survey by Hurley & Rickard ( 2009 ) , among the six desired heuristic criteria of sparsity measures , the Hoyer measure satisfies five , more than all other commonly applied sparsity measures . Given its success as a sparsity measure in NMF , the Hoyer measure has been applied as a sparsity-inducing regularizer in optimization problems such as blind deconvolution ( Repetti et al. , 2015 ) and image deblurring ( Krishnan et al. , 2011 ) . Without the range constraint , the Hoyer regularizer in these works adopts the form R ( X ) = ∑ i|xi|√∑ i x 2 i directly , as the ratio of the ` 1 and ` 2 norms . Figure 1 compares the Hoyer regularizer and the ` 1 regularizer . Unlike the the ` 1 norm with a single minimum at the origin , the Hoyer regularizer has minima along axes , the structure of which is very similar to the ` 0 norm ’ s . Moreover , the Hoyer regularizer is scale-invariant , i.e . R ( αX ) = R ( X ) , because both the ` 1 norm and the ` 2 norm are proportional to the scale of X . The gradients of the Hoyer regularizer are purely radial , leading to “ rotations ” towards the nearest axis . These features make the Hoyer regularizer outperform the ` 1 regularizer on various tasks ( Esser et al. , 2013 ; Tran et al. , 2018 ; Krishnan et al. , 2011 ; Repetti et al. , 2015 ) . The theoretical analysis by Yin et al . ( 2014 ) also proves that the Hoyer regularizer has a better guarantee than the ` 1 norm on recovering sparse solutions from coherent and redundant representations .

Summary: To enforce sparsity in neural networks, the paper proposes a scale-invariant regularizer (DeepHoyer) inspired by the Hoyer measure. It is simply the ratio between l1 and l2 norm, which is almost everywhere differentiable, and enforces element-wise sparsity. It further proposes the Hoyer measure to quantify sparsity and applies the DeepHoyer in DNN training to train pruned models. The extension of Hoyer-Square is also straightforward.