Sparsifying Networks via Subdifferential Inclusion

1 INTRODUCTION . Deep neural networks have evolved to the state-of-the-art techniques in a wide array of applications : computer vision ( Simonyan & Zisserman , 2015 ; He et al. , 2016 ; Huang et al. , 2017 ) , automatic speech recognition ( Hannun et al. , 2014 ; Dong et al. , 2018 ; Li et al. , 2019 ; Watanabe et al. , 2018 ; Hayashi et al. , 2019 ; Inaguma et al. , 2020 ) , natural language processing ( Turc et al. , 2019 ; Radford et al. , 2019 ; Dai et al. , 2019b ; Brown et al. , 2020 ) , and time series forecasting ( Oreshkin et al. , 2020 ) . While their performance in various applications has matched and often exceeded human capabilities , neural networks may remain difficult to apply in real-world scenarios . Deep neural networks leverage the power of Graphical Processing Units ( GPUs ) , which are power-hungry . Using GPUs to make billions of predictions per day , thus comes with a substantial energy cost . In addition , despite their quite fast response time , deep neural networks are not yet suitable for most real-time applications where memory-limited low-cost architectures need to be used . For all those reasons , compression and efficiency have become a topic of high interest in the deep learning community . Sparsity in DNNs has been an active research topic generating numerous approaches . DNNs achieving the state-of-the-art in a given problem usually have a large number of layers with non-uniform parameter distribution across layers . Most sparsification methods are based on a global approach , which may result in a sub-optimal compression for a reduced accuracy . This may occur because layers with a smaller number of parameters may remain dense , although they may contribute more in terms of computational complexity ( e.g. , for convolutional layers ) . Some methods , also known as magnitude pruning , use a hard or soft-thresholding to remove less significant parameters . Soft thresholding techniques achieve a good sparsity-accuracy trade-off at the cost of additional parameters and increased computation time during training . Searching for a hardware efficient network is another area that has been proven quite useful , but it requires a huge amount of computational resources . Convex optimization techniques such as those used in ( Aghasi et al. , 2017 ) often rely upon fixed point iterations that make use of the proximity operator ( Moreau , 1962 ) . The related concepts are fundamental for tackling nonlinear problems and have recently come into play in the analysis of neural networks ( Combettes & Pesquet , 2020a ) and nonlinear systems ( Combettes & Woodstock , 2020 ) . This paper shows that the properties of nonlinear activation functions can be utilized to identify highly sparse subnetworks . We show that the sparsification of a network can be formulated as an approximate subdifferential inclusion problem . We provide an iterative algorithm called subdifferential inclusion for sparsity ( SIS ) that uses partial training data to identify a sparse subnetwork while maintaining good accuracy . SIS makes even small-parameter layers sparse , resulting in models with significantly lower inference FLOPs than the baselines . For example , SIS for 90 % sparse MobileNetV3 on ImageNet-1K achieves 66.07 % top-1 accuracy with 33 % fewer inference FLOPs than its dense counterpart and thus provides better results than the state-of-the-art method RigL . For non-convolutional networks like Transformer-XL trained on WikiText-103 , SIS is able to achieve 70 % sparsity while maintaining 21.1 perplexity score . We evaluate our approach across four domains and show that our compressed networks can achieve competitive accuracy for potential use on commodity hardware and edge devices . 2 RELATED WORK . 2.1 INDUCING SPARSITY POST TRAINING . Methods inducing sparsity after a dense network is trained involve several pruning and fine-tuning cycles till desired sparsity and accuracy are reached ( Mozer & Smolensky , 1989 ; LeCun et al. , 1990 ; Hassibi et al. , 1993 ; Han et al. , 2015 ; Molchanov et al. , 2017 ; Guo et al. , 2016 ; Park et al. , 2020 ) . ( Renda et al. , 2020 ) proposed weight rewinding technique instead of vanilla fine-tuning post-pruning . Net-Trim algorithm ( Aghasi et al. , 2017 ) removes connections at each layer of a trained network by convex programming . The proposed method works for networks using rectified linear units ( ReLUs ) . Lowering rank of parameter tensors ( Jaderberg et al. , 2014 ; vahid et al. , 2020 ; Lu et al. , 2016 ) , removing channels , filters and inducing group sparsity ( Wen et al. , 2016 ; Li et al. , 2017 ; Luo et al. , 2017 ; Gordon et al. , 2018 ; Yu et al. , 2019 ; Liebenwein et al. , 2020 ) are some methods that take network structure into account . All these methods rely on pruning and fine-tuning cycle ( s ) often from full training data . 2.2 INDUCING SPARSITY DURING TRAINING . Another popular approach has been to induce sparsity during training . This can be achieved by modifying the loss function to consider sparsity as part of the optimization ( Chauvin , 1989 ; CarreiraPerpiñán & Idelbayev , 2018 ; Ullrich et al. , 2017 ; Neklyudov et al. , 2017 ) . Dynamically pruning during training ( Zhu & Gupta , 2018 ; Bellec et al. , 2018 ; Mocanu et al. , 2018 ; Dai et al. , 2019a ; Lin et al. , 2020b ) by observing network flow . ( Mostafa & Wang , 2019 ; Dettmers & Zettlemoyer , 2020 ; Evci et al. , 2020 ) computes weight magnitude and reallocates weights at every step . Bayesian priors ( Louizos et al. , 2017 ) , L0 , L1 regularization ( Louizos et al. , 2018 ) , and variational dropout ( Molchanov et al. , 2017 ) get accuracy comparable to ( Zhu & Gupta , 2018 ) but at a cost of 2×memory and 4× computations during training . ( Liu et al. , 2019 ; Savarese et al. , 2020 ; Kusupati et al. , 2020 ; Lee , 2019 ; Xiao et al. , 2019 ; Azarian et al. , 2020 ) have proposed learnable sparsity methods through training of the sparse masks and weights simultaneously with minimal heuristics . Although these methods are cheaper than pruning after training , they need at least the same computational effort as training a dense network to find a sparse sub-network . This makes them expensive when compressing big networks where the number of parameters ranges from hundreds of millions to billions ( Dai et al. , 2019b ; Li et al. , 2019 ; Brown et al. , 2020 ) . 2.3 TRAINING SPARSELY INITIALIZED NETWORKS . ( Frankle & Carbin , 2019 ) showed that it is possible to find sparse sub-networks that , when trained from scratch , were able to match or even outperform their dense counterparts . ( Lee et al. , 2019 ) presented SNIP , a method to estimate , at initialization , the importance that each weight could have later during training . In ( Lee et al. , 2020 ) the authors perform a theoretical study of pruning at initialization from a signal propagation perspective , focusing on the initialization scheme . Recently , ( Wang et al. , 2020 ) proposed GraSP , a different method based on the gradient norm after pruning , and showed a significant improvement for moderate levels of sparsity . ( Ye et al. , 2020 ) starts with a small subnetwork and progressively grow it to a subnetwork that is as accurate as its dense counterpart . ( Tanaka et al. , 2020 ) proposes SynFlow that avoids flow collapse of a pruned network during training . ( Jorge et al. , 2020 ) proposed FORCE , an iterative pruning method that progressively removes a small number of weights . This method is able to achieve extreme sparsity at little accuracy expense . These methods are not usable for big pre-trained networks and are expensive as multiple training rounds are required for different sparse models depending on deployment scenarios ( computing devices ) . 2.4 EFFICIENT NEURAL ARCHITECTURE SEARCH . Hardware-aware NAS methods ( Zoph et al. , 2018 ; Real et al. , 2019 ; Cai et al. , 2018 ; Wu et al. , 2019 ; Tan et al. , 2019 ; Cai et al. , 2019 ; Howard et al. , 2019 ) directly incorporate the hardware feedback into efficient neural architecture search . ( Cai et al. , 2020 ) proposes to learn a single network composing of a large number of subnetworks from which a hardware aware subnetwork can be extracted in linear time . ( Lin et al. , 2020a ) proposes a similar approach wherein they identify subnetworks that can be run efficiently on microcontrollers ( MCUs ) . Our proposed algorithm applies to possibly large pre-trained networks . In contrast with methods presented in Section 2.1 , ours can use a small amount of training data during pruning and fewer epochs during fine-tuning . As we will see in the next section , a key feature of our approach is that it is based on a fine analysis of the mathematical properties of activation functions , so allowing the use of powerful convex optimization tools that offer sound convergence guarantees . 3 PROPOSED METHOD . 3.1 VARIATIONAL PRINCIPLES . A basic neural network layer can be described by the relation : y = R ( Wx+ b ) ( 1 ) where x ∈ RM is the input , y ∈ RN the output , W ∈ RN×M is the weight matrix , b ∈ RN the bias vector , and R is a nonlinear activation operator from RN to RN . A key observation is that most of the activation operators currently used in neural networks are proximity operators of convex functions ( Combettes & Pesquet , 2020a ; b ) . We will therefore assume that there exists a proper lower-semicontinuous convex function f from RN to R ∪ { +∞ } such that R = proxf . We recall that f is a proper lower-semicontinuous convex function if the area overs its graph , its epigraph { ( y , ξ ) ∈ RN × R ∣∣ f ( y ) 6 ξ } , is a nonempty closed convex set . For such a function the proximity operator of f at z ∈ RN ( Moreau , 1962 ) is the unique point defined as proxf ( z ) = argmin p∈RN 1 2 ‖z − p‖2 + f ( p ) . ( 2 ) It follows from standard subdifferential calculus that Eq . ( 1 ) can be re-expressed as the following inclusion relation : Wx+ b− y ∈ ∂f ( y ) , ( 3 ) where ∂f ( y ) is the Moreau subdifferential of f at y defined as ∂f ( y ) = { t ∈ RN ∣∣ ( ∀z ∈ RN ) f ( z ) > f ( y ) + 〈t | z − y〉 } . ( 4 ) The subdifferential constitutes a useful extension of the notion of differential , which is applicable to nonsmooth functions . The set ∂f ( y ) is closed and convex and , if y satisfies Eq . ( 1 ) , it is nonempty . The distance to this set of a point z ∈ RN is given by d∂f ( y ) ( z ) = inf t∈∂f ( y ) ‖z − t‖ . ( 5 ) We thus see that the subdifferential inclusion in Eq . ( 3 ) is also equivalent to d∂f ( y ) ( Wx+ b− y ) = 0 . ( 6 ) Therefore , a suitable accuracy measure for approximated values of the layer parameters ( W , b ) is d∂f ( y ) ( Wx+ b− y ) .

Summary: In this paper the authors propose a new model compression method based on subdifferential inclusion. The key idea is to make the outputs of the neurons in the sparse and dense networks at the same input close enough. They rewrite the activation function as the proximity operator of a proper convex function and finally formulate the compression problem into a constraint minimization problem using the technique of subdifferential inclusion. They conduct a series of experiments to evaluate the performance of their proposed methods.