Sparsifying Networks via Subdifferential Inclusion

1 INTRODUCTION . Deep neural networks have evolved to the state-of-the-art techniques in a wide array of applications : computer vision ( Simonyan & Zisserman , 2015 ; He et al. , 2016 ; Huang et al. , 2017 ) , automatic speech recognition ( Hannun et al. , 2014 ; Dong et al. , 2018 ; Li et al. , 2019 ; Watanabe et al. , 2018 ; Hayashi et al. , 2019 ; Inaguma et al. , 2020 ) , natural language processing ( Turc et al. , 2019 ; Radford et al. , 2019 ; Dai et al. , 2019b ; Brown et al. , 2020 ) , and time series forecasting ( Oreshkin et al. , 2020 ) . While their performance in various applications has matched and often exceeded human capabilities , neural networks may remain difficult to apply in real-world scenarios . Deep neural networks leverage the power of Graphical Processing Units ( GPUs ) , which are power-hungry . Using GPUs to make billions of predictions per day , thus comes with a substantial energy cost . In addition , despite their quite fast response time , deep neural networks are not yet suitable for most real-time applications where memory-limited low-cost architectures need to be used . For all those reasons , compression and efficiency have become a topic of high interest in the deep learning community . Sparsity in DNNs has been an active research topic generating numerous approaches . DNNs achieving the state-of-the-art in a given problem usually have a large number of layers with non-uniform parameter distribution across layers . Most sparsification methods are based on a global approach , which may result in a sub-optimal compression for a reduced accuracy . This may occur because layers with a smaller number of parameters may remain dense , although they may contribute more in terms of computational complexity ( e.g. , for convolutional layers ) . Some methods , also known as magnitude pruning , use a hard or soft-thresholding to remove less significant parameters . Soft thresholding techniques achieve a good sparsity-accuracy trade-off at the cost of additional parameters and increased computation time during training . Searching for a hardware efficient network is another area that has been proven quite useful , but it requires a huge amount of computational resources . Convex optimization techniques such as those used in ( Aghasi et al. , 2017 ) often rely upon fixed point iterations that make use of the proximity operator ( Moreau , 1962 ) . The related concepts are fundamental for tackling nonlinear problems and have recently come into play in the analysis of neural networks ( Combettes & Pesquet , 2020a ) and nonlinear systems ( Combettes & Woodstock , 2020 ) . This paper shows that the properties of nonlinear activation functions can be utilized to identify highly sparse subnetworks . We show that the sparsification of a network can be formulated as an approximate subdifferential inclusion problem . We provide an iterative algorithm called subdifferential inclusion for sparsity ( SIS ) that uses partial training data to identify a sparse subnetwork while maintaining good accuracy . SIS makes even small-parameter layers sparse , resulting in models with significantly lower inference FLOPs than the baselines . For example , SIS for 90 % sparse MobileNetV3 on ImageNet-1K achieves 66.07 % top-1 accuracy with 33 % fewer inference FLOPs than its dense counterpart and thus provides better results than the state-of-the-art method RigL . For non-convolutional networks like Transformer-XL trained on WikiText-103 , SIS is able to achieve 70 % sparsity while maintaining 21.1 perplexity score . We evaluate our approach across four domains and show that our compressed networks can achieve competitive accuracy for potential use on commodity hardware and edge devices . 2 RELATED WORK . 2.1 INDUCING SPARSITY POST TRAINING . Methods inducing sparsity after a dense network is trained involve several pruning and fine-tuning cycles till desired sparsity and accuracy are reached ( Mozer & Smolensky , 1989 ; LeCun et al. , 1990 ; Hassibi et al. , 1993 ; Han et al. , 2015 ; Molchanov et al. , 2017 ; Guo et al. , 2016 ; Park et al. , 2020 ) . ( Renda et al. , 2020 ) proposed weight rewinding technique instead of vanilla fine-tuning post-pruning . Net-Trim algorithm ( Aghasi et al. , 2017 ) removes connections at each layer of a trained network by convex programming . The proposed method works for networks using rectified linear units ( ReLUs ) . Lowering rank of parameter tensors ( Jaderberg et al. , 2014 ; vahid et al. , 2020 ; Lu et al. , 2016 ) , removing channels , filters and inducing group sparsity ( Wen et al. , 2016 ; Li et al. , 2017 ; Luo et al. , 2017 ; Gordon et al. , 2018 ; Yu et al. , 2019 ; Liebenwein et al. , 2020 ) are some methods that take network structure into account . All these methods rely on pruning and fine-tuning cycle ( s ) often from full training data . 2.2 INDUCING SPARSITY DURING TRAINING . Another popular approach has been to induce sparsity during training . This can be achieved by modifying the loss function to consider sparsity as part of the optimization ( Chauvin , 1989 ; CarreiraPerpiñán & Idelbayev , 2018 ; Ullrich et al. , 2017 ; Neklyudov et al. , 2017 ) . Dynamically pruning during training ( Zhu & Gupta , 2018 ; Bellec et al. , 2018 ; Mocanu et al. , 2018 ; Dai et al. , 2019a ; Lin et al. , 2020b ) by observing network flow . ( Mostafa & Wang , 2019 ; Dettmers & Zettlemoyer , 2020 ; Evci et al. , 2020 ) computes weight magnitude and reallocates weights at every step . Bayesian priors ( Louizos et al. , 2017 ) , L0 , L1 regularization ( Louizos et al. , 2018 ) , and variational dropout ( Molchanov et al. , 2017 ) get accuracy comparable to ( Zhu & Gupta , 2018 ) but at a cost of 2×memory and 4× computations during training . ( Liu et al. , 2019 ; Savarese et al. , 2020 ; Kusupati et al. , 2020 ; Lee , 2019 ; Xiao et al. , 2019 ; Azarian et al. , 2020 ) have proposed learnable sparsity methods through training of the sparse masks and weights simultaneously with minimal heuristics . Although these methods are cheaper than pruning after training , they need at least the same computational effort as training a dense network to find a sparse sub-network . This makes them expensive when compressing big networks where the number of parameters ranges from hundreds of millions to billions ( Dai et al. , 2019b ; Li et al. , 2019 ; Brown et al. , 2020 ) . 2.3 TRAINING SPARSELY INITIALIZED NETWORKS . ( Frankle & Carbin , 2019 ) showed that it is possible to find sparse sub-networks that , when trained from scratch , were able to match or even outperform their dense counterparts . ( Lee et al. , 2019 ) presented SNIP , a method to estimate , at initialization , the importance that each weight could have later during training . In ( Lee et al. , 2020 ) the authors perform a theoretical study of pruning at initialization from a signal propagation perspective , focusing on the initialization scheme . Recently , ( Wang et al. , 2020 ) proposed GraSP , a different method based on the gradient norm after pruning , and showed a significant improvement for moderate levels of sparsity . ( Ye et al. , 2020 ) starts with a small subnetwork and progressively grow it to a subnetwork that is as accurate as its dense counterpart . ( Tanaka et al. , 2020 ) proposes SynFlow that avoids flow collapse of a pruned network during training . ( Jorge et al. , 2020 ) proposed FORCE , an iterative pruning method that progressively removes a small number of weights . This method is able to achieve extreme sparsity at little accuracy expense . These methods are not usable for big pre-trained networks and are expensive as multiple training rounds are required for different sparse models depending on deployment scenarios ( computing devices ) . 2.4 EFFICIENT NEURAL ARCHITECTURE SEARCH . Hardware-aware NAS methods ( Zoph et al. , 2018 ; Real et al. , 2019 ; Cai et al. , 2018 ; Wu et al. , 2019 ; Tan et al. , 2019 ; Cai et al. , 2019 ; Howard et al. , 2019 ) directly incorporate the hardware feedback into efficient neural architecture search . ( Cai et al. , 2020 ) proposes to learn a single network composing of a large number of subnetworks from which a hardware aware subnetwork can be extracted in linear time . ( Lin et al. , 2020a ) proposes a similar approach wherein they identify subnetworks that can be run efficiently on microcontrollers ( MCUs ) . Our proposed algorithm applies to possibly large pre-trained networks . In contrast with methods presented in Section 2.1 , ours can use a small amount of training data during pruning and fewer epochs during fine-tuning . As we will see in the next section , a key feature of our approach is that it is based on a fine analysis of the mathematical properties of activation functions , so allowing the use of powerful convex optimization tools that offer sound convergence guarantees . 3 PROPOSED METHOD . 3.1 VARIATIONAL PRINCIPLES . A basic neural network layer can be described by the relation : y = R ( Wx+ b ) ( 1 ) where x ∈ RM is the input , y ∈ RN the output , W ∈ RN×M is the weight matrix , b ∈ RN the bias vector , and R is a nonlinear activation operator from RN to RN . A key observation is that most of the activation operators currently used in neural networks are proximity operators of convex functions ( Combettes & Pesquet , 2020a ; b ) . We will therefore assume that there exists a proper lower-semicontinuous convex function f from RN to R ∪ { +∞ } such that R = proxf . We recall that f is a proper lower-semicontinuous convex function if the area overs its graph , its epigraph { ( y , ξ ) ∈ RN × R ∣∣ f ( y ) 6 ξ } , is a nonempty closed convex set . For such a function the proximity operator of f at z ∈ RN ( Moreau , 1962 ) is the unique point defined as proxf ( z ) = argmin p∈RN 1 2 ‖z − p‖2 + f ( p ) . ( 2 ) It follows from standard subdifferential calculus that Eq . ( 1 ) can be re-expressed as the following inclusion relation : Wx+ b− y ∈ ∂f ( y ) , ( 3 ) where ∂f ( y ) is the Moreau subdifferential of f at y defined as ∂f ( y ) = { t ∈ RN ∣∣ ( ∀z ∈ RN ) f ( z ) > f ( y ) + 〈t | z − y〉 } . ( 4 ) The subdifferential constitutes a useful extension of the notion of differential , which is applicable to nonsmooth functions . The set ∂f ( y ) is closed and convex and , if y satisfies Eq . ( 1 ) , it is nonempty . The distance to this set of a point z ∈ RN is given by d∂f ( y ) ( z ) = inf t∈∂f ( y ) ‖z − t‖ . ( 5 ) We thus see that the subdifferential inclusion in Eq . ( 3 ) is also equivalent to d∂f ( y ) ( Wx+ b− y ) = 0 . ( 6 ) Therefore , a suitable accuracy measure for approximated values of the layer parameters ( W , b ) is d∂f ( y ) ( Wx+ b− y ) .

Summary: The paper propose a network compression algorithm by exploiting a reformulation of activation function as proximity operator. The latter is an optimization problem whose optimality condition reveals constraints on the weight matrix W of the neural net. The main idea is then to "biasedly" select W as a minimizer of a sparsity inducing penalties under a relaxation of the previous optimality conditions. The authors provide details on solving such problem as well as numerical experiments that leads to similar results than competitors.