iSparse: Output Informed Sparsification of Neural Networks

1 INTRODUCTION . Deep neural networks ( DNNs ) , particularly convolutional neural networks ( CNN ) , have shown impressive success in many applications , such as facial recognition ( Lawrence et al. , 1997 ) , time series analysis ( Yang et al. , 2015 ) , speech recognition ( Hinton et al. , 2012 ) , object classification ( Liang & Hu , 2015 ) , and video surveillance ( Karpathy & et . at. , 2014 ) . As the term “ deep ” neural networks implies , this success often relies on large networks , with large number of trainable edges ( weights ) ( Huang et al. , 2017 ; Zoph et al. , 2018 ; He et al. , 2016 ; Simonyan & Zisserman , 2015 ) . While a large number of trainable edges help generalize the network for complex and diverse patterns in large-scale datasets , this often comes with enormous computation cost to account for the non-linearity of the deep networks ( ReLU , sigmoid , tanh ) . In fact , DNNs owe their recent success to hardware level innovations that render the immense computational requirements practical ( Ovtcharov & et . al. , 2015 ; Matthieu Courbariaux et al. , 2015 ) . However , the benefits of hardware solutions and optimizations that can be applied to a general purpose DNN or CNN are limited and these solutions are fast reaching their limits . This has lead to significant interest in networkspecific optimization techniques , such as network compression ( Choi & et . al. , 2018 ) , pruning ( Li et al. , 2016 ; Yu et al. , 2018 ) , and regularization ( Srivastava & et . al. , 2014 ; Wan et al. , 2013 ) , aim to reduce the number of edges in the network . However , many of these techniques require retraining the pruned network , leading to the significant amount of computational waste . 1.1 NETWORK SPARSIFICATION . Many successful networks nevertheless often contain large numbers of redundant edges . Consider for example , the weights of sample network shown in Figure 1a . As we see here , the weight distribution is centered around zero and has significant number of weights with insignificant contribution to the network output . Such edges may add noise or non-informative information leading to reduction in the network performance . ( Denil et al. , 2013 ; Ashouri et al. , 2018 ; Yu et al. , 2018 ) has shown that it is possible to predict 95 % network parameters while only learning 5 % parameters . Sparsification techniques can generally be classified into neuron/kernel sparcification ( Li et al. , 2016 ; Yu et al. , 2018 ) and edge/weight sparcification techniques ( Wan et al. , 2013 ; Ashouri et al. , 2018 ) : ( Li et al. , 2016 ) proposed to eliminate neurons that have low l2-norms of their weights , whereas ( Yu et al. , 2018 ) proposed a neuron importance score propagation ( NISP ) technique where neuron importance scores ( using Roffo & et . al . ( 2015 ) - See Equation 5 ) are propagated from the output layer to the input layer in a back-propagation fashion . Drop-out ( Srivastava & et . al. , 2014 ) technique instead deactivates neuron activations at random . As an edge sparsification technique , DropConnect ( Wan et al. , 2013 ) selects edges to be sparsified randomly . ( Ashouri et al. , 2018 ) showed that the network performance can be maintained by eliminating insignificant weights without modifying the network architecture . 1.2 OUR CONTRIBUTIONS : OUTPUT INFORMED EDGE SPARSIFICATION . Following these works , we argue network sparsification can be a very effective tool for reducing sizes and complexities of DNNs and CNNs , without any significant loss in accuracy . However , we also argue that edge weights can not be used “ as is ” for pruning the network . Instead , one needs to consider the significance of each edge within the context of their place in the network ( Figure 2 ) : “ Two edges in a network with the same edge weight may have different degrees of contributions to the final network output ” and in this paper , we show that it is possible to quantify significance of each edge in the network , relative to their contributions to the final network output and use these measures significance to minimize the redundancy in the network by sparsifying the weights that contributes insignificantly to network . We , therefore , propose a novel iSparse framework , and experimentally show , that we can sparsify the network , by almost 50 % , without impacting the network performance . The key contributions of our proposed work are as follows : • Output-informed quantification of the significance of network parameters : Informed by the final layer network output , iSparse computes and propagates edge significant scores that measure the importance of each edge with respect to the model output ( Section 3 ) . • Retraining-free network sparsification ( Sparsify-with ) : The proposed iSparse framework is robust to edge sparsification and can maintain network performance without having to retraining the network . This implies that one can apply iSparse on pre-trained networks , on-the-fly , to achieve the desired level of sparsification ( Section 3.3 ) • Sparsification during training ( Train-with ) : iSparse can also be used as a regularizer during the model training allowing for learning of sparse networks from scratch ( Section 4 ) . As the sample results in Figure 1b shows , iSparse is able to achieve 30-50 % sparsification with minimal impact on model accuracy . More detailed experimental comparisons ( See Section 5 ) of iSparse against PFEC , NISP , Retraining-Free and DropConnect on benchmark datasets illustrated that iSparse leads to more effective network sparsifications . 2 RELATED WORKS . A neural network is a sequence of layers of neurons to help learn ( and remember ) complex nonlinear patterns in a given dataset ( Grossberg , 1988 ) . Recently , deep neural networks ( DNNs ) , and particularly CNNs , which leverage recent hardware advances to increase the number of layers in the network to scales that were not practical until recently , ( Lawrence et al. , 1997 ; Yang et al. , 2015 ; Hinton et al. , 2012 ; Liang & Hu , 2015 ; Karpathy & et . at. , 2014 ) have shown impressive success in several data analysis and machine learning applications . A typical CNN consists of ( 1 ) feature extraction layers are responsible for learning complex patterns in the data and remember them through layer weights . An m× n layer , Ll , maps an n-dimensional input ( Yl−1 ) to an m-dimensional output ( Yl ) , by training for a weight matrix Wl Rml×nl ( see Section 3.1 for further details ) ; ( 2 ) activation layers , which help capture non-linear patterns in the data through activation functions ( σ ) which maps the output from a feature extraction layer to a nonlinear space ( ReLU and softmax are commonly used activation functions ) ; and ( 3 ) pooling layers , ( sampling ) which up- or down-sample the intermediate data in the network . The training process of a neural network often comprises of two key stages : ( 1 ) forward-propagation ( upstream ) maps the input data , X , to an output variable , Ŷ . At each layer , we have Ŷl = L ( Yl ) = σ ( WlYl+Bl ) , where Y1 = X . Intuitively , each layer learns to extract new features from the features extracted by the prior layer . ( 2 ) backward-propagation ( downstream ) revises network weights , W , based on the training error , Err = |Y − Ŷ | , where Y and Ŷ are the ground truth and predictions of the model , respectively , such that update weights are defined as W ′ = W − Err . The number of trainable parameters in a deep network can range from as low as tens of thousands ( LeCun et al. , 1999 ) to hundreds of millions ( Simonyan & Zisserman , 2015 ) ( Table 1 in Section 5 ) . The three order increase in the number trainable parameter may lead to parameters being redundant or may have negligible contribution to the overall network output . This redundancy and insignificance of the network parameters has led to advancements in network regularization , by introducing dynamic or informed sparsification in the network . These advancements can be broadly classified into two main categories : parameter pruning and parameter regularization . In particular , pruning focuses on compressing the network by eliminating the redundant or insignificant parameters . ( Han et al. , 2015 ; Han & et . al. , 2016 ) aims to prune the parameters with near-zero weights inspired from l1 and l2 regularization ( Tibshirani , 1996 ; Tikhonov , 1963 ) . ( Li et al. , 2016 ) choose to filter out convolutional kernel with minimum weight values in given layer . Recently , ( Yu et al. , 2018 ) minimizes the change in final network performance by eliminating the neuron that have minimal impact on the network output by leveraging neuron importance score ( NL ) ( See Section 5.3 ) - computed using Inf-FS ( Roffo & et . al. , 2015 ) . More complex approaches have been proposed to tackle the problem of redundancy in the network through weight quantization . ( Rastegari & et . al. , 2016 ) propose to the quantize the inputs and output activations of the layers in a CNN by using step function and also leveraging the binary operation by using the binary weights opposed to the real-values weights . ( Chen & et . al. , 2015 ) focuses on low-level mobile hardware with limited computational power , and proposed to leverage the inherent redundancy in the network for using hashing functions to compress the weights in the network . ( Bahdanau et al. , 2014 ; Woo et al. , 2018 ) showed that the each input feature to a given layer in the network rarely have the same importance , therefore , learning there individual importance ( attention ) helps improve the performance of the network . More recently , ( Garg & Candan , 2019 ) has shown that input data informed deep networks can provide high-performance network configurations . In this paper , we rely on output information for identifying and eliminating insignificant parameters from the network , without having to update the edge weights or retraining the network . 3 ISPARSE : OUTPUT INFORMED SPARSIFICATION OF NEURAL NETWORKS . As discussed in Section 1 , in order to tackle complex inputs , deep neural networks have gone increasingly deeper and wider . This design strategy , however , often results in large numbers of insignificant edges1 ( weights ) , if not redundant . In this section , we describe the proposed iSparse , framework which quantifies the significance of each individual connection in the network with respect to the overall network output to determine the set of edges that can be sparsified to alleviate network redundancy and eliminate insignificant edges . iSparse aims to determine the significance of the edges in the network to make informed sparsification of the network . 3.1 MASK MATRIX . A typical neural network , N , can be viewed as a sequential arrangement of convolutional ( C ) and fully connected ( F ) layers : N ( X ) = LL ( LL−1 . . . ( L2 ( L1 ( X ) ) ) ) , here , X is the input , L is the total number of layers in the network and L { C , F } , s.t. , any given layer , Ll | 1 ≤ l L , can be generalized as Ll ( Yl ) = σl ( WlYl +Bl ) = Ŷl , ( 1 ) where , Yl is the input to the layer ( s.t . Yl = Ŷl−1 and for l = 1 , Y1 = X ) and σl , Wl , and Bl are the activation function , weight , and bias respectively . Note that , if the lth layer has ml neurons and the ( l − 1 ) th layer has nl neurons , then Ŷl Rml×1 , Yl Rnl×1 , Wl Rml×nl and B Rml×1 . Given this formulation , the problem of identifying insignificant edges can be formulated as the problem of generating a sequence of binary mask matrices , M1 , . . . , ML , that collectively represents whether any given edge in the network is sparsified ( 0 ) or not ( 1 ) : Ml = Bnl×ml , B { 0 , 1 } , 1 ≤ l ≤ L ( 2 )

Summary: This work proposes iSparse framework, which aims to sparsify a neural network by removing redundant edges. Unlike previous works where the edges were removed based on their weight value, or based on the relationship between input and output neurons, iSparse selects the edges to remove by computing the contribution of each edge with respect to the final outcome. The experiments show that, compared to several baseline methods, iSparse perform favorably on multiple datasets.