Mean Replacement Pruning  

Pruning units in a deep network can help speed up inference and training as well as reduce the size of the model . We show that bias propagation is a pruning technique which consistently outperforms the common approach of merely removing units , regardless of the architecture and the dataset . We also show how a simple adaptation of an existing scoring function allows us to select the best units to prune . Finally , we show that the units selected by the best performing scoring functions are somewhat consistent over the course of training , implying the dead parts of the network appear during the early stages of training . 1 INTRODUCTION . Pruning is a successful method for reducing the size of a trained neural network and accelerating inference . Pruning consists of deleting the parts of the network whose removal least affects the network performance . Many pruning methods proposed in the literature differ in computational cost and in effectiveness in ways that are hard to assess . In an interesting recent work , Frankle & Carbin ( 2018 ) argue for the so called “ winning ticket ” hypothesis . More precisely , they train a large network after saving the random initial value of each parameter . After training , they prune the large network to produce a smaller network with one fifth of the weights . Setting its weights to their saved initial values and retraining achieves a performance close to that of the large trained network with a much reduced computational cost . This result opens up a new frontier for pruning methods , where they are used to detect useless units early in the training and therefore accelerating the inference . This contribution studies the effect of pruning methods throughout the training process . We also present mean replacement , a unit pruning method that extends the idea of bias propagation introduced in ( Ye et al. , 2018 ) to the non-constrained training setting . The main observations of our work can then be summarized as follows : • Regardless of the scoring function used , bias propagation reduces the pruning penalty for networks without batch normalization . • Fine-tuning the pruned network with additional training iterations reduces the bias propagation advantage but not very quickly . • Absolute valued approximation of the pruning penalty provides superior performance over the normal first order approximation . This finding confirms the observations made by Molchanov et al . ( 2016 ) . • Units that are selected by the best performing scoring function seem to come from a small subset of units . This finding confirms Frankle & Carbin ( 2018 ) ’ s comments on the lottery ticket and Evci ( 2018 ) ’ s claims about dead units . The rest of the paper is organized as follows . After reviewing the related work in Section 2. we define our pruning methods and scoring functions in Section 3 . Section 4 provides an empirical evaluation comparing various combinations of scoring functions and methods under varying pruning fractions , datasets , and models . We briefly provide some concluding remarks and discuss future work in Section 5 . 2 RELATED WORK . One simple and common technique to select what parts of a network to prune is to select small magnitude parameters , similar to the technique of weight decay ( Hanson & Pratt , 1989 ) . LeCun et al . ( 1990 ) and Hassibi & Stork ( 1993 ) proposed using second-order saliency measures to prune trained networks with zero gradient . With the era of deep networks , redundancy in trained networks became even more obvious and various works tackled this problem aiming to reduce the size of the network . Han et al . ( 2015 ) applied magnitude based parameter pruning on deep networks and reported around 30x compression combining various methods like weight quantization and Huffman coding . Zhu & Gupta ( 2017 ) perform pruning during training and report similar compression rates . Achterhold et al . ( 2018 ) focus on pruning Bayesian neural networks . Evci ( 2018 ) claims that the removed parameters tend to gather around specific units , so directly pruning full units might prove to be an efficient strategy . Further , pruning units also comes with direct gains in terms of storage and speed since it meshes well with dense representations ( Luo et al. , 2017 ; Wen et al. , 2016 ) . Wen et al . ( 2016 ) prune entire channels using the group lasso penalty . Hu et al . ( 2016 ) observe high percentage of zero activations in deep networks with ReLU units and propose Average Percentage of Zeros as a new saliency function . The work of Molchanov et al . ( 2016 ) focuses on iterative pruning with single unit removals in the context of transfer learning . They compare various scoring functions and propose the absolute-valued Taylor approximation as the best performing one . However , one practical drawback of their investigation is that they prune one unit at a time . Once units have been selected for removal , there is still the question of how to minimize the impact of the deletion . Most works , ( Molchanov et al. , 2016 ) for instance , focus on mere removal of the units followed by retraining the network . While retraining the full network after pruning can greatly minimize the loss in accuracy induced by the deletion , it can be computationally expensive . Rather than recovering from the damage post-pruning , another line of research focuses on preemptively mitigating the effects . Ye et al . ( 2018 ) propose penalizing the variance of the activations for networks using batch normalization . They then propose replacing units with low variance with constant values using bias propagation . Our work extends this idea of bias propagation to various other pruning methods . Morcos et al . ( 2018 ) suggest ablating units ( which mimics removal ) by replacing them with their mean activation ; however , the authors report this yields inferior performance compared to simply removing the units . Luo et al . ( 2017 ) propose using the l2-norm of unit activations to iteratively prune convolutional layers for VGG-16 and ResNet . They also propose performing updates on the outgoing weights that minimize the reconstruction loss on the next layer . Their method relies on a matrix inversion rendering it impractical for large networks . 3 UNIT PRUNING AND MEAN REPLACEMENT IDEA . Even with a careful unit selection , pruning can significantly damage the performance of the network . Fine-tuning the damaged network with retraining iterations may or may not recover the full performance . Thus our goal is to minimize the damage as much as possible at pruning time . In this section we introduce mean replacement , a simple pruning method that significantly reduces the loss incurred by the ablation . 3.1 PRUNING UNITS AND NOTATION . For a given dataset D with samples ( x ( i ) , y ( i ) ) , output f ( x ( i ) ; w ) using parameters w , and loss function l ( f ( x ( i ) ; w ) , y ( i ) ; w ) , the loss of the optimization problem is L ( w ) = ∑ i∈D l ( f ( x ( i ) ; w ) , y ( i ) ) . Pruning a network is often defined as setting some of its parameters to 0 . Given w , our goal then consists of finding the mask m ∈ { 0 , 1 } d that minimizes L ( w m ) = ∑ i∈D l ( f ( x ( i ) ; w m ) , y ( i ) ) . The mask m must respect some constraints . First , even though the mask is defined at the parameter level , i.e . it has as many components as the number of parameters in the model , we are pruning units . Hence , all parameters corresponding to the same unit must have the same value in the mask . Second , we are interested in pruning only a limited number of units , so the number of elements set to 0 in m is constrained . Finally , we might also want to enforce the number of units pruned at each layer , or simply prevent the pruning at some layers . Denoting as M the set of all masks satisfying these constraints , the optimal pruning is given by m∗ = arg min m∈M ∑ i∈D l ( f ( x ( i ) ; w m ) , y ( i ) ) . ( 1 ) For the remainder of the paper , we shall assume that the number of units to remove is set for each layer independently . This is without loss of generality and will allow us to focus on a single layer , greatly simplifying the presentation . The complexity of solving Eq . 1 increases exponentially with the number of units to prune . Therefore , in practice , people rank all units using a per-unit scoring function s ( w ; D ) . Several examples of such scoring functions will be discussed in Section 3.4 . Units with the lowest score are then pruned . This approach implicitly assumes that the scores of individual units are independent of each other . In other words , pruning one unit is assumed to not affect the score of any other unit . Thus , a good scoring function needs to have a small inter-unit correlation . Once a scoring function s ( w ; D ) has been chosen , we can define m through its elements mi : mi = { 0 if i belongs to a unit u where u ∈ B ( s ( w ; D ) , k ) 1 otherwise . ( 2 ) where B ( x , k ) is the set of k elements of x with the lowest value . In other words , we will set to 0 all the parameters belonging to units whose score is one of the k lowest , k being the number of units we wish to remove in that layer . Pruning a fraction of the units in a particular layer can have a big impact on the network and induce a large loss penalty , that we call the pruning penalty : PruningPenalty = L ( w m ) − L ( w ) ( 3 ) Retraining the network might reduce the pruning penalty at the cost of additional computation . We shall now see how adjusting the biases of the following layer can reduce the pruning penalty with low computational overhead . In order to show this , we need to depart from our earlier definition of pruning as consisting of zeroeing a subset of the weights . 3.2 MEAN REPLACEMENT . We intend to remove k units from a layer of the network in a manner that has a reasonable impact on the network performance . This is often done by replacing these units with zeroes . However , zero is an arbitrary choice and any constant would work . This constant would be “ propagated ” by multiplying it with the outgoing weights of the layer above , which is equivalent to updating the bias of that layer with the resulting sum . Mean replacement consists of replacing the output of pruned units by a constant that is equal to the mean of the unit outputs collected on the training samples before pruning . A theoretical justification for that choice will be presented in Section 3.3 . We will first focus on the removal of a single unit . In a fully connected network , each unit is associated with a single activation . However , in a convolutional layer , a unit is associated with a set of outputs , one per location . In that case , each of these output will be replaced with the same constant . Let a ( x , p ) represent the unit output for training example x at location p ∈ P . Let us randomly choose a subset1 Ds ⊂ D of examples from the training set . We first compute the mean unit output ā = 1 |Ds| ∑ x∈Ds 1 |P| ∑ p∈P a ( x , p ) . 1Usually smaller than the full training set , but big enough to get a good approximation . Mean replacement consists in replacing the pruned unit by the constant ā . This can be implemented by removing the pruned unit in the normal way —which amounts to replace its output by a zero— and folding the constant ā into the bias parameter of the downstream units . b← b + āw , ( 4 ) where b represents the vector of the biases of the downstream units and w represents the outgoing weights of the pruned unit , that is the weights that were connecting the pruned unit to each of its downstream units before the pruning operation . This process is illustrated in Figure 1 . We now justify our choice of constant by showing that , in the context of a quadratic loss , mean replacement is the optimal strategy .

Summary: This paper presents a mean-replacement pruning strategy and utilizes the absolute-valued Taylor expansion as the scoring function for the pruning. Some computer vision problems are used as test beds to empirically show the effect of the employment of bias-propagation and different scoring functions. The empirical results validates the effectiveness of bias-propagation and absolute-valued Taylor expansion scoring functions.