Mean Replacement Pruning  

Pruning units in a deep network can help speed up inference and training as well as reduce the size of the model . We show that bias propagation is a pruning technique which consistently outperforms the common approach of merely removing units , regardless of the architecture and the dataset . We also show how a simple adaptation of an existing scoring function allows us to select the best units to prune . Finally , we show that the units selected by the best performing scoring functions are somewhat consistent over the course of training , implying the dead parts of the network appear during the early stages of training . 1 INTRODUCTION . Pruning is a successful method for reducing the size of a trained neural network and accelerating inference . Pruning consists of deleting the parts of the network whose removal least affects the network performance . Many pruning methods proposed in the literature differ in computational cost and in effectiveness in ways that are hard to assess . In an interesting recent work , Frankle & Carbin ( 2018 ) argue for the so called “ winning ticket ” hypothesis . More precisely , they train a large network after saving the random initial value of each parameter . After training , they prune the large network to produce a smaller network with one fifth of the weights . Setting its weights to their saved initial values and retraining achieves a performance close to that of the large trained network with a much reduced computational cost . This result opens up a new frontier for pruning methods , where they are used to detect useless units early in the training and therefore accelerating the inference . This contribution studies the effect of pruning methods throughout the training process . We also present mean replacement , a unit pruning method that extends the idea of bias propagation introduced in ( Ye et al. , 2018 ) to the non-constrained training setting . The main observations of our work can then be summarized as follows : • Regardless of the scoring function used , bias propagation reduces the pruning penalty for networks without batch normalization . • Fine-tuning the pruned network with additional training iterations reduces the bias propagation advantage but not very quickly . • Absolute valued approximation of the pruning penalty provides superior performance over the normal first order approximation . This finding confirms the observations made by Molchanov et al . ( 2016 ) . • Units that are selected by the best performing scoring function seem to come from a small subset of units . This finding confirms Frankle & Carbin ( 2018 ) ’ s comments on the lottery ticket and Evci ( 2018 ) ’ s claims about dead units . The rest of the paper is organized as follows . After reviewing the related work in Section 2. we define our pruning methods and scoring functions in Section 3 . Section 4 provides an empirical evaluation comparing various combinations of scoring functions and methods under varying pruning fractions , datasets , and models . We briefly provide some concluding remarks and discuss future work in Section 5 . 2 RELATED WORK . One simple and common technique to select what parts of a network to prune is to select small magnitude parameters , similar to the technique of weight decay ( Hanson & Pratt , 1989 ) . LeCun et al . ( 1990 ) and Hassibi & Stork ( 1993 ) proposed using second-order saliency measures to prune trained networks with zero gradient . With the era of deep networks , redundancy in trained networks became even more obvious and various works tackled this problem aiming to reduce the size of the network . Han et al . ( 2015 ) applied magnitude based parameter pruning on deep networks and reported around 30x compression combining various methods like weight quantization and Huffman coding . Zhu & Gupta ( 2017 ) perform pruning during training and report similar compression rates . Achterhold et al . ( 2018 ) focus on pruning Bayesian neural networks . Evci ( 2018 ) claims that the removed parameters tend to gather around specific units , so directly pruning full units might prove to be an efficient strategy . Further , pruning units also comes with direct gains in terms of storage and speed since it meshes well with dense representations ( Luo et al. , 2017 ; Wen et al. , 2016 ) . Wen et al . ( 2016 ) prune entire channels using the group lasso penalty . Hu et al . ( 2016 ) observe high percentage of zero activations in deep networks with ReLU units and propose Average Percentage of Zeros as a new saliency function . The work of Molchanov et al . ( 2016 ) focuses on iterative pruning with single unit removals in the context of transfer learning . They compare various scoring functions and propose the absolute-valued Taylor approximation as the best performing one . However , one practical drawback of their investigation is that they prune one unit at a time . Once units have been selected for removal , there is still the question of how to minimize the impact of the deletion . Most works , ( Molchanov et al. , 2016 ) for instance , focus on mere removal of the units followed by retraining the network . While retraining the full network after pruning can greatly minimize the loss in accuracy induced by the deletion , it can be computationally expensive . Rather than recovering from the damage post-pruning , another line of research focuses on preemptively mitigating the effects . Ye et al . ( 2018 ) propose penalizing the variance of the activations for networks using batch normalization . They then propose replacing units with low variance with constant values using bias propagation . Our work extends this idea of bias propagation to various other pruning methods . Morcos et al . ( 2018 ) suggest ablating units ( which mimics removal ) by replacing them with their mean activation ; however , the authors report this yields inferior performance compared to simply removing the units . Luo et al . ( 2017 ) propose using the l2-norm of unit activations to iteratively prune convolutional layers for VGG-16 and ResNet . They also propose performing updates on the outgoing weights that minimize the reconstruction loss on the next layer . Their method relies on a matrix inversion rendering it impractical for large networks . 3 UNIT PRUNING AND MEAN REPLACEMENT IDEA . Even with a careful unit selection , pruning can significantly damage the performance of the network . Fine-tuning the damaged network with retraining iterations may or may not recover the full performance . Thus our goal is to minimize the damage as much as possible at pruning time . In this section we introduce mean replacement , a simple pruning method that significantly reduces the loss incurred by the ablation . 3.1 PRUNING UNITS AND NOTATION . For a given dataset D with samples ( x ( i ) , y ( i ) ) , output f ( x ( i ) ; w ) using parameters w , and loss function l ( f ( x ( i ) ; w ) , y ( i ) ; w ) , the loss of the optimization problem is L ( w ) = ∑ i∈D l ( f ( x ( i ) ; w ) , y ( i ) ) . Pruning a network is often defined as setting some of its parameters to 0 . Given w , our goal then consists of finding the mask m ∈ { 0 , 1 } d that minimizes L ( w m ) = ∑ i∈D l ( f ( x ( i ) ; w m ) , y ( i ) ) . The mask m must respect some constraints . First , even though the mask is defined at the parameter level , i.e . it has as many components as the number of parameters in the model , we are pruning units . Hence , all parameters corresponding to the same unit must have the same value in the mask . Second , we are interested in pruning only a limited number of units , so the number of elements set to 0 in m is constrained . Finally , we might also want to enforce the number of units pruned at each layer , or simply prevent the pruning at some layers . Denoting as M the set of all masks satisfying these constraints , the optimal pruning is given by m∗ = arg min m∈M ∑ i∈D l ( f ( x ( i ) ; w m ) , y ( i ) ) . ( 1 ) For the remainder of the paper , we shall assume that the number of units to remove is set for each layer independently . This is without loss of generality and will allow us to focus on a single layer , greatly simplifying the presentation . The complexity of solving Eq . 1 increases exponentially with the number of units to prune . Therefore , in practice , people rank all units using a per-unit scoring function s ( w ; D ) . Several examples of such scoring functions will be discussed in Section 3.4 . Units with the lowest score are then pruned . This approach implicitly assumes that the scores of individual units are independent of each other . In other words , pruning one unit is assumed to not affect the score of any other unit . Thus , a good scoring function needs to have a small inter-unit correlation . Once a scoring function s ( w ; D ) has been chosen , we can define m through its elements mi : mi = { 0 if i belongs to a unit u where u ∈ B ( s ( w ; D ) , k ) 1 otherwise . ( 2 ) where B ( x , k ) is the set of k elements of x with the lowest value . In other words , we will set to 0 all the parameters belonging to units whose score is one of the k lowest , k being the number of units we wish to remove in that layer . Pruning a fraction of the units in a particular layer can have a big impact on the network and induce a large loss penalty , that we call the pruning penalty : PruningPenalty = L ( w m ) − L ( w ) ( 3 ) Retraining the network might reduce the pruning penalty at the cost of additional computation . We shall now see how adjusting the biases of the following layer can reduce the pruning penalty with low computational overhead . In order to show this , we need to depart from our earlier definition of pruning as consisting of zeroeing a subset of the weights . 3.2 MEAN REPLACEMENT . We intend to remove k units from a layer of the network in a manner that has a reasonable impact on the network performance . This is often done by replacing these units with zeroes . However , zero is an arbitrary choice and any constant would work . This constant would be “ propagated ” by multiplying it with the outgoing weights of the layer above , which is equivalent to updating the bias of that layer with the resulting sum . Mean replacement consists of replacing the output of pruned units by a constant that is equal to the mean of the unit outputs collected on the training samples before pruning . A theoretical justification for that choice will be presented in Section 3.3 . We will first focus on the removal of a single unit . In a fully connected network , each unit is associated with a single activation . However , in a convolutional layer , a unit is associated with a set of outputs , one per location . In that case , each of these output will be replaced with the same constant . Let a ( x , p ) represent the unit output for training example x at location p ∈ P . Let us randomly choose a subset1 Ds ⊂ D of examples from the training set . We first compute the mean unit output ā = 1 |Ds| ∑ x∈Ds 1 |P| ∑ p∈P a ( x , p ) . 1Usually smaller than the full training set , but big enough to get a good approximation . Mean replacement consists in replacing the pruned unit by the constant ā . This can be implemented by removing the pruned unit in the normal way —which amounts to replace its output by a zero— and folding the constant ā into the bias parameter of the downstream units . b← b + āw , ( 4 ) where b represents the vector of the biases of the downstream units and w represents the outgoing weights of the pruned unit , that is the weights that were connecting the pruned unit to each of its downstream units before the pruning operation . This process is illustrated in Figure 1 . We now justify our choice of constant by showing that , in the context of a quadratic loss , mean replacement is the optimal strategy .

Summary: This paper proposes a simple improvement to methods for unit pruning. After identifying a unit to remove (selected by the experimenter’s pruning heuristic of choice), the activation of that unit is approximately incorporated into the subsequent unit by “mean replacement”. The mean unit activation (computed on a small subset of the training set) is multiplied by each outgoing weight (or convolutional filter) and added to each corresponding bias instead. Experiments show this method is generally better than the typical method of zero-replacement before fine-tuning, though the advantage is smaller after several epochs of fine-tuning.