Title: MLPRUNE: MULTI-LAYER PRUNING FOR AUTO-MATED NEURAL NETWORK COMPRESSION

Abstract: Model compression can significantly reduce the computation and memory footprint of large neural networks. To achieve a good trade-off between model size and accuracy, popular compression techniques usually rely on hand-crafted heuristics and require manually setting the compression ratio of each layer. This process is typically costly and suboptimal. In this paper, we propose a Multi-Layer Pruning method (MLPrune), which is theoretically sound, and can automatically decide appropriate compression ratios for all layers. Towards this goal, we use an efficient approximation of the Hessian as our pruning criterion, based on a Kroneckerfactored Approximate Curvature method. We demonstrate the effectiveness of our method on several datasets and architectures, outperforming previous state-of-theart by a large margin. Our experiments show that we can compress AlexNet and VGG16 by 25x without loss in accuracy on ImageNet. Furthermore, our method has much fewer hyper-parameters and requires no expert knowledge.

1. INTRODUCTION:

Deep neural networks have proven very successful in many artificial intelligence tasks such as computer vision, natural language processing, and robotic control. In exchange for such success, modern architectures are usually composed of many stacked layers parameterized with a large number of learnable weights. This contrasts classical networks which have hundreds or even thousands times fewer parameters. As a consequence, modern architectures require considerable memory storage and intensive computation. This is problematic in applications that need to run on small embedded systems or that require low-latency to make safety-critical decisions.

Fortunately, we can compress these large networks with little to no loss of accuracy, by exploiting the fact that many redundancies exist within their parameters. A popular approach is to quantize the parameters using lower precision, thus encoding the network with fewer bits (Courbariaux et al., 2016; Gong et al., 2014; Rastegari et al., 2016; Wu et al., 2016; Zhu et al., 2016) . On the other hand, pruning techniques (Dong et al., 2017; Guo et al., 2016; Han et al., 2015a; b; Li et al., 2016; Wen et al., 2016) aim at removing redundant connections. It has been widely exploited due to its simplicity and efficacy. In this paper, we follow this line of work and propose a new pruning approach.

Pruning algorithms differ on the pruning criteria employed. Popular pruning methods typically rely on heuristics, such as weight magnitudes. However, a small magnitude does not necessarily mean unimportance (Hassibi & Stork, 1993; LeCun et al., 1990) ; if the input neuron has a large expected value, a small weight could still have a large effect on its output neuron. As a consequence, magnitudebased pruning might delete important parameters, or preserve unimportant ones. Furthermore, magnitude-based pruning requires manually setting the compression ratio of each layer. These ratios are not easy to tune, as different layers (and architectures) typically have different sensitivities for compression. Directly comparing the parameters' magnitudes from all layers and pruning the smallest one would not solve the problem, since magnitudes are not calibrated across layers. Because of this, existing approaches usually involve time-consuming trial and error processes that require domain expertise.

In this paper we employ the Hessian matrix as a principled pruning criterion, as it characterizes the local curvature of the training loss. The Hessian was exploited in the past, but was either assumed to be computable (Hassibi & Stork, 1993) , which is not possible in practice for modern neural networks with millions of parameters, or simplistic approximations (e.g., diagonal (LeCun et al., 1990 )) were utilized resulting in poor performance. Here we first note that the Fisher Information matrix is close to the Hessian under certain conditions. We then use a Kronecker-factored Approximate Curvature (K-FAC) (Martens & Grosse, 2015) method to efficiently estimate the Fisher matrix. In our experiments, the overhead of estimating the Hessian approximation is very small compared with the pre-train and re-train time. Therefore, our method is efficient to use, while exhibiting better compression performance compared with previous second-order pruning. Importantly, this criterion is also calibrated across layers, allowing us to directly compare parameters from different layers and set one global compression ratio for the whole network, rather than one ratio per layer. This makes our method significantly easier to use when compared to magnitude-based pruning. Our method achieves state-of-the-art results on several benchmark datasets and model architectures. Furthermore, it can be easily applied to new customized network architectures, without costly tuning many compression hyper-parameters.

2. BACKGROUND:

In this section, we first introduce the notation used in the paper. Since our work is inspired by the Optimal Brain Surgeon (Hassibi & Stork, 1993) , we also briefly review the idea and its limitations.

2.1. NOTATION:

Suppose we have a neural network with L fully-connected layers1 . Let p(y|x, Θ) be the output distribution and L be the loss function. y denotes the target outputs, x the inputs, and Θ the network's weights. Each layer l is associated with its input a l-1 ∈ R d l-1 , a weight matrix W l ∈ R d l-1 ×d l , a binary mask matrix Γ l ∈ R d l-1 ×d l and its output s l ∈ R d l . The elements γ (i,j) l ∈ {0, 1} in the binary matrix Γ l indicate whether the corresponding weight is retained or pruned away. We use subscript to denote the layer index, and superscript to denote the element index in a matrix. We also distinguish a matrix (vector) from a scalar by capital boldface2 . The weights in a layer l can be computed as θ

(i,j) l = w (i,j) l γ (i,j) l

, and thus the forward-pass can be written as,

EQUATION

where denotes element-wise product, and h is the Relu activation in our experiments. We also use W to denote the concatenation of

[vec{W 1 }, vec{W 2 }, • • • , vec{W L }],

which is a column vector with the same size as the total number of parameters in this neural network, similarly for Γ and Θ. Index q denotes the index number of a parameter in W(or Γ, Θ) that is associated with w

(i,j) l (or γ (i,j) l , θ (i,j) l

). Parameters, weights and connections are used interchangeably.

2.2. A REVIEW ON OPTIMAL BRAIN SURGEON (OBS):

Given a neural network with parameters Θ, the local surface of the training loss can be characterized by its Taylor expansion:

EQUATION

)

where δΘ is a small change of the parameter values (e.g., setting some parameters to zero) and H is the Hessian matrix defined as ∂ 2 L/∂Θ 2 . Note that at convergence the first-order term vanishes and the higher-order term can be neglected. OBS aims to find an index q, such that when changing θ q from its original value to zero, the change in training loss δL is minimized:

EQUATION

where e T q is the unit vector in the weight space corresponding to θ q . The inner minimization problem can be solved by the method of Lagrangian multipliers. Hassibi & Stork (1993) applied OBS on a small network with 18,000 weights and showed good results. However, since the exact solution involves the inverse of the Hessian matrix, original OBS is intractable for modern neural networks that contain millions of parameters.

3. MULTI-LAYER PRUNING:

Our method follows a similar pipeline when compared to conventional pruning methods. We first pre-train a neural network until converged, and then apply Multi-Layer Pruning (including surgeon) upon it. This will give us a compressed network with slightly worse performance. Finally, the network is re-trained to recover the model performance. In the following paragraph, we first introduce the problem formulation and its ideal solution in Section 3.1. We then show how to make approximations in Section 3.2.

3.1. MULTI-LAYER PRUNING AND SURGEON:

Our objective is to prune as much as possible while maintaining good model performance, e.g. high accuracy in classification. This is similar to the objective of Minimum Description Length (MDL, (Rissanen, 1978) )

EQUATION

In our case, the first term in Eq. ( 4) is equivalent to the training loss, and the second term is the description length denoting how many bits we need to encode the model. Here, we use the number of parameters as an indicator of such description length. We also introduce a relative importance λ, which characterizes the trade-off between sparsity and model performance, and controls the final size of a compressed model. Our objective can then be written as

EQUATION

)

where Γ is the binary mask matrix defined in Section 2.1. This objective cannot be directly optimized with SGD as γ (i,j) l is contrained to be binary. Instead, we consider an easier case where we first pre-train the model to the local minimum of L, then update one γ (i,j) l from 1 to 0 if such change will decrease Ψ, or equivalently, if such change will not increase L more than λ. Since the parameter for forward-pass is θ

(i,j) l = γ (i,j) l w (i,j) l , updating γ (i,j) l is equivalent to prune a parameter θ (i,j) l

. Similar to Hassibi & Stork (1993) , we also consider a surgeon operation after pruning. This will change other preserved parameters accordingly, so as to minimize the loss increment caused by the pruning. More specifically, after pruning one parameter θ (i,j) l and applying the surgeon, the increment of L will be,

min δΘ 1 2 δΘ T HδΘ , s.t. δθ (i,j) l + w (i,j) l = 0. (6) δθ (i,j) l

is the corresponding scalar value in δΘ, and H is the Hessian matrix of L w.r.t (W Γ). Solving such a minimization problem with Lagrangian multipliers results in the following update

EQUATION

Here, δΘ * is the surgeon applied on the remaining Θ (and W accordingly). ∆L q is the increment of loss after pruning and the surgeon operation. Again, superscript (q, q) denotes the element at the q th row and q th column, (., q) denotes the q th column in that matrix, and q is the index of the parameter in H that associates with w (i,j) l

. The estimation of H -1 will be introduced in section 3.2.

Theoretically, we can calculate ∆L q for each parameter and prune the one with the smallest ∆L, but this will become impractically time-consuming. Therefore, at each pruning step, we calculate ∆L q for each remaining parameter, and prune all parameters with ∆L q < λ. This is fine as long as such assumption holds, that H does not change significantly when we prune those unimportant parameters one by one. In our experiments, we simply set λ to be the p th percentile of all ∆L q from all layers. Therefore, our method can jointly determine the importance of parameters from all layers and prune multiple layers simultaneously.

After pruning and doing surgeon, we apply SGD to those retained parameters to re-train the network,

EQUATION

3.2. APPROXIMATING HESSIAN USING FISHER:

Performing the pruning and surgeon operations as in Eq. ( 7) involves estimating and inverting the Hessian matrix, which is intractable for modern neural networks that contain millions of parameters.

To efficiently calculate Eq. ( 7), we approximate the Hessian. Towards this goal, we first employ the Fisher Information matrix to approximate the Hessian and further use a Kronecker-factored Approximate Curvature (K-FAC) method to approximate the Fisher matrix. The first approximation comes from the fact that if the training objective is the negative log-likelihood, the Hessian matrix consists of expected second-order derivatives under the data distribution, while the Fisher matrix consists of expected second-order derivatives under the model distribution. Since modern neural networks usually have strong model capacities, we expect those two distributions are close for a well-trained model. The second approximation is demonstrated to be reasonable in optimization tasks (Grosse & Martens, 2016; Martens & Grosse, 2015) , and will further help us to calculate Eq. ( 7) efficiently.

The K-FAC method approximates the Fisher matrix as a block-diagonal matrix and estimates each block by the Kronecker-product of two much smaller matrices, which are the second-order statistics of inputs and derivatives. Given a neural network with stacked fully-connected layers3 , the Fisher Information matrix is defined as

EQUATION

)

Unless specified otherwise, the expectation is taken with respect to the model distribution. We can re-write the Fisher in a block-wise manner by partitioning rows and columns of F if they correspond to parameters within the same layer. The (i, j) block is then4 

EQUATION

which is the covariance of the gradients from i and j layers5 . As noted by Martens & Grosse (2015) , the elements of the covariance of derivatives from different layers are generally smaller than the ones within the same layer. Therefore, F can be approximated by a block-diagonal matrix where the l th block is simply F ll .

Using back-propagation, the gradients of layer l can be calculated as

∇ W l L = (∇ s l L) (a l-1 )

T . The l th diagonal block F ll can then be written as

EQUATION

where ⊗ denotes Kronecker-product. The second equality comes from the property of the Kroneckerproduct6 . The last approximation is supposed to maintain the "coarse structure" of the Fisher matrix (Martens & Grosse, 2015) , and such "coarse structure" is sufficient for our method to achieve a good compression ratio as empirically demonstrated in our experiments. Therefore, F (and thus H) can be approximated by several much smaller matrices

EQUATION

)

For a typical modern neural network such as AlexNet, the original Hessian matrix is a 61M × 61M matrix, while A l-1 and DS l of the largest fully-connected layer have sizes of only 9126 × 9126 and 4096 × 4096 respectively. We use exponential moving average to estimate the expectation in A l-1 and DS l , with a decay factor of 0.95 and a horizon of 1000 steps in all experiments. This adds only a small overhead to the normal forward-backward pass. Furthermore, the inverse F -1 and matrix-vector product F -1 h can be efficiently calculated by leveraging the block diagnoal structure and property of Kronecker-product7 .

Algorithm 1 Multi-Layer Pruning Initialization: W, Γ, Pruning fraction p.

Pre-training Stage::

1: Pre-train the network Pruning Stage:

2: for t = 0, • • • , 1000 do 3: Update A l-1 ← 0.95 × A l-1 + 0.05 × a l-1 a T l-1 4:

Update DS l ← 0.95 × DS l + 0.05 × (∇ s l L)(∇ s l L) T 5: end for 6: Compute Fisher matrix F by Eq. ( 13). 7: Compute importance measure ∆L q for each parameter by Eq. ( 7). 8: Normalize ∆L q within each layer by Eq. ( 14). 9: Compute p th percentile of ∆L q from all layers as λ. 10: Update mask γ (i,j) l to 0 if its corresponding ∆L q is smaller than λ. 11: Compute δΘ * by Eq. ( 7) and update Θ, W ← Θ + δΘ * . 12: Re-train the network by Eq. ( 8).

However, such a block diagonal approximation decorrelates the gradients from different layers, and might lead to different scales for different layers. This will become problematic when we jointly prune all layers, i.e. sorting ∆L q from all layers. We empirically find that ∆L q from different layers will sometimes differ by several orders of magnitudes, which makes it impossible to perform the multi-layer pruning. Therefore, we propose a simple yet effective modification for this problem. After calculating ∆L q , we normalize it within the same layer:

EQUATION

Then, we use ∆L q as an importance measure to prune parameters. Surgeon and other parts remain unchanged. We show our algorithm as Algorithm 1. During pruning, we only need to set an overall compression ratio, and the MLPrune method will automatically decide the compression ratio for each layer. This free us from tuning hyper-parameters, and thus it is easier and more efficient to apply.

4. RELATED WORK:

Model compression task aims to compress a large model into a smaller one while maintaining good performance. There are several popular families of approaches, including pruning, quantization, and low-rank approximation. Quantization (Courbariaux et al., 2016; Gong et al., 2014; Rastegari et al., 2016; Wu et al., 2016; Zhu et al., 2016) aims to use fewer bits to encode each parameter, e.g. binary neural network. Low-rank approximation (Denton et al., 2014; Jaderberg et al., 2014; Lebedev et al., 2014; Novikov et al., 2015) approximates layer weights by low-rank representations, and thus saves the storage and makes speedup. Pruning, being one of the most popular methods due to its simplicity and effectiveness, aims to delete unimportant parameters from a large network. These techniques could be further integrated together and result in better compression ratio (Han et al., 2015a) .

The pruning operation involves finding and deleting unimportant parameters, and thus needs a criterion for deciding importance. Magnitude-based methods (Guo et al., 2016; Han et al., 2015b; Li et al., 2016; Wen et al., 2016) use the absolute value of a parameter as an indicator, assuming smaller weights generally have smaller importance upon a network's output, but there is no theoretical evidence to support this criterion. On the other hand, some other work (Hassibi & Stork, 1993; LeCun et al., 1990) use the second-order Hessian matrix to calculate the importance of each parameter, and show significant improvements when applied to toy neural networks. However, these methods cannot be easily adapted to modern neural networks due to the large size of the Hessian matrix. Recently, Dong et al. (2017) use the layer-wise reconstruction as an indicator, which provides a theoretical upper bound for the pruning error, while avoiding the intractable large Hessian matrix. However, all these methods require specifying one compression ratio for each layer, and searching such ratios are time-consuming for very deep neural networks. In our work, we utilize K-FAC to approximate the Fisher matrix, which in turn approximates the exact Hessian matrix. K-FAC (Grosse & Martens, 2016; Martens & Grosse, 2015) provides an efficient way to estimate and invert an approximation of the Fisher matrix of a neural network. It first approximates the Fisher by a block diagonal matrix, and then decomposes each block by two much smaller matrix via Kronecker-product. Such approximation and its variants (Ba et al., 2016; Wu et al., 2017; Zhang et al., 2017) have shown success in the field of optimization.

5. EXPERIMENTS:

Following previous work, we use compression ratio as our evaluation metric, which is defined as the ratio of the number of parameters that remaining active after pruning to the number of original parameters. To show the generality of our method, we perform experiments using three different datasets and five different deep network architectures, from shallow to deep. This includes LeNet-300-100 and LeNet-5 (LeCun et al., 1998) on MNIST, CifarNet (Krizhevsky & Hinton, 2009) on Cifar-10, and AlexNet (Krizhevsky et al., 2012) and VGG16 (Simonyan & Zisserman, 2014) on Imagenet ILSVRC-2012. The aforementioned architectures contain both fully-connected layers and convolution layers, and have different sizes from 267K to 138M parameters. We compare our results with some strong pruning baselines, including: 1) Randomly Pruning (Dong et al., 2017) , 2) OBD (LeCun et al., 1990) , 3) LWC (Han et al., 2015b) , 4) DNS (Guo et al., 2016) , 5) L-OBS (Dong et al., 2017) . Following these work, we also apply pruning in an iterative manner and report the results in Table 1 . Due to minor differences in implementation, different papers report different performance of pre-trained models. Therefore, we also report the original error and ∆error before and after pruning for each method. More implementation details could be found in Appendix A.

5.1. DOES MLPRUNE ACHIEVES BETTER COMPRESSION RESULTS?:

MNIST: We first conduct experiments on the MNIST dataset with LeNet-300-100 and LeNet-5. LeNet-300-100 has 2 fully-connected layers, with 300 and 100 hidden units respectively. LeNet-5 is a CNN with 2 convolutional layers, followed by 2 fully-connected layers. Table 1 shows that we can compress LeNet-300-100 to 1.3% of original size with almost no loss in performance. Similarly for LeNet-5, we can compress to 0.5%, which is much smaller than previous best result.

Cifar-10: We conduct experiments on Cifar-10 image classification benchmark with Cifar-Net architecture. Cifar-Net is a variant of AlexNet, containing 3 convolutional layers and 2 fullyconnected layers. Following previous work (Dong et al., 2017) , we first pre-train the network to achieve 18.43% error rate on the testing set. We then iteratively prune the network to [50%, 25%, 12.5%, 10%, 8%, 6.4%] of its original size. This gives us a better compression ratio than our baselines while maintaining similar performance.

ImageNet: To demonstrate our method's effectiveness on larger models and datasets, we prune AlexNet on ImageNet ILSVRC-2012. AlexNet has 5 convolutional layers, 3 fully-connected layers, and 61M parameters in total. We first pre-train the network, and achieve similar top-1 accuracy with other implementations. We then iteratively prune the network to [50%, 25%, 12.5%, 6.25%, 5%, 4%] of its original size. To the best of our knowledge, this is the best pruning result so far. Also notice that Han et al. (2015b) and Guo et al. (2016) find out it is necessary to fix the convolutional layers when pruning and retraining the fully-connected layer (and vice versa), otherwise the model cannot recover the pruning error. However, we don't observe such difficulty in our experiments, as we simply prune and retrain all layers simultaneously. This also indicates that our pruning operation only brings small error and still preserves the model capacity.

We further apply our method to a modern neural network architecture, VGG16. VGG16 is a deep network with 13 convolutional layers, 3 fully-connected layers and 138M parameters in total. Due to its large number of layers, the search space of per-layer compression ratio is exponentially larger than that of AlexNet. Therefore, conventional methods would have difficulties to tune and set the compression ratio for each of those 16 layers. On the contrary, our method can be used in a pushbutton-to-start manner, and automatically finds out appropriate ratios. We use the same pruning set-up as in AlexNet experiment, and achieve the state-of-the-art result as shown in Table 1 .

5.2. DOES MLPRUNE FIND REASONABLE PER-LAYER COMPRESSION RATIOS?:

We compare the compression ratios, layer by layer, between our method and the baselines, as shown in Table 2 . The compression ratios of Han et al. (2015b) and Guo et al. (2016) are manually tuned to achieve good results. In contrast, our method automatically determines the compression ratio for each layer during the pruning process. It involves no time-consuming procedure for tuning hyper-parameters yet achieves better results. We also notice that our compression ratios are not the same as previous manually-tuned ratios, but share some similarities; layers with smaller compression ratios in our method also have smaller ratios in the baselines. Also, fully-connected layers are generally be pruned more severe than convolutional layers, which is in accord with the observation that fully-connected layers usually have more redundancies. These suggest that our method can find reasonable per-layer compression ratios according to the sensitivities of each layer. Detailed compression ratios for other architectures could be found in Appendix B. Figure 1 shows the distribution of parameters' magnitudes, coming from the first fully-connected layer of AlexNet. Before pruning, weight distribution is peaked at 0, and drops quickly as the absolute value increasing. This is very much like a gaussian distribution as what the weights were initialized from. Figure 1b shows the distribution after pruning using our method. It is obvious to see that parameters with magnitudes close to 0 (center region) are pruned away, indicating that the parameters regarded as having small impacts upon the loss by our method also usually have small magnitudes. We further explore the correlation between a parameter's magnitude and its importance measured by our method. From figure 2 , we can observe that: 1) A parameter's absolute value indeed has a strong correlation with its importance computed by MLPrune. This explains why magnitude-based methods also achieve fairly good performance. Despite this strong correlation, directly compare the magnitudes from all layer will give different pruning result than our method: setting the pruning threshold as the median of parameters' magnitudes in Figure 2a will prune 14% of conv5, while the median of our importance will prune 7%. This explains why magnitude-based pruning cannot do multi-layer pruning. 2) FC layer initially has smaller importance(larger redundancies) than convolutional layer measured by MLPrune, and thus are pruned more severely. As more and more parameters are pruned away(from Figure 2a to Figure 2b ), the importance of FC layer becomes closer to that of convolutional layer, and both layers will be pruned with similar ratios. This shows that our method can dynamically adjust the compression ratio of each layer, as the pruning going on. 

6. CONCLUSION:

In this paper we have proposed an automated Multi-Layer Pruning (MLPrune) method to compress deep neural networks. Our approach exploits an efficient approximation of the Hessian based on the K-FAC method as our pruning criterion. It has much fewer hyper-parameters and can be used in an automated manner without costly test and trial processes. We have demonstrated the effectiveness of our method on several datasets and architectures and achieved state-of-the-art results. In the future, we plan to extend our method to prune coarser components such as filters, rather than individual weights, so as to speed up the network and save memory furthermore.

B PER-LAYER COMPRESSION RATIOS:

We compare the compression ratio, layer by layer, between our method and the baselines, as shown in Table 4 . The compression ratios of Han et al. (2015b) and Guo et al. (2016) are manually tuned to achieve good results. In contrast, our method automatically determines the compression ratio for each layer during the pruning process. It involves no time-consuming procedure for tuning hyper-parameters yet achieves better results. 

:

Convolutional layers can be transformed into fully-connected layers if we vectorize filters within each channel and expand the input feature map with repeated patches

with minor abuse of notation for a l-1 and s l

Details for convolutional layers are discussed inGrosse & Martens (2016).

Note that vec{} denotes vectorization, as Wi is a matrix, while previously W is a vector. See section2.1

E [∇ W L] = 0, since the expectation is taken over the model distribution

vec uv T = v ⊗ u, (A ⊗ B)(C ⊗ D) = (A ⊗ C)(B ⊗ D).

(A ⊗ B) -1 = A -1 ⊗ B -1 , (A ⊗ B)vec{X} = vec{BXA T }

Original DNS paper doesn't provide result on VGG16. Therefore we adopt numbers from the reimplementation inDong et al. (2017)

Figures and Tables:

Figure 1: Weight distribution of the first FC layer of AlexNet before and after pruning.

Figure 2: Correlation between parameters' absolute values and importances.

Compression Ratios (CR) of Different Architectures

Table content: <table><tr><td>Method</td><td>Architecture</td><td colspan="3">Original Error Final Error ∆ Error</td><td>CR</td></tr><tr><td>Random</td><td>LeNet-300-100</td><td>1.76%</td><td>2.25%</td><td>0.49%</td><td>8%</td></tr><tr><td>OBD</td><td>LeNet-300-100</td><td>1.76%</td><td>1.96%</td><td>0.20%</td><td>8%</td></tr><tr><td>LWC</td><td>LeNet-300-100</td><td>1.64%</td><td>1.59%</td><td>-0.05%</td><td>8%</td></tr><tr><td>DNS</td><td>LeNet-300-100</td><td>2.28%</td><td>1.99%</td><td colspan="2">-0.29% 1.8%</td></tr><tr><td>L-OBS</td><td>LeNet-300-100</td><td>1.76%</td><td>1.96%</td><td>0.20%</td><td>1.5%</td></tr><tr><td colspan="2">MLPrune(Ours) LeNet-300-100</td><td>1.86%</td><td>1.94%</td><td>0.08%</td><td>1.3%</td></tr><tr><td>OBD</td><td>LeNet-5</td><td>1.27%</td><td>2.65%</td><td>1.38%</td><td>8%</td></tr><tr><td>LWC</td><td>LeNet-5</td><td>0.80%</td><td>0.77%</td><td>-0.03%</td><td>8%</td></tr><tr><td>DNS</td><td>LeNet-5</td><td>0.91%</td><td>0.91%</td><td>0.00%</td><td>0.9%</td></tr><tr><td>L-OBS</td><td>LeNet-5</td><td>1.27%</td><td>1.66%</td><td>0.39%</td><td>0.9%</td></tr><tr><td colspan="2">MLPrune(Ours) LeNet-5</td><td>0.85%</td><td>0.89%</td><td>0.04%</td><td>0.5%</td></tr><tr><td>LWC</td><td>Cifar-Net</td><td>18.57%</td><td>19.36%</td><td>0.79%</td><td>9%</td></tr><tr><td>L-OBS</td><td>Cifar-Net</td><td>18.57%</td><td>18.76%</td><td>0.19%</td><td>9%</td></tr><tr><td colspan="2">MLPrune(Ours) Cifar-Net</td><td>18.43%</td><td>18.60%</td><td colspan="2">0.17% 6.4%</td></tr><tr><td>DNS</td><td>AlexNet</td><td>43.42%</td><td>43.09%</td><td colspan="2">-0.33% 5.7%</td></tr><tr><td>LWC</td><td>AlexNet</td><td>42.78%</td><td>42.77%</td><td>-0.01%</td><td>11%</td></tr><tr><td>L-OBS</td><td>AlexNet</td><td>43.30%</td><td>43.11%</td><td>-0.19%</td><td>11%</td></tr><tr><td colspan="2">MLPrune(Ours) AlexNet</td><td>43.17%</td><td>43.14%</td><td colspan="2">-0.03% 4.0%</td></tr><tr><td>DNS 8</td><td>VGG16</td><td>31.66%</td><td>63.38%</td><td colspan="2">31.72% 7.5%</td></tr><tr><td>LWC</td><td>VGG16</td><td>31.50%</td><td>31.44%</td><td colspan="2">-0.06% 7.5%</td></tr><tr><td>L-OBS</td><td>VGG16</td><td>31.66%</td><td>32.02%</td><td>0.36%</td><td>7.5%</td></tr><tr><td colspan="2">MLPrune(Ours) VGG16</td><td>30.95%</td><td>30.78%</td><td colspan="2">-0.17% 4.0%</td></tr></table>

AlexNet Per-layer Compression Ratios if MLPrune prune similar parameters as magnitude-based pruning, or they prune totally different parameters.

Table content: <table><tr><td colspan="6">Architecture Layer Parameters Han et al. (2015b) Guo et al. (2016) MLPrune (Ours)</td></tr><tr><td/><td>conv1</td><td>35K</td><td>84%</td><td>53.8%</td><td>67.2%</td></tr><tr><td/><td>conv2</td><td>307K</td><td>38%</td><td>40.6%</td><td>37.8%</td></tr><tr><td/><td>conv3</td><td>885K</td><td>35%</td><td>29.0%</td><td>27.7%</td></tr><tr><td/><td>conv4</td><td>663K</td><td>37%</td><td>32.3%</td><td>33.2%</td></tr><tr><td>AlexNet</td><td>conv5</td><td>442K</td><td>37%</td><td>32.5%</td><td>38.6%</td></tr><tr><td/><td>fc1</td><td>38M</td><td>9%</td><td>3.7%</td><td>1.5%</td></tr><tr><td/><td>fc2</td><td>17M</td><td>9%</td><td>6.6%</td><td>3.2%</td></tr><tr><td/><td>fc3</td><td>4M</td><td>25%</td><td>4.6%</td><td>13.7%</td></tr><tr><td/><td>total</td><td>61M</td><td>11%</td><td>5.7%</td><td>4%</td></tr><tr><td colspan="6">5.3 DOES MLPRUNE PRUNE SIMILAR PARAMETERS AS MAGNITUDE-BASED PRUNING?</td></tr></table>

Per-layer Compression Ratios

Table content: <table><tr><td>Architecture</td><td>Layer</td><td colspan="4">Parameters Han et al. (2015b) Guo et al. (2016) MLPrune (Ours)</td></tr><tr><td/><td>fc1</td><td>235K</td><td>8%</td><td>1.8%</td><td>0.73%</td></tr><tr><td>LeNet-300-100</td><td>fc2 fc3</td><td>30K 1K</td><td>9% 26%</td><td>1.8% 5.5 %</td><td>4.74% 39.4%</td></tr><tr><td/><td>total</td><td>267K</td><td>8%</td><td>1.8%</td><td>1.3%</td></tr><tr><td/><td>conv1</td><td>0.5K</td><td>66%</td><td>14.2%</td><td>41.2%</td></tr><tr><td/><td>conv2</td><td>25K</td><td>12%</td><td>3.1%</td><td>3.1%</td></tr><tr><td>LeNet-5</td><td>fc1</td><td>400K</td><td>8%</td><td>0.7%</td><td>0.2%</td></tr><tr><td/><td>fc2</td><td>5K</td><td>19%</td><td>4.3%</td><td>8.9%</td></tr><tr><td/><td>total</td><td>431K</td><td>8%</td><td>0.9%</td><td>0.5%</td></tr><tr><td/><td>conv1</td><td>35K</td><td>84%</td><td>53.8%</td><td>67.2%</td></tr><tr><td/><td>conv2</td><td>307K</td><td>38%</td><td>40.6%</td><td>37.8%</td></tr><tr><td/><td>conv3</td><td>885K</td><td>35%</td><td>29.0%</td><td>27.7%</td></tr><tr><td/><td>conv4</td><td>663K</td><td>37%</td><td>32.3%</td><td>33.2%</td></tr><tr><td>AlexNet</td><td>conv5</td><td>442K</td><td>37%</td><td>32.5%</td><td>38.6%</td></tr><tr><td/><td>fc1</td><td>38M</td><td>9%</td><td>3.7%</td><td>1.5%</td></tr><tr><td/><td>fc2</td><td>17M</td><td>9%</td><td>6.6%</td><td>3.2%</td></tr><tr><td/><td>fc3</td><td>4M</td><td>25%</td><td>4.6%</td><td>13.7%</td></tr><tr><td/><td>total</td><td>61M</td><td>11%</td><td>5.7%</td><td>4%</td></tr><tr><td/><td>conv1_1</td><td>2K</td><td>58%</td><td>-</td><td>92.1%</td></tr><tr><td/><td>conv1_2</td><td>37K</td><td>22%</td><td>-</td><td>66.4%</td></tr><tr><td/><td>conv2_1</td><td>74K</td><td>34%</td><td>-</td><td>55.7%</td></tr><tr><td/><td>conv2_2</td><td>148K</td><td>36%</td><td>-</td><td>46.9%</td></tr><tr><td/><td>conv3_1</td><td>295K</td><td>53%</td><td>-</td><td>38.2%</td></tr><tr><td/><td>conv3_2</td><td>590K</td><td>24%</td><td>-</td><td>30.8%</td></tr><tr><td/><td>conv3_3</td><td>590K</td><td>42%</td><td>-</td><td>30.5%</td></tr><tr><td/><td>conv4_1</td><td>1M</td><td>32%</td><td>-</td><td>20.6%</td></tr><tr><td>VGG16</td><td>conv4_2</td><td>2M</td><td>27%</td><td>-</td><td>12.9%</td></tr><tr><td/><td>conv4_3</td><td>2M</td><td>34%</td><td>-</td><td>12.8%</td></tr><tr><td/><td>conv5_1</td><td>2M</td><td>35%</td><td>-</td><td>13.1%</td></tr><tr><td/><td>conv5_2</td><td>2M</td><td>29%</td><td>-</td><td>13.9%</td></tr><tr><td/><td>conv5_3</td><td>2M</td><td>36%</td><td>-</td><td>13.3%</td></tr><tr><td/><td>fc1</td><td>103M</td><td>4%</td><td>-</td><td>0.3%</td></tr><tr><td/><td>fc2</td><td>17M</td><td>4%</td><td>-</td><td>1.9%</td></tr><tr><td/><td>fc3</td><td>4M</td><td>23%</td><td>-</td><td>9.2%</td></tr><tr><td/><td>total</td><td>138M</td><td>7.5%</td><td>-</td><td>4%</td></tr></table>

Peer review (iclr2019):
 This paper introduces an approach to pruning the parameters of a trained neural network. The idea is inspired by the Optimal Brain Surgeon method, which relies on second derivatives of the loss w.r.t. the network parameters. Here, the corresponding Hessian matrix is approximated using the Fisher information to make the algorithm scalable to very deep networks. Strengths: - The method does not require hyper-parameter tuning. - The results show the good behavior of the approach. Weaknesses: Novelty: - In essence, this method relies on the work of Marten & Grosse to approximate the Hessian matrix used in the Optimal Brain Surgeon strategy. This is fine, but not of great novelty. Method: - It is not clear to me why the notion of binary parameters gamma is necessary. Instead of varying the gammas from 1 to 0, why not directly zero out the corresponding network parameters w? - In essence, the objective function in Eq. 5 adds an L_1 penalty on the gamma parameters, which would be related to an L_1 penalty on the ws. Note that this strategy has been employed in the past, e.g., Collins & Kohli, 2014, "Memory Bounded Deep Convolutional Networks". - It is not clear to me how zeroing out individual parameters will truly allows one to reduce the model afterwards. In fact, one would rather want to remove entire rows or columns of the matrix W_l, which would truly correspond to a smaller model. This was what was proposed by Wen et al., NIPS 2016 and Alvarez & Salzmann, NIPS 2016, "Learning the Number of Neurons...". - In the past, when dealing with the Hessian matrix, people have used the so-called Pearlmutter trick (Pearlmutter, Neural Computation 2014, "Fast exact multiplication by the Hessian". In fact, in this paper, the author mentions the application to the Optimal Brain Surgeon strategy. Is there a benefit of the proposed approach over this alternative strategy? Experiments: - While the reported compression rates are good, it is not clear to me what they mean in practice, because the proposed algorithm zeroes out individual parameters in the matrix W_l of each layer. This does not guarantee entire channels to be removed. As such, I would not know how to make the model actually smaller in practice. It would seem relevant to show the true gains in memory usage and in inference speed (both measured on the computer, not theoretically). Summary: I do appreciate the fact that the proposed method does not require hyper-parameters and that it seems to yield higher compression rates than other pruning strategies that act on individual parameters. However, novelty of the approach is limited, and I am not convinced of its actual benefits in practice.