Title: AUTO NETWORK COMPRESSION WITH CROSS-VALIDATION GRADIENT

Abstract: Network compression technology can compress large and complex networks into small networks, so that it can be deployed on devices with limited resources. Sparse regularization method, such as L 1 or L 21 regularization, is the most popular method that can induce sparse model. However, it introduces new hyperparameters, which not only affects the degree of sparsity, but also involves whether the network can be effectively trained (gradient explosion or model nonconvergence). How to select hyperparameters becomes an important and open problem for regularization-based network compression method. In this paper, we propose an auto network compression framework with cross-validation gradient which can automatically adjust the hyperparameters. Firstly, we design an unified framework which combines model parameter learning with hyperparametric learning. Secondly, in order to solve the problem of non-derivability of L 1 norm, we introduce auxiliary variables to transform it into a solvable problem, and then obtain the derivative of model parameters with respect to hyperparameters. Finally, the derivative of the hyperparametric vector is solved by the chain rule. In solving the inverse problem of Heisen matrix, we compare three methods and only calculate the mixed partial derivatives. To a certain extent, this method realizes the automatic network compression. Classical network structures such as VGG, ResNet and DensNet are tested on CIFAR-10 and CIFAR-100 datasets to prove the effectiveness of our algorithm.

1. INTRODUCTION:

Network compression technology compresses the increasing size of model into one that occupies less memory resources and reduces the amount of computation of the big model, making it possible for AI to run on mobile phones. Common model compression methods are divided into four different categories: network pruning, network quantization, knowledge distillation, and matrix-based lowrank approximation (Cheng et al., 2018; Lee et al., 2019; Sakr & Shanbhag, 2019; Crowley et al., 2018; Peng et al., 2018) . Pruning simplifies the large networks and select one of the subsets of large model essentially. It can reduce more than half of the parameters without significantly reducing the accuracy and greatly improve the compression rate. Our method discussed in this paper fall into the category of pruning.

Pruning based on regularization term is a very popular method. Classical machine learning methods use L 0 regularization to induce sparse models. Because L 0 norm is difficult to optimize, it is usually replaced by L 1 norm. In the field of deep learning, previous work used various norms to train the model parameters (Han et al., 2015; Mummadi et al., 2019; Liu et al., 2019; Li et al., 2019; Mehta et al., 2019) . However, few work consider the problem of selecting hyper-parameters which introducd by regularization term. Hyperparameters not only determine the penalty intensity of the model, but also affect the network accuracy after fine-tuning. So how to select hyperparameters becomes an important and open problem for model compression.

In order to alleviate the above problem, a general method for automatically determining hyperparameters is studied. Both L 1 norm and L 2 norm are conducive to induce sparse model. The objective function of our study is based on the above two norms. We calculate the gradients of the coefficients of L 1 and L 2 regularization terms, and use cross validation to update hyperparameters. Our contributions are summarized as follows: • We propose a unified framework (see figure 1 ) that combines the whole parameters training process with hyperparametric optimization.

• Using auxiliary variables, we construct a problem that satisfies the differentiable condition and solve the derivative problem of network parameters with respect to hyperparameter. The chain rule of derivatives is used to obtain the complete solution of derivatives of validation set loss with respect to hyperparameters. Finnally, the stochastic gradient descent algorithm are used to update and optimize the hyperparameters.

• Compared with state of art methods, our algorithm achieves a higher sparse rate in VGG, ResNet, DensNet on CIFAR-10 and CIFAR-100 datasets. Accuracy loss is controlled in a negligible range.

The structure of the paper is as follows: The second part introduces the related works: regularizationbased sparse method and gradient-based hyperparameter selection; the third part introduces the whole framework, focuses on the cross-validation gradient solution of non-derivative norm, and gives the corresponding theorem and proof. Finally, a hyperparameter gradient approximation method and algorithm are introduced, which effectively solves the optimization problem of deep neural network. The fourth part mainly compares with state of art methods using three different types of network models and two data sets.

2. RELATED WORKS:



2.1. DEEP NEURAL NETWORK PRUNING:

The pruning methods are divided into supervised pruning and unsupervised pruning according to different criteria. Unsupervised criteria, as the name implies, analyze the importance of model parameters after training, and the analysis rules need not to be acquired through learning. For example, the L 1 norm criterion was used in Li et al. (2017) and followed by retraining. This method calculates the L 1 norm values of every channel in each layer, and retains the channel weights with large norm values. It is simple and effective. The learned rules (Liu et al., 2017; Zhuang et al., 2018; Luo et al., 2017) are more persuasive than simply determined by experience. Liu et al. (2017) learned an important factor for each channel and adds sparse constraints to the objective function.

After the training is completed, the less important channels under a certain threshold are removed and retrain the new network. DCP (Zhuang et al., 2018) chose the larger F norm of the current channel gradient as the channel to be retained. Luo et al. (2017) used the output of the next layer to guide the pruning of the current layer, that is, to approximate the output with a subset of the input of the current layer. The channels only in the subset are retained. We synthesize the advantages of L 1 and L 2 norms and add them to the objective function. The difference between L 2 norm and weight delay is that L 2 norm only acts on weight coefficients. Compared with other norms, L 1 norm has better sparsity performance.

Adding sparse regularization term to the original loss function is another technical means in pruning algorithm. Han et al. (2015) proposed to use L 1 and L 2 norms to induce network sparsity, and found that L 1 regularization can achieve better accuracy after pruning, while sparse network induced by L 2 regularization can achieve higher accuracy after fine-tuning. Mummadi et al. (2019) proposed that the bounded-L p0 norm is used to approximate the parameters to zero to ensure the accuracy of the task, and then the bounded-L p0 norm is used to further induce the parameters of additional gating layer to 0. However, two regularization coefficients are directly given without further discussion. Liu et al. (2019) used L 21 norm of each instance feature to learn sparsity during training.

The selection of the coefficients of the regularization factor is only based on a limited test. Li et al. (2019) proposed a criterion to measure the importance of the channel by using the Group Lasso of two consecutive layers, and then the pruned network is trained with a OICSR loss. Mehta et al. (2019) discussed the error rate of different regularization coefficients under different optimizers. Although regularization-based pruning methods have been extensively studied, none of them explored regularization coefficients in a manner of automatically updating hyperparameters. Based on this background, we analyze how to optimize the regularized hyperparameters. As far as we know, this is the first time that the coefficient of a regular term is determined by an automatic method.

2.2. HYPERPARAMETER OPTIMIZATION BASED ON GRADIENT:

There are many studies on hyperparameter selection, such as grid search, random search, bayesian optimization and gradient-based cross-validation (Barratt & Sharma, 2018) . The gradient-based method can find a suitable solution more quickly. Rencent works have explored gradient-based hyperparameter optimization. They focus on optimizing multiple parameters and reducing computational costs. Barratt (2018) proposed that the solution map of convex optimization problems satisfying certain conditions is differentiable by implicit function theorem. Barratt & Sharma (2018) demonstrated that the derivatives of the solution for many common convex machine learning algorithms: logistic regression, elastic-net regression, support vector machines.

In the field of deep learning, there are also some studies on gradient optimization of hyperparameter for different purposes. Ren et al. (2018) solved the problem of hyperparameter adjustment for example reweighting algorithms using cross validation based on gradient descent. Reverse mode automatic differentiation allow for optimizing hyperparameters with gradients (Maclaurin et al., 2015) . But they consider that every iteration in the whole gradient descent, it is a huge overhead to store all gradients in memory, so they propose a reverse mode differentiation of SGD with finite precision arithmetic to avoid avoid the memory limitation on the task of hyperparameters optimization. Fu et al. (2016) distilled the knowledge of the gradients in the forward propagation and transfer them by adding short connections, so that the gradients in the forward propagation can be approximately calculated in the reverse process. This algorithm greatly improves computational efficiency and reduces memory consumption. Luketina et al. (2016) only considered one iteration in the training process, and approximated the Heisen matrix as a identity matrix.

These works mainly focus on the optimization of multiple parameters and only consider traditional machine learning problems or shallow neural network due to a large amount of computation and memory consumption. We mainly study the hyperparameter selection of large-scale neural network models for the purpose of model compression. In the context of sparse regularization compression, we often encounter non-derivative cases, such as L 1 norm, L 21 norm. Direct hyperparameter gradient is almost impossible to obtain. In this paper, we use auxiliary variables to transform non-derivable norms into equivalent derivable optimization problems. We also redesign the training process and propose a new framework which combines model parameter training with hyperparametric learning to compress big model, which makes the hyperparameter optimization of large CNN models possible and obtains sparse neural network models.

3. THE PROPOSED ALGORITHM:



3.1. REGULARIZATION-BASED NETWORK SPARSIFICATION METHOD:

In order to obtain sparse network structures and minimize the loss drop, we use an objective function with L 1 and L 2 regularization terms for neural networks. (see equation 1). Adding L 1 regularization term can induce sparse neuronal connections. The purpose of adding L 2 regularization term is to reduce the over-fitting of the model and increase the generalization performance of the model.

Weight delay is such a technique to avoid model over-fitting. The L 2 norm used in this paper is slightly different from the weighted delay. The weighted delay is limited to all parameters, including bias terms, but the L 2 norm is only limited to the weighted coefficients.

EQUATION

EQUATION

EQUATION

)

The loss function is cross entropy function -

m k=1 (y n k log o n k ).

o is the output of a neural network model, y is the one-hot laber. w 1 is the L 1 norm sum of all weight tensors of the target model (see definition 2). w 2 is the square sum of Frobenius norm of all weight tensors (see definition 3).

The training set is {(X n , y n )} for n = 1, 2, 3, . . . , N . (X n , y n ) ∈ R d×d × R m .

The input is a d-dimensional image matrix. Our target is minimize e on the training set. Our objective function not only guarantees the sparsity of the model, but also improves the accuracy of the model on the test set. In order to balance loss function and regularization terms, λ 1 , λ 2 are introduced to represent the weight of each part. Automatic hyperparameter selection based on gradient descent method reduces the labor and time cost. We use cross-validation gradient to select appropriate hyperparameters and obtain sparse models. The cross-validation loss function used in this paper is cv. It is also possible to use other loss functions instead of cross-entropy loss function.

EQUATION

In addition, we minimize cv on the verification set {( Xn , ŷn )} for n = 1, 2, 3, . . . , M . In fact, M and N represent the batch size of training set and verification set respectively.

3.2. OUR OVERALL FRAMEWORK:

The proposed process is as follows:

• 1. Initialize hyperparameter λ = [λ 1 , λ 2 ];

• 2. On the training set {(X n , y n )}, using a standard mini-batch SGD algorithm to learn the optimal W (t) :

EQUATION

• 3. On the verification set { Xn , ỹn }, update the λ (t) , this is also achieve by using a minibatch SGD algorithm.

λ (t) = λ (t-1) -η∇ λ cv.

(6) • 4. Go to step 2 or stop when satisfie the termination condition of iteration (see equation 7)

and obtain sparse model.

EQUATION

The key point and difficulty of the whole algorithm is to solve the hyperparameter gradient:

EQUATION

In equation 8, the left part derivative is the conventional derivative, while the right part derivative does not necessarily exist because that the L 1 norm is not differentiable. s(λ) means the intermediate variable consist of the optimal weight variables in W. W can be expressed as an implicit function vector. We will give the results in the next section.

3.3. DERIVATIVES OF SOLUTION MAPPING WITH RESPECT TO HYPERPARAMETERS:

In order to solve the problem of nondifferentiability of L 1 norm, we introduce auxiliary variables Wl+ > 0, Wl-> 0 to transform the original problem into a general solvable form. Satisfy W l = W l+ -W l-. w is a vector which consisit of the elements of Wl+ , Wl-, μ. µ is dual variables with respect to W l+ , W l-. ˜means the optimal solution. Theorem 1. Deep Neural Network performs classification, where {(X n , y n )} is sample datas. The activation function is differentiable, the derivatives of the optimal solution w = [ Wl+ , Wl-, μ] with respect to λ are:

EQUATION

Proof. Because L 1 is not differentiable, then we use W l+ , W l-(W l = W l+ -W l-) to rewrite the problem as (1 is a column vector and its elements are 1):

EQUATION

Namely minimize e(W, λ 1 , λ 2 ), subject to W l+ i,j,p,q > 0, W l- i,j,p,q > 0, for l = 1, 2, 3, . . . , L.

We expansion the optimal solution Wl+ , Wlwith dual variables μ = [ μ1 , μ2 , µ 3 , . . . , µ d2 ] by vector, then we can get w. θ is only the optimal solution Wl+ , Wlexpansioned by vector. The dimension of w is

d 1 = L l=1 (4i l j l ). d 2 = 2 L l=1 (i l j l ) is dimension of µ.

Let s * (λ) = wT denote the optimal w for a given λ in equation 10, we can get the derivative of w with respect to λ (Barratt, 2018; Barratt & Sharma, 2018) .

EQUATION

EQUATION

)

The general form of derivatives of model parameter with respect to hyperparameters is given by this theorem, which is the most important part of cross-validation gradient. The model here includes but is not limited to CNN, DNN, LSTM, RNN and other deep neural network models.

3.4. APPROXIMATE GRADIENT FOR HYPERPARAMETER:

Because the calculation of hyperparameter gradient involves the inverse operation of Heisen matrix, its complexity is cubic dimension. The order of magnitude of the parameters of the deep neural network is 10 6 . The time complexity of the matrix inversion is o(d 3 ) ≈ o(10 18 ). Unless the number of network parameters is small, it is impossible to calculate them. In order to calculate the inverse of Heisen matrix conveniently, we only consider the diagonal approximation of Heisen matrix. Ricotti et al. (1988) given an exact formula for calculating diagonal elements:

EQUATION

Becker & Lecun (1989) and LeCun et al. (1989) given the formula for calculating the diagonal elements:

EQUATION

)

This formula is still complicated. Here we follow the example used in (Luketina et al., 2016) :

EQUATION

and approximate it to a identity matrix:

EQUATION

Then we can get ∇ λ cv ≈ -∂cv ∂s(λ) ∇ λ g( Wl+ , Wl-, μ, λ). By simplifying the multiplication of zero Fix W (t) , minimize cv on verification set { Xn , ỹn } 3:

Calculate the gradients of hyperparameters: ∇ λ cv 4:

Update λ (t) = λ (t-1) + η ∂cv ∂s(λ) ∇ λ g( Wl , λ)

5:

Reinitialize parameters W (t) 6:

for all W (t) not converged do 7:

Fix λ, minimize e on training set {(X n , y n )} 8:

Update parameters W (t) by mini-batch SGD method 9:

end for 10: end for elements, we can see that dual variables µ can be simplified eventually. By observing the auxiliary variables in the mixed partial derivatives, their forms are consistent and can be reconstituted into one variable. Then we get:

EQUATION

Finally, we use the automatic derivation module in PyTorch, an open source framework, to assist in calculating the hyperparameter gradient. The computational cost of the algorithm 1 is product of outer training epoches and inner training epoches.

4. EXPERIMENT:



4.1. NETWORK MODEL AND DATA SET:

The network models we use are VGG (Simonyan & Zisserman, 2015) , ResNet (He et al., 2016) and DenseNet (Huang et al., 2017) . The vision(VGG-16) of VGG we used is consisted of 13 convolution layer, 2 full connected layer and 4 max pooling layer. The dimension of the full connection layer is 512*512 and 512*class number. We also compare residual networks with different depths of ResNet-56 and ResNet-110. DenseNet-40 is a dense connected network of 3 blocks. In each block, the features of each layer stack up and flow down to the next layer.

The data sets we use are CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009) . CIFAR-10 has 60,000 images: 50,000 training images and 10,000 test images. Each sample is a 32*32 color image.

There are 5000 training samples in each category in CIFAR-10. The sample partitioning of CIFAR-100 is the same with CIFAR-10, but with the increase of the number of categories, the number of samples for each category is reduced to 500. This greatly improves the difficulty of classification tasks. The same model structure shows worse performance in CIFAR-10. In addition, due to the requirement of our algorithm, we divide a part of the data set from the training set as the verification set which number is 5000. Reducing the number of training samples will affect the accuracy of classification to a certain extent. Our experimental implementation is based on PyTorch with a Tesla P100 GPU. The training and sparsification of the model are accomplished together, and it is an end-to-end process. Except for the coefficients of regularization terms, the other hyperparameters are fixed. The batch sizes of training and testing are 64 and 256 respectively. The SGD momentum is 0.9. The initial learning rate is 0.1 and then divided by 10 at 1/2 and 3/4 total epoches. The weight decay is not set because that we add an L 2 regularization term only on weight. Other hyperparameters affecting the algorithm are demonstrated and analyzed in experiments. Our algorithm is an automatic pruning method. The L 1 regularization term is used to induce the sparse network. When the training stage is completed, the pruning network can be obtained without fine-tuning stage. The whole framework is an end-to-end iterative process. The training and validation stages alternate to achieve faster convergence results. In the training set, the objective function is the cross-entropy loss function and two regularization terms. The cross-entropy function is to better fit the probability distribution of the training set and achieve higher accuracy. The L 2 regularization term is to constantly punish the model to improve the generalization performance of the model in the test set. The L 1 regularization term shrinks the weight coefficients to a small numerical range and tends to zero. Because L 1 is not differentiable at zero, the approximate gradient calculation can not make the weight zero in the limited number of training rounds. We design a threshold to assist the L 1 regularization term. On the verification set, we still choose the cross-entropy function as the verification loss. At this time, the parameters of the model are fixed and do not participate in the update. We calculated the gradients of L 1 and L 2 regularization coefficients. Because the number of samples in the validation set is still large, we adopted mini-batch SGD to update the super-parameters.

The sample size of the verification set is only 1/10 of the training set, and the number of training rounds of each iteration is also 1/10 of the training set for empirical consideration. The above process is called an iteration. Strictly speaking, every iteration should train the model from scratch on training set. Because a complete training takes a long time, we take a roundabout way that, each iteration does not need to re-initialize the weight, but re-train the network of existing parameters.

The training epoches in each iteration is regard as the total number of training epoches. Compared with GAL-0.1, CV-80 achieves few parameters of 2.3M which is less than 2.67M achieved by GAL-0.1. At the same time, our accuracy is 0.33% higher than GAL-0.1. The total computation of our algorithm is not reduced, but many parameters of the filter are changed to zero in the process of training. For ResNet-56, our algorithm achieves the same accuracy loss compared with NISP, however we have a higher compression rate of 60% than that of 42.6%. Compared with GAL-0.6 and L 1 norm, our accuracy is improved by 0.26 in the case of low compression rate. For deeper ResNet-110, we achieve a the best performance to the extent we know, the compression ratio is 72.8%. However, the accuracy rate is 0.16 higher than the baseline. The configuration of our algorithm is exactly the same with the baseline, the only difference is whether to prune or not. For DenseNet-40, compared with Synapse Pruning, we achieve a higher pruning rate with accuracy rate loss of 0.62% (CV-40-lr). "CV-40-lr" means the starting pruning epoch is 40 with learning rate of 10 -4 . Compared to GAL-0.05, we get a high pruning rate of 71.3% than 56.7% and a lower accuracy loss of 0.23% than 0.31%.

Ablation Experiment for the Start Prune Epoch Table 2 show the differences in different start prune epoch. The pruning threshold is 0.0001, and the weight below this threshold will be reset to 0. We divide the total training epoches by four, -x means the number of epoches that start pruning. Observed by us, the earlier the pruning starts, the better the sparsity of the network structure is. Surprisingly, the pruned network can achieves better accuracy than the baseline. The essence of pruning is to select a better network structure suitable for the task, and its performance will not decline or even improve without over-pruning. Similar results are also found on deeper residual networks, as shown in Table 3 . Each iteration implies a hyperparameter update. The total number of epoch is 160 with each iteration has only 40 epoches for quickly trial run.

Ablation Experiment for the Learning Rate of Hyperparameter The learning rate of hyperparameters has a great influence on the sparsity and prediction accuracy of the network (see figure 2 ).

Through experiments, we find that simple structure, easy-to-train network such as VGG can use the learning rate of 10 -4 appropriately, while for densely connected composite structures such as DenseNet, a smaller learning rate of 10 -5 can make the model still be fully trained in the process of sparse.

Results on CIFAR-100 As shown in figure 3 , we compare the accuracy of VGG-16 on CIFAR-100 before and after compression. Intuitively, without considering pruning rate, the performances of models under different settings on the test set have very little difference. From the view of data, pruning does not reduce the accuracy rate, and to some extent improves the generalization of the model. Among them, the blue line is slightly higher and achieves the best performance of 72.79%. It is also the blue line that achieves the highest compression rate of 90.6%.

A APPENDIX

Example 1. Deep Neural Network performs classification, where {(x n , y n )} is sample datas. The activation function is differentiable, the derivatives of the optimal solution w = [ W l+ , W l-, μ] with respect to λ are:

∇ λ s(λ) * = -∇ wg( W l+ i,j , W li,j , μt , μt+1 λ 1 , λ 2 ) -1 ∇ λ g( W l+ i,j , W li,j , μt , μt+1 , λ 1 , λ 2 ) (18)

When the gradient of the weight parameter with respect to the hyperparameter is solved, the gradient of the hyperparameter can be given by the chain rule:∇ λ cv = ∂cv ∂s(λ) * ∇ λ s(λ) * . The following equation is just an example of four representative variables:

W l+1+ i,j , W l+1- i,j

, μt , μt+1 . 

EQUATION

g ∈ R 1×d , ∇ wg ∈ R d×d , ∇ λ g ∈ R d×2 , ∇ λ s(λ) ∈ R d×2 , ∂cv ∂s(λ) ∈ R 1×d , ∇ λ cv ∈ R 1×2 . Two elements in ∇ λ cv are represented the cross-validation gradients of λ 1 and λ 2 separately.

:

CONCLUSIONIn order to solve the problem of setting hyperparameters in model compression algorithm based on regularization term, we propose a gradient-based hyperparameter learning method. In order to calculate the derivative of network parameters with respect to hyperparameter, we introduce auxiliary variables to construct a solvable problem. With the help of the chain rule, the derivative of verification loss with resprct to hyperparameter can be solved. This method solves two problems, one is the sparsity of deep neural network, and the other is the solution of regularization coefficient. We calculate the exact gradient of L 1 and L 2 regularization coefficients, and propose an approximate calculation method to realize the hyperparameter optimization of large convolutional neural networks. In addition, we have done comparative experiments on several classical network structures and data sets to demonstrate the effectiveness of our algorithm. In the future, we will investigate the effects of other hyperparameters such as batch size and momentum coefficient which may also affect the sparsity.

Figures and Tables:

Figure 1: The whole big circle is only one iteration, and the white bottom rectangle on the left is the optimization process of the inner neural network. The inner process optimizes W by mini-batch SGD to minimize the loss function in training set. The blue part on the right is the gradient descent of cross validation loss on validation set. Weight is fixed and regularization coefficient is updated for several epoches.

Figure 2: This Figure shows the performance of DenseNet-40 on CIFAR-10 with different learning rate of hyperparameters. We only list the data at the end of each iteration. Left: With the LR=0.00001, the test accuracy rate of DenseNet-40 is steady increasing. When the learning rate increases, there is a downward trend in test accuracy rate. Right: The larger the learning rate of hyperparameters is, the sparser the model obtained in the same iteration process is.

Algorithm 1 Cross-Validation Gradient Method for Model Compression Input: Initialize hyperparameter λ = [λ 1 , λ 2 ]

Table content: <table/>

The overall results compared with different methods on CIFAR-10. We compare four different neural network: Acc gain is the absolute error of the percentage of accuracy compared to each algorithm' own baseline for relatively fair comparison. Parameters and FLOPs represent the total parameters of the network model and the sum of addition and multiplication operations. Pruned and Reduced represent the percentage of parameter reduction and computation reduction. CV represent the proposed Cross-validation method by us.

Table content: <table><tr><td>Model</td><td colspan="4">Acc gain Parameters Pruned FLOPs</td><td>Reduced</td></tr><tr><td>VGG-16(Our baseline)</td><td>93.55%</td><td>14.99M</td><td>0%</td><td>0.31G</td><td>0%</td></tr><tr><td>CV-40(Ours)</td><td>-0.26%</td><td>0.34M</td><td>97.7%</td><td>-</td><td>-</td></tr><tr><td>CV-120(Ours)</td><td>-0.21%</td><td>2.30M</td><td>84.7%</td><td>-</td><td>-</td></tr><tr><td>L 1 norm (Li et al., 2017)</td><td>0.15%</td><td>5.4M</td><td>64%</td><td>0.21G</td><td>34.2%</td></tr><tr><td>GAL-0.05 (Lin et al., 2019)</td><td>-0.19%</td><td>3.36M</td><td>77.6%</td><td colspan="2">189.49M 39.6%</td></tr><tr><td>GAL-0.1 (Lin et al., 2019)</td><td>-0.54%</td><td>2.67M</td><td>82.2%</td><td colspan="2">171.89M 45.2%</td></tr><tr><td colspan="2">SSS* (Huang &amp; Wang, 2018)(GAL) -0.33%</td><td>4.99M</td><td>66.7%</td><td colspan="2">199.93M 36.3%</td></tr><tr><td colspan="2">SSS* (Huang &amp; Wang, 2018)(GAL) -0.94%</td><td>3.93M</td><td>73.8%</td><td colspan="2">183.13M 41.6%</td></tr><tr><td>COP (Wang et al., 2019)</td><td>-0.25%</td><td>1.44M</td><td>92.8%</td><td>211.2M</td><td>73.5%</td></tr><tr><td>ResNet-56(Our baseline)</td><td>93.25%</td><td>0.85M</td><td>0%</td><td>0.25G</td><td>0%</td></tr><tr><td>CV-40(Ours)</td><td>-0.03%</td><td>0.34M</td><td>60.0%</td><td>-</td><td>-</td></tr><tr><td>CV-80(Ours)</td><td colspan="2">+0.26% 0.79M</td><td>7.0%</td><td>-</td><td>-</td></tr><tr><td>L 1 norm (Li et al., 2017)</td><td>0.02%</td><td>0.73M</td><td>13.7%</td><td>90.9M</td><td>27.6%</td></tr><tr><td>GAL-0.6 (Lin et al., 2019)</td><td>0.12%</td><td>0.75M</td><td>11.8%</td><td>78.30M</td><td>37.6%</td></tr><tr><td>GAL-0.8 (Lin et al., 2019)</td><td>-1.68%</td><td>0.29M</td><td>65.9%</td><td>49.99M</td><td>60.2%</td></tr><tr><td>NISP (Yu et al., 2018)</td><td>-0.03%</td><td>0.49M</td><td colspan="2">42.60% 70.49M</td><td>43.61%</td></tr><tr><td>CP (He et al., 2017)</td><td>-1.0%</td><td>-</td><td>-</td><td>62.5M</td><td>50%</td></tr><tr><td>ADC (He &amp; Han, 2018)</td><td>-0.9%</td><td>-</td><td>-</td><td>62.5M</td><td>50%</td></tr><tr><td>PP-1 (Singh et al., 2019)</td><td>-0.03%</td><td>0.06M</td><td>92.5%</td><td>21.5M</td><td>82.8%</td></tr><tr><td>PP-2 (Singh et al., 2019)</td><td>-0.14%</td><td>0.05M</td><td>94.3%</td><td>19.4M</td><td>84.5%</td></tr><tr><td>ResNet-110(Our baseline)</td><td>94.07%</td><td>1.73M</td><td>0%</td><td>0.51G</td><td>0%</td></tr><tr><td>CV-40(Ours)</td><td>0.16%</td><td>0.47M</td><td>72.8%</td><td>-</td><td>-</td></tr><tr><td>L 1 norm (Li et al., 2017)</td><td>-0.23%</td><td>1.16M</td><td>32.4%</td><td>155M</td><td>38.6%</td></tr><tr><td>GAL-0.1 (Lin et al., 2019)</td><td>0.09%</td><td>1.65M</td><td>4.1%</td><td>205.7M</td><td>18.7%</td></tr><tr><td>GAL-0.5 (Lin et al., 2019)</td><td>-0.76%</td><td>0.95M</td><td>44.8%</td><td>130.2M</td><td>48.5%</td></tr><tr><td>NISP (Yu et al., 2018)</td><td>-0.18%</td><td>0.98M</td><td colspan="2">43.25% 142M</td><td>43.78%</td></tr><tr><td>DenseNet-40(Our baseline)</td><td>94.35%</td><td>1.07M</td><td>0%</td><td>0.57G</td><td>0%</td></tr><tr><td>CV-40(Ours)</td><td>-0.23%</td><td>0.31M</td><td>71.3%</td><td>-</td><td>-</td></tr><tr><td>CV-40-lr(Ours)</td><td>-0.62%</td><td>0.17M</td><td>84.2%</td><td>-</td><td>-</td></tr><tr><td>GAL-0.05 (Lin et al., 2019)</td><td>-0.31%</td><td>0.45M</td><td>56.7%</td><td colspan="2">128.11M 54.7%</td></tr><tr><td>GAL-0.1 (Lin et al., 2019)</td><td>-1.58%</td><td>0.26M</td><td>75.0%</td><td>80.89M</td><td>71.4%</td></tr><tr><td colspan="2">network slimming (Liu et al., 2017) 0.92%</td><td>0.66M</td><td>35.7%</td><td>381M</td><td>28.4%</td></tr><tr><td colspan="2">network slimming (Liu et al., 2017) 0.92%</td><td>0.35M</td><td>65.2%</td><td>240M</td><td>55.0%</td></tr><tr><td>Synapse Pruning (Lin et al., 2018)</td><td>-0.34%</td><td>0.21M</td><td>80.4%</td><td>71M</td><td>74.82%</td></tr><tr><td colspan="3">4.2 EXPERIMENTAL ENVIRONMENT AND CONFIGURATION</td><td/><td/><td/></tr></table>

Ablation experiment for different start epoch of pruning on CIFAR-10 with VGG-16. Start Epoch means the epoch that begin to prune. Iter means one alternate training in training set and test set. P Paras and P rate mean the number of pruned parameters in the iteration and the percentage of pruned parameter. Acc means accuracy rate.

Table content: <table><tr><td colspan="4">Start Epoch Iter P Paras P rate</td><td colspan="2">Test Acc Valid Acc</td></tr><tr><td/><td>1</td><td>6.19M</td><td colspan="2">41.3% 91.02%</td><td>90.78%</td></tr><tr><td>10 epoch</td><td>2 3</td><td colspan="3">12.74M 85.0% 91.83% 13.63M 90.9% 92.48%</td><td>91.60% 91.68%</td></tr><tr><td/><td>4</td><td colspan="4">13.95M 93.1% 92.53% 91.80%</td></tr><tr><td/><td>1</td><td>2.88M</td><td colspan="2">19.2% 91.01%</td><td>91.32%</td></tr><tr><td>20 epoch</td><td>2 3</td><td colspan="3">8.65M 11.31M 75.5% 92.49% 57.7% 91.90%</td><td>91.86% 91.88%</td></tr><tr><td/><td>4</td><td colspan="4">12.18M 81.3% 92.60% 92.10%</td></tr><tr><td/><td>1</td><td>2.13M</td><td colspan="2">14.2% 91.10%</td><td>90.66%</td></tr><tr><td>30 epoch</td><td>2 3</td><td colspan="3">8.55M 11.28M 75.3% 92.11% 57.0% 92.20%</td><td>91.70% 92.30%</td></tr><tr><td/><td>4</td><td colspan="4">12.19M 81.3% 92.61% 92.64%</td></tr><tr><td/><td>1</td><td>0M</td><td>0%</td><td>91.01%</td><td>91.18%</td></tr><tr><td>baseline</td><td>2 3</td><td>0M 0M</td><td>0% 0%</td><td>91.98% 92.55%</td><td>91.98% 92.16%</td></tr><tr><td/><td>4</td><td>0M</td><td>0%</td><td colspan="2">92.42% 92.00%</td></tr></table>

Ablation experiment for different number of start pruning epoches on CIFAR-10 with ResNet-56 and ResNet-110. Model-Start Epoch means model and the number of start pruning epoches with the same meaning as table 2. -B means the baseline without pruning. P Paras and P rate means the number of the pruned parameter and the the percentage of pruned parameter. Test Acc means the accuracy rate.

Table content: <table><tr><td colspan="4">Model-Start Epoch P Paras P rate Test Acc</td></tr><tr><td>ResNet-56-10</td><td>0.27M</td><td colspan="2">32.2% 92.39%</td></tr><tr><td>ResNet-56-B</td><td>0</td><td>0%</td><td>92.58%</td></tr><tr><td>ResNet-110-10</td><td>0.72M</td><td colspan="2">41.4% 92.70%</td></tr><tr><td>ResNet-110-B</td><td>0</td><td>0%</td><td>92.57%</td></tr><tr><td colspan="2">4.3 EXPERIMENTAL RESULTS AND ANALYSIS</td><td/><td/></tr></table>

with different learning rate of hyperparameters. We only list the data at the end of each iteration. Left: With the LR=0.00001, the test accuracy rate of DenseNet-40 is steady increasing. When the learning rate increases, there is a downward trend in test accuracy rate. Right: The larger the learning rate of hyperparameters is, the sparser the model obtained in the same iteration process is.Results on CIFAR-10 We compare our results with state-of-art methods in Table1. For VGG-16, our algorithm(CV-40) achieves the highest compression rate of 97.7% with only 0.26% accuracy loss, which is higher than COP in prune rate and 0.01 lower than COP in accuracy. SSS* means it is reimplemented by GAL(Lin et al., 2019). Other results in Table1are copy from their original paper. "-40" means the begining of prune epoch is 40 in each iteration and the total epoch number in each iteration is 160.

Table content: <table><tr><td/><td>80</td><td/><td colspan="4">VGG on CIFAR100 for Different Start Epoch</td><td/><td/></tr><tr><td/><td>70</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>(%)</td><td>60</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>accuracy</td><td>40 50</td><td/><td/><td/><td/><td colspan="3">st=1/4 iter: pr=90.6%, test acc=72.79%</td></tr><tr><td>test</td><td>30</td><td/><td/><td/><td/><td colspan="3">st=2/4 iter: pr=60.1%, test acc=72.36% st=3/4 iter: pr=54.9%, test acc=72.36%</td></tr><tr><td/><td>20</td><td/><td/><td/><td/><td colspan="3">baseline: pr=0.0%, test acc=71.80%</td></tr><tr><td/><td>10</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0</td><td>100</td><td>200</td><td>300</td><td>epoch</td><td>400</td><td>500</td><td>600</td><td>700</td></tr><tr><td colspan="10">Figure 3: The performance of VGG-16 on CIFAR-100 with different starting pruning epoch. "st"</td></tr><tr><td colspan="10">means starting pruning epoch and the fraction (such as 1/4, 2/4) means the ratio of the initial pruning</td></tr><tr><td colspan="10">epoch to the total epoch in an iteration. "iter" meaans one iteration. "pr" is the pruning rate and "test</td></tr><tr><td colspan="4">acc" means the accuracy rate in test set.</td><td/><td/><td/><td/><td/></tr></table>

Peer review (iclr2020):
Summary: This paper presents a method for automatically tuning the L1 and L2 penalty terms of sparsity-inducing deep network training so as to maximize cross-validation performance. The basic idea is to train a network with L1 and L2 regularization terms, and then use implicit differentiation to get the gradient of a cross-validation loss with respect to the corresponding lambda_1 and lambda_2 hyperparameters on these terms (since the L1 term is non-differentiable they use the standard optimization formulation to derive the gradients via implicit differentiation). They show that the method Comments: Overall I very much like the idea of using implicit differentiation of a cross-validation loss for automatic hyperparameter tuning, though this has been done in a very similar fashion by past work (e.g. in the cited Maclaurin et al., though see also e.g. paper). The issue I have with this paper, though, is that neither the method nor results are sufficient to convince me that this approach provides either a theoretical or practical improvement in this case. On the more mathematical/algorithmic side, the formulas for implicit differentiation only hold when W has reached an actual minima, which is almost certainly not the case in this setting. Furthermore, the authors eventually just use an extremely crude approximation of the actual implicit derivative, replacing the Hessian with the identity, so it*s unclear what their *gradient* term really even corresponds to here. On the practical side, since the inner loop optimization needs to run a fair amount before the gradient can be computed, and since there are only two hyperparameters of interest (and it*s not clear to me we even need the L2 term), it wasn*t clear whether there was any real speed benefit to the proposed approach over simple grid search (and this speed benefit is really the only potential advantage of a gradient based approach over grid search). No explicit timing results are presented in the paper, but the number of epochs used e.g. for CIFAR-100 are _well_ above what*s needed for training a single model. Finally, in terms of the experiments, it*s not at all clear to me the methods are outperforming the other approaches. The authors bold their entries, but there is no metric in which they are better than all the others, and they could very well just indicate different points on the Pareto frontier. There are also several typos in the paper, such as referring to the Hessian matrix as the *Heisen* and non-differentiability as *non-derivability*.