Title: DEEP-TRIM: REVISITING L1 REGULARIZATION FOR CONNECTION PRUNING OF DEEP NETWORK Anonymous

Abstract: State-of-the-art deep neural networks (DNNs) typically have tens of millions of parameters, which might not fit into the upper levels of the memory hierarchy, thus increasing the inference time and energy consumption significantly, and prohibiting their use on edge devices such as mobile phones. The compression of DNN models has therefore become an active area of research recently, with connection pruning emerging as one of the most successful strategies. A very natural approach is to prune connections of DNNs via 1 regularization, but recent empirical investigations have suggested that this does not work as well in the context of DNN compression. In this work, we revisit this simple strategy and analyze it rigorously, to show that: (a) any stationary point of an 1 -regularized layerwisepruning objective has its number of non-zero elements bounded by the number of penalized prediction logits, regardless of the strength of the regularization; (b) successful pruning highly relies on an accurate optimization solver, and there is a trade-off between compression speed and distortion of prediction accuracy, controlled by the strength of regularization. Our theoretical results thus suggest that 1 pruning could be successful provided we use an accurate optimization solver. We corroborate this in our experiments, where we show that simple 1 regularization with an Adamax-L1(cumulative) solver gives pruning ratio competitive to the state-of-the-art.

1. INTRODUCTION:

State-of-the-art Deep Neural Networks (DNNs) typically have millions of parameters. For example, the VGG-16 network (Simonyan and Zisserman (2014) ), from the winning team of ILSVRC-2014, contains more than one hundred million parameters; inference with this network on a single image takes tens of billions of operations, prohibiting its use on edge devices such as mobile phones or in real-time applications. In addition, the huge size of DNNs often precludes them from being placed at the upper level of the memory hierarchy, with resulting slow access times and expensive energy consumption.

A recent thread of research has thus focused on the question of how to compress DNNs. One successful approach that has emerged is to trim the connections between neurons, which reduces the number of non-zero parameters and thus the model size (Han et al. (2015a; b) ; Guo et al. (2016) ; Molchanov et al. (2017) ; Aghasi et al. (2017) ; Dong et al. (2017) ; Tung and Mori (2018) ). However, there has been a gap between the theory and practice: the trimming algorithms that have been practically successful (Han et al. (2015a; b) ; Guo et al. (2016) ) do not have theoretical guarantees, while theoretically-motivated approaches have been less competitive compared to the heuristics-based approaches Aghasi et al. (2017) , and often relies on stringent distributional assumption such as Gaussian-distributed matrices which might not hold in practice. With a better theoretical understanding, we might be able to answer how much pruning one can achieve via different approaches on different tasks, and moreover when a given pruning approach might or might not work. Indeed, as we discuss in our experiments, even the generally practically successful approaches are subject to certain failure cases. Beyond simple connection pruning, there have been other works on structured pruning that prune a whole filter, whole row, or whole column at a time (Anwar et al. (2017) ; Alvarez and Salzmann (2016) ; Wen et al. (2016) ; Li et al. (2016) ; Luo and Wu (2017) ). The structured pruning strategy can often speed up inference speed at prediction time more than simple connection pruning, but the pruning ratios are typically not as high as non-structured connection pruning; so that the storage complexity is still too high, so that the caveats we noted earlier largely remain.

A very natural strategy is to use 1 regularized training to prune DNNs, due to their considerable practical success in general sparse estimation in shallow model settings. However, many recent investigations seemed to suggest that such 1 regularization does not work as well with non-shallow DNNs, especially compared to other proposed methods. Does 1 regularization not work as well in non-shallow models? In this work, we theoretically analyze this question, revisit the trimming of DNNs through 1 regularization. Our analysis provides two interesting findings: (a) for any stationary point under 1 regularization, the number of non-zero parameters in each layer of a DNN is bounded by the number of penalized prediction logits-an upper bound typically several orders of magnitude smaller than the total number of DNN parameters, and (b) it is critical to employ an 1 -friendly optimization solver with a high precision in order to find the stationary point of sparse support.

Our theoretical findings thus suggest that one could achieve high pruning ratios even via 1 regularization provided one uses high-precision solvers (which we emphasize are typically not required if we only care about prediction error rather than sparsity). We corroborate these findings in our experiments, where we show that solving the 1 -regularized objective by the combination of SGD pretraining and Adamax-L1(cumulative) yields competitive pruning results compared to the state-ofthe-art.

2. PROBLEM FORMULATION:

Let X (0) : N × D (0) 1 • • • × D (0)

p × K 0 be an input tensor where N is the number of samples (or batch size). We are interested in DNNs of the form

X (j) := σ W (j) (X (j-1) ), l = j . . . J

where σ W (j) (X (j-1) ) are piecewise-linear functions of both the parameter tensor W (j) :

K (j-1) × C (j) 0 • • • × C (j) p × K (j) and the input tensor X (j-1) : N × D (j-1) 1 • • • × D (j-1) p × K (j-1

) of (j)-th layer. Examples of such piecewise-linear function include:

(a) convolution layers with Relu activation (using • to denote the p-dimensional convolution operator)

EQUATION

fully-connected layers with Relu activation

[σ W (X)] i,k := [X i,: W :,k ] + ,

(c) commonly used operations such as max-pooling, zero-padding and reshaping.

Note X (J) : N × K provide K scores (i.e. logits) of each sample that relate to the labels of our target task Y : N × K. Denote L(X (J) , Y ) as the task-specific loss function. We define Support Labels of a DNN X (J) as indices (i, k) of non-zero loss subgradient w.r.t. the prediction logit:

Definition 1 (Support Labels). Let L(X, Y ) be a convex loss function w.r.t. the prediction logits X.

The Support Labels regarding DNN outputs X (J) (W ) are defined as

S(W ) := (i, k) ∈ [N ] × [K] | [Q] i,k = 0, for some Q ∈ ∂ X L(X (J) , Y ) .

We will denote k S (W ) := |S(W )| N ≤ K as the average number of support labels per sample.

We illustrate these concepts in the context of some standard machine learning tasks.

Multiple Regression. In multiple regression, we are interested in multiple real-valued labels, such as the location and orientation of objects in an image, which over the set of N samples, can be expressed as an N × K real-valued matrix Y . A popular loss function for such tasks is: However, smaller λ makes convergence to the stationary point harder for optimization algorithms.

L(X, Y ) := 1 2 X -Y 2 F ,

which is convex and differentiable, and in general we have [∇ X L] i,k = 0, therefore all labels are support labels (i.e. k S = K).

Binary Classification. In binary classification, the labels are binary-valued, and over the set of N samples, can be represented as a binary vector y ∈ {-1, 1} N . Popular loss functions include the logistic loss:

EQUATION

and the hinge loss:

EQUATION

For the logistic loss (1), we have

k S = 1 since [∇L] i = 0, ∀i ∈ [N ].

On the other hand, the hinge loss (2) typically has only a small portion of samples i ∈ [N ] with [∂L] i = {0}, called Support Vectors, and which coincides with our definition of Support Labels in this context. In applications with unbalanced positive and negative examples, such as object detection, we have k S 1. Multiclass/Multilabel Classification. In multiclass or multilabel classification, the labels of each sample can be represented as a K-dimensional binary vector {0, 1} K where 1/0 denotes the presence/absence of a class in the sample. Let P i := {k | y ik = 1} and N i := {k | y ik = 0} denote the positive and negative label sets. Popular loss functions include the cross-entropy loss:

EQUATION

and the maximum margin loss:

EQUATION

Although the cross-entropy loss (3) has number of support labels k S = K, it has been shown that the maximum-margin loss (4) typically has k S K in recent studies of classification problems with extremely large number of classes (Yen et al. (2016; 2017) ).

3. DEEP-TRIM: THEORY:

In this section, we aim to solve the following DNN compression problem. Definition 2 (Deep-Trim ( )). Suppose we are given a target loss function L(X, Y ) between prediction X and training labels Y : N × K, and a pre-trained DNN X (J) parameterized by weights W := {W (j) } J j=1 , with loss L * := L(X (J) (W ), Y ). The Deep-Trim ( ) task is to find a compressed DNN with weights W such that its number of non-zero parameters nnz( W ) ≤ τ , for some τ nnz(W ) and where L(X (J) ( W ), Y ) ≤ L * + for some > 0.

In the following, we show that the Deep-Trim problem with budget τ = (N k S ) × J can be solved via simple 1 regularization under a couple of mild conditions, with the caveat that with suitable optimization algorithms be used, and where k S is the maximum number of support labels for any W with L(

X (L) (W ), Y ) ≤ L * .

Trimming Objective Given a loss function L(., Y ) and a pre-trained DNN parameterized by W * := {W (j) } J j=1 , we initialize the iterate with W * and apply an optimization algorithm that guarantees descent of the following layerwise 1 -regularized objective min

EQUATION

for all j ∈ [J], where vec(W (j) ) denotes the vectorized version of the tensor W (j) .

The following theorem states that most of stationary points of ( 5) have the number of non-zero parameters per layer bounded by the total number of support labels in the training set.

Theorem 1 (Deep-Trim with 1 penalty). Let W (j) be any stationary point of objective (5) with dim(W (j) )=d that lies on a single linear piece of the piecewise-linear function X (J) (W ). Let V : (N K) × d be the Jacobian matrix of that corresponding linear piece of the linear (vector-valued) function vec(X (J) )(vec(W (j) )). For any regularization parameter λ > 0 and V in general position we have

nnz( W (j) ) ≤ N k S ( Ŵ ).

where k S ( W ) is the average number of support labels of the stationary point W (j) .

Proof. Any stationary point of ( 5) should satisfy the condition

V T vec(A) + λρ = 0 (6)

where A ∈ ∂L is an N × K subgradient matrix of the loss function w.r.t. the prediction logits, and (j) )] r = 0} be the set of indices of non-zero parameters, we have [ρ] r ∈ {-1, 1} and thus the linear system

ρ ∈ ∂ vec(W (j) ) 1 is a d-dimensional subgradient of the 1 norm penalty. Then let Q := {r | [vec(W

EQUATION

cannot be satisfied if nnz(A) < nnz(W (j) ) for V is in general position (as defined in, for example, Tibshirani et al. ( 2013)). Therefore, we have nnz

(A) = N k S ≥ nnz(W (j) ) = |Q|.

Note the concept of general position is studied widely in the literature of LASSO and sparse recovery, and it is a weak assumption in the sense that any matrix drawn from a continuous probability distribution is in general position (Tibshirani et al. ( 2013)).

Figure 1 illustrates an example of a regression task where, no matter how small λ > 0 is, the second coordinate is always 0 at the stationary point. Note since Theorem 1 holds for any λ > 0, one can guarantee to trim a DNN without hurting the training loss by choosing an appropriately small λ, as stated by the following corollary.

Corollary 1 (Deep-Trim without Distortion). Given a DNN with weights W and with loss L * := L(X (J) (W ), Y ), for any > 0, one can find weights W with L(X (J) ( W ), Y ) ≤ L * + , and nnz( W (j) ) ≤ J N k S , where k S is a bound on the number of support labels of parameters W with loss no more than L * + .

Proof. By choosing λ ≤ /(J vec(W (j) ) 1 ), any descent optimization algorithm can guarantee to find Ŵ (j) with λ vec( Ŵ (j) ) 1 + L(X (J) (W (1) , . . . , Ŵ (j) , . . . , W (J) ), Y ) ≤ λ vec(W (j) ) 1 + L * by minimizing (5), which guarantees L(X (J) (W (1) , . . . , Ŵ (j) , . . . , W (J) ), Y ) ≤ L * + /J and nnz(W (j) ) ≤ N k S . Then by applying the procedure for each layer j ∈

[J], one can obtain Ŵ with L(X (J) ( Ŵ ), Y ) ≤ L * + and nnz( Ŵ ) ≤ (N k S ) × J.

In practice, however, the smaller λ, the harder for the optimization algorithm to get close to the stationary point, as illustrated in the figure 1 . Therefore, it is crucial to choose optimization algorithms targeting for high precision for the convergence to the stationary point of (5) with sparse support, while the widely-used Stochastic Gradient Descent (SGD) method is notorious for being inaccurate in terms of the optimization precision.

4. DEEP-TRIM: ALGORITHMS:

Although our analysis is conducted on the layerwise pruning objective (5), in practice we have observed joint pruning of all layers to be as effective as layerwise pruning. For ease of presentation of this section, we will denote our objective function min

EQUATION

in the following form min

EQUATION

where w := vec(W ) and f (w) := L(X (J) (W ), Y ). Note the same formulation ( 9) can be also used to represent the layerwise pruning objective (5) by simply replacing their definitions as w := vec(W (j) ) and f (w) := L(X (J) (W (1) , . . . , W (j) , . . . , W (J) ), Y ).

As mentioned previously, even when the stationary point of an objective has sparse support, if the optimization algorithm does not converge close enough to the stationary point, the iterates would still have very dense support. In this section, we propose a two-phase strategy for the non-convex optimization problem (9). In the first phase, we initialize with the given model and use a simple Stochastic Gradient Descent (SGD) algorithm to optimize (9). During this phase, we do not aim to reduce the number of non-zero parameters but only to reduce the 1 norm of the model parameters.

We run the SGD till both the training loss and 1 norm of model parameters have converged. Then in the second phase, we employ an Adamax-L1 (cumulative) method to reduce the total number of non-zero parameters, and achieves pruning result on-par with state-of-the-art methods.

SGD with L1 Penalty For a simple optimization problem min w∈R d f (w), the SGD update follows the form w t+1 = w t -η t ∂f (w)

∂w . We consider general SGD-like algorithms which update in the form w t+1 = w t -η t g( ∂f (w) ∂w , θ), where θ is a set of parameters specific to the SGD-like update procedure. This includes the commonly used Momentum (Qian (1999)), Adamax, Adam (Kingma and Ba ( 2014)), and RMSProp (Tieleman and Hinton ( 2012)) optimization algorithms.

When employing SGD-like optimizers, (9) can be rewritten as the following:

EQUATION

where j denotes one mini-batch of data and N b is the number of mini-batches. The weight updated by the SGD-like optimizers can then be performed as

EQUATION

where sign(w i ) = 0 when w i = 0. We note that after the update in (11), the weight does not become 0 unless w t i = η t • g( ∂f (w,j) ∂wi + λ N b sign(w t i ), which rarely happens. Therefore, adding the L1 penalty term to SGD-like optimizers only minimizes the L1-norm but does not induce a sparse weight matrix. To achieve a sparse solution, we combine the L1 friendly update trick SGD-L1 (cumulative) (Tsuruoka et al. (2009) ) along with SGD-like optimization algorithms.

Adamax-L1 (cumulative) SGD-L1 (clipping) is an alternative to perform L1 regularizing along with SGD to obtain a sparse w (Carpenter (2008) ). Different to (11), SGD-L1 (clipping) divides the update into two steps. The first step is updated without considering the L1 penalty term, and the second step updates the L1 penalty separately. In the second step, any weight that has changed its sign during the update will be set to 0. In other words, when the L1 penalty is larger than the weight value, it will be truncated to the weight value. Therefore, SGD-L1 (clipping) can be seen as a special case of truncated gradient. With a learning rate η k , the update algorithm can be written as

EQUATION

SGD-L1 (cumulative) is a modification of the SGD-L1 (clipping) algorithm proposed by Tsuruoka et al. (2009) , but uses the cumulative L1 penalty instead of the standard L1 penalty. The intuition is that the cumulative L1 penalty is the amount of penalty that would be applied on the weight if true gradient is applied instead of stochastic gradient. By applying the cumulative L1 penalty, the weight would not be moved away from zero by the noise of the stochastic gradient. When applied to SGD-like optimization algorithms, the update rule can be written as

EQUATION

where q k i is the total amount of L1 penalty received until now

q k i = k t=1 (w t+1 i -w t+ 1 2 i

).

By updating with ( 13) and adopting the Adamax optimization algorithm (Kingma and Ba ( 2014)), we obtain Adamax-L1 (cumulative). Originally, SGD-L1 (cumulative) was proposed to be used with the vanilla SGD optimizer, where we generalize it to be used with any SGD-like optimizer by separating the update on objective f (w) and the l1-cumulative update on λ w 1 .

5. EXPERIMENTS:

In this section, we compare the -regularized pruning method discussed in section 4 with other state-of-the-art approaches. In section 5.1, we evaluate different pruning methods on the convolution network LeNet-51 on the Mnist data set. In section 5.2, we compare our method to VD on pruning VGG-16 network Simonyan and Zisserman (2014) on the CIFAR-10 data set. In section 5.3, we then conduct experiments with Resnet on CIFAR-10. Finally, we show the trade-off for pruning Resnet-50 on the ILSVRC dataset. 

5.1. LENET-5 ON MNIST:

We first compare our methods with other compression methods on the standard MNIST dataset with the LeNet-5 architecture. We consider the following methods: Prune: The pruning algorithm proposed in Han et al. (2015a) , which iterates between pruning the network after training with L2 regularization and retraining. DNS: Dynamic Network Surgery pruning algorithm proposed in Guo et al. (2016) , which was reported to improve upon the iterative pruning method proposed in Han et al. (2015a) by dynamically pruning and splicing variables during the training process. VD: Variational Dropout method introduced by Molchanov et al. (2017) , a variant of dropout that induces sparsity during the training process with unbounded dropout rates. L1 Naive: Ablation study of our method by training the 1 -regularized objective with SGD. Ours: Our method which optimizes the 1 -regularized objective (8) in two phases (SGD and Adamax-L1(cumulative)). The LeNet-5 network is trained from a random initialization and without data augmentation which achieves 99.2% accuracy. We report the per layer sparsity and the total reduction of weights and FLOP in Table 1 . For LeNet-5, our method achieves comparable sparsity performance against other methods, with a slight accuracy drop. Nevertheless, our compressed model still achieves over 99 percent testing accuracy, while achieving 260× weight reduction and 84× FLOP reduction. We also observe that the L1 Naive does not induce any sparsity, even when the L1-norm is significantly reduced. This demonstrates the effectiveness of adopting a L1-friendly optimization algorithm.

5.2. VGG-LIKE ON CIFAR-10:

To test how our method works on large scale modern architecture, we perform experiments on the VGG-like network with CIFAR-10 dataset, which is used in Molchanov et al. (2017) . The network contains 13 convolution layers and 2 fully connected layers and achieves 92.9% accuracy with pretraining. We report the per layer weights and FLOP reduction for our Deep-Trim algorithm and VD (Molchanov et al. (2017) ) in Table 5 .

Our model achieves a weight pruning ratio of 57× and reduces FLOP by 7.7× with a negligible accuracy drop, and VD achieves 48× weight pruning ratio and reduces FLOP by 6.4×.2 Compared to VD, our model achieved sparser weights from Conv1_1 to Conv5_2 and VD achieved sparser weights from Conv5_2 to FC layers. Interestingly, we observe that in both pruning methods, most remaining nnz and FLOPs lie in block2 and block3, where originally block4 and block5 have dominating amount of weights and equal amount of FLOPs.

The layer with the most non-zero parameters after pruning is conv3_2 with 65.9K. In the experiments we employ the cross-entropy loss (3) which has a number of support labels N K = 500K on the CIFAR-10 data set. We suspect a more careful analysis could improve our Theorem 1 to give a tighter bound for loss with entries of gradient close to 0 but not exactly 0, making the bound for cross-entropy loss (3) closer to that of maximum-margin loss (4).

5.3. RESNET-32 ON CIFAR-10:

While VGG-network are notorious for its large parameter size, it is not surprising that a large compression rate can be achieved. Therefore, we evaluate the compression performance of our Deep-Trim algorithm on a smaller Resnet-32 model trained on CIFAR-10 data. The Resnet-32 model contains 3 main blocks. The first block contains the first 11 convolution layers with 64 filters in each layer, the second block contains the next 10 convolution layers with 128 filters each, and the last block contains 10 convolution layers with 256 filters and a fully connected layer. We list the detailed architecture in the supplementary. The pretrained Resnet-32 model reaches 94.0% accuracy.

We evaluate our Deep-Trim algorithm and compare it to variational dropout (Molchanov et al. (2017) ) and report the results in Table 3 . We report the pruning results for each main block of the resnet-32 model. Our model achieves a 33× overall pruning ratio and 21× reduced FLOP with an accuracy drop of 1.4%, where VD has attained 28× overall pruning ratio and 13.5× reduction with similar accuracy. We further observe that nnz(W) increases much gentler from the first block to the third block compared to the total number of parameters in each block. This is not surprising since the upper bound of nnz(W) per layer given by Corollary 1 does not depend on the total number of unpruned parameters. 

Method:



5.4. EFFECT OF DATA ON PRUNING:

In this section, we compare the pruning results of our method on VGG-16 with different number of samples. The pruning ratio and number of non-zero parameters are shown in Table 4 , we can see that the number of non-zero parameters after pruning clearly grows with the number of samples. This can be understood intuitively, as the number of constraints to be satisfied grows in the training set, the more degree of freedom the model needs to fit the data. This shows that our theory analysis matches our empirical results well. 

6. CONCLUSION:

In this work, we revisit the simple idea of pruning connections of DNNs through 1 regularization.

While recent empirical investigations suggested that this might not necessarily achieve high sparsity levels in the context of DNNs, we provide a rigorous theoretical analysis that does provide small upper bounds on the number of non-zero elements, but with the caveat that one needs to use a high-precision optimization solver (which is typically not needed if we care only about prediction error rather than sparsity). When using such an accurate optimization solver, we can converge closer to stationary points than traditional SGD, and achieve much better pruning ratios than SGD, which might explain the poorer performance of 1 regularization in recent investigations. We perform experiments across different datasets and networks and demonstrate state-of-the-art result with such simple 1 regularization.

:

We use the modified version of LeNet5 from(LeCun et al., 1998) with Caffe Model specification: https: //goo.gl/4yI3dL .

We ran the experiments based on authors' code and tuned the coefficient of dropout regularization loss within the interval [10 2 , 10 -3 ] with binary search. We note that although we are able to reproduce the 48× weight reduction ratio in the VD paper, we are only able to achieve Acc. 92.2% instead of 92.7% as reported in their paper.

Figures and Tables:

Figure 1: The level curves of the 1 -regularized objective with 2 parameters and 1 sample: λ w 1 + 1 2 (x T w -1) 2 where x = [0.6, 0.4]. Note that the stationary points have w 2 = 0 as long as λ > 0. However, smaller λ makes convergence to the stationary point harder for optimization algorithms.

Compression Results with LeNet-5 model on MNIST.

Table content: <table><tr><td>Method</td><td>Acc.%</td><td>nnz per Layer%</td><td>W W =0</td><td>FLOP%</td></tr><tr><td>Prune</td><td>99.2</td><td>68 -12 -8.0 -19.0</td><td>12</td><td>16</td></tr><tr><td>DNS</td><td>99.09</td><td>14 -3.1 -0.7 -4.3</td><td>111</td><td>-</td></tr><tr><td>VD</td><td>99.25</td><td>33 -2.0 -0.2 -5.0</td><td>280</td><td>-</td></tr><tr><td>L1 Naive</td><td>99.25</td><td>100 -100 -100 -100</td><td>1</td><td>100</td></tr><tr><td>Ours</td><td>99.05</td><td>20 -1.9 -0.2 -3.2</td><td>260</td><td>1.2</td></tr></table>

Compression Results with VGG-like model on CIFAR-10 for VD and our method.

Table content: <table><tr><td>Models</td><td colspan="2">Base Model</td><td>VD</td><td/><td>Ours</td><td/></tr><tr><td>Layers</td><td>Weights</td><td>FLOP</td><td colspan="4">Weights% FLOP% Weights% FLOPS%</td></tr><tr><td>Conv1_1</td><td>1.7K</td><td>716K</td><td>67.8</td><td>65.8</td><td>38.4</td><td>38.4</td></tr><tr><td>Conv1_2</td><td>36.8K</td><td>17.3M</td><td>25.1</td><td>39.3</td><td>16.8</td><td>26.6</td></tr><tr><td>Conv2_1</td><td>73.7K</td><td>11.7M</td><td>33.8</td><td>28.9</td><td>27.5</td><td>28.4</td></tr><tr><td colspan="2">Conv2_2 147.5K</td><td>15.5M</td><td>27.6</td><td>28.3</td><td>24.3</td><td>25.2</td></tr><tr><td colspan="2">Conv3_1 294.9K</td><td>7.9M</td><td>22.1</td><td>20.9</td><td>20.0</td><td>20.4</td></tr><tr><td colspan="2">Conv3_2 589.8K</td><td>8.7M</td><td>11.5</td><td>11.1</td><td>11.2</td><td>11.7</td></tr><tr><td colspan="2">Conv3_3 589.8K</td><td>7.8M</td><td>8.6</td><td>7.8</td><td>6.0</td><td>5.1</td></tr><tr><td>Conv4_1</td><td>1.2M</td><td>6.4M</td><td>2.3</td><td>0.83</td><td>1.4</td><td>0.75</td></tr><tr><td>Conv4_2</td><td>2.4M</td><td>8.8M</td><td>0.65</td><td>0.04</td><td>0.22</td><td>0.13</td></tr><tr><td>Conv4_3</td><td>2.4M</td><td>13.4M</td><td>0.23</td><td>0.03</td><td>0.11</td><td>0.11</td></tr><tr><td>Conv5_1</td><td>2.4M</td><td>6.3M</td><td>0.07</td><td>0.02</td><td>0.04</td><td>0.04</td></tr><tr><td>Conv5_2</td><td>2.4M</td><td>5.7M</td><td>0.06</td><td>0.06</td><td>0.04</td><td>0.04</td></tr><tr><td>Conv5_3</td><td>2.4M</td><td>6.7M</td><td>0.05</td><td>0.04</td><td>0.07</td><td>0.07</td></tr><tr><td>FC6</td><td>262.1K</td><td>239.0K</td><td>0.79</td><td>0.2</td><td>2.6</td><td>2.7</td></tr><tr><td>FC7</td><td>5.1K</td><td>1.0K</td><td>41.8</td><td>57.3</td><td>88.1</td><td>75.3</td></tr><tr><td>Total</td><td>15.0M</td><td>117.3M</td><td>2.1</td><td>15.6</td><td>1.8</td><td>13.0</td></tr><tr><td>W W =0</td><td>1</td><td/><td>47.5</td><td/><td>57.1</td><td/></tr><tr><td>Acc. %</td><td colspan="2">92.9</td><td>92.2  *</td><td/><td>92.8</td><td/></tr></table>

Compression Results with Resnet-32 model on CIFAR-10.

Table content: <table><tr><td/><td>Acc.%</td><td>nnz(W) per block</td><td>W W =0</td><td>FLOP%</td></tr><tr><td>Resnet Ref.</td><td>94.0</td><td>371K-1.4M-5.6M</td><td>1</td><td>100</td></tr><tr><td>VD</td><td>92.5</td><td>37.8K-90.4K-136.9K</td><td>27.6</td><td>7.4</td></tr><tr><td>Ours</td><td>92.6</td><td>34.6K-65.2K-120.9K</td><td>33.3</td><td>4.8</td></tr></table>

Training data Train Acc.% Test Acc.% nnz(W) Compression Results with VGG-like on CIFAR-10 with varying number of training data.

Table content: <table><tr><td>W</td></tr><tr><td>W =0</td></tr></table>

Peer review (iclr2019):
 This paper discusses the effect of L1 penalization for deep neural network. In particular it shows the stationary point of an l1 regularized layer has bounded non-zero elements. The perspective of the proof is interesting: By chain rule, the stationary point satisfies nnz(W^j) linear equations, but the subgradients of the loss function w.r.t. the logits have at most N.times ks variables. If the coefficients of the linear equation are distributed in general positions, then the number of variables should not be larger than the number of equations. While I mostly like the paper, I would like to point out some possible issues: main concerns: 1. the columns of V may not be independent during the optimization(training) process. In this situation, I am not quite sure if the assumption of “general position” still holds. I understand that in literatures of Lasso and sparse coding it is common to assume “general position”. But in those problems the coefficient matrix is not Jacobian from a learning procedure. 2. the claim is a little bit counter intuitive: Theorem 1 claims the sparse inequality holds for any .lambda. It is against the empirical observation that when lambda is extremely small, effect of the regularizer tends to be almost zero. Can authors also show this effects empirically, i.e., when the regularization coefficients decrease, the nnz does not vary much? (Maybe there is some optimization details or approximations I missed?) Some minor notation issues: 1. in theorem 1: dim(W^{(j)})=d should be dim(vec(W^{(j)}))=d 2. in theorem 1: Even though I understand what you are trying to say, I would suggest we describe the jacobian matrix V in details. Especially it is confusing to stack vec(X^J) (vec(W^j)) in the description. 3. the notations of subgradient and gradient are used without claim