Title: Alternating Direction Method of Multipliers for Sparse Convolutional Neural Networks
Authors: Farkhondeh Kiaee
Christian Gagné
and Mahdieh Abbasi
Venue: ICLR 2017
Keywords: Deep learning
Computer vision
Optimization

Abstract: The storage and computation requirements of Convolutional Neural Networks (CNNs) can be prohibitive for exploiting these models over low-power or embedded devices. This paper reduces the computational complexity of the CNNs by minimizing an objective function, including the recognition loss that is augmented with a sparsity-promoting penalty term. The sparsity structure of the network is identified using the Alternating Direction Method of Multipliers (ADMM), which is widely used in large optimization problems. This method alternates between promoting the sparsity of the network and optimizing the recognition performance, which allows us to exploit the two-part structure of the corresponding objective functions. In particular, we take advantage of the separability of the sparsity-inducing penalty functions to decompose the minimization problem into sub-problems that can be solved sequentially. Applying our method to a variety of state-of-the-art CNN models, our proposed method is able to simplify the original model, generating models with less computation and fewer parameters, while maintaining and often improving generalization performance. Accomplishments on a variety of models strongly verify that our proposed ADMM-based method can be a very useful tool for simplifying and improving deep CNNs. 

TL;DR: A method to sparsify (prune) pre-trained deep neural networks.

Review discussion:
1) Can you please add a table that includes datasets/models/methods vs accuracies/sparsities? It would be easier to compare the results.
2)  Does the proposed sparsity-promoting penalty term provide with a more sparse network than by just using L1,L2 regularizer during training?
Thank you for your questions, here are our answers:
1) We prepared and uploaded a new version of the paper with Table 5, which shows joint variations of sparcity and accuracy for increasing \mu, for the various datasets/models/regularization tested, from single runs that were picked randomly. Please check the updated PDF of the paper.
2) The proposed method can be seen as a post-processing approach. It can be applied over a trained network where regularization has been used.
In fact, all of the baseline networks presented in this paper are pre-trained with a cost function that includes a weight decaying L2 penalty term. The results reported with ADMM are in addition to this L2 regularization.
The paper presents a method for sparsifying the weights of the convolutional
filters and fully-connected layers of a CNN without loss of performance. Sparsification is achieved by using augmenting the CNN objective function with a regularization term promoting sparisty on groups of weights. The authors use ADMM for solving this optimization task, which allows decoupling of the two terms. The method alternates between promoting the sparsity of the network and optimizing the recognition performance
The method is technically sound and clearly explained. The paper is well organised and the ideas presented in a structured manner. I believe that sometimes the wording could be improved.
The proposed method is simple and effective. Using it in combination with other CNN compression techniques such as quantization/encoding is a promising direction for future research. The experimental evaluation is convincing in the sense that the method seems to work well. The authors do not use state-of-the-art CNNs architectures, but I don't see this a requirement to deliver the message.
On the other hand, the proposed method is closely related to recent works (as shown in the references posted in the public comment titled "Comparison with structurally-sparse DNNs using group Lasso), that should be cited and compared against (at least at a conceptual level). I will of course consider the authors rebuttal on this matter.
It would be very important for the authors to comment on the differences between these works and the proposed approach. It seems that both of these references use sparsity regularized training for neural networks, with a very similar formulation.
The authors choose to optimize the proposed objective function using ADMM. It is not clear to me, why this approach should be more effective than proximal gradient descent methods. Could you please elaborate on this? ADMM is more demanding in terms of memory (2 copies of the parameters need to be stored).
The claimed contributions (Section 5) seem a bit misleading in my opinion. Using sparsity promoting regularization in the parameters of the model has been used by many works, in particular in the dictionary learning literature but also in the neural network community (as stated above).  Claims 1,2 and 3, are well understood properties of L1-type regularizers. As written in the current version of the manuscript, it seems that these are claimed contributions of this particular work.
A discussion on why sparsity sometimes helps improve performance could be interesting.
In the experimental section, the authors mainly concentrate on comparing accuracy vs parameter reduction.
While this is naturally very relevant property to report, it would be also interesting to show more results in terms of speed-up. which should also be improved by the proposed approach.
Other minor issues:
- In (3): I think a "^2" is missing in the augmented term (the rightmost term).
- The authors could cite the approach by Han et all for compressing DNNS:
Han, ICLR 2016 "Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding"
Thank you for your careful and thoughtful review, that's really appreciated.
[Comparison with group lasso work]: Compared to the closely related Sparse Structure Learning (SSL) method, which learns sparse block structures (e.g. sparse filters) and minimizes the classification error simultaneously, our proposed approach uses ADMM method to provide a separate scheme to optimize the sparse blocks and classification error. ADMM brings the differentiability and separability advantages to the proposed sparse CNN method which are the basis of the distinctive contributions of our proposed approach compared to SSL method. The algorithm has the advantage that it is partially and analytically solvable due to the separability property. This contributes to the efficient trainability of the model.
Moreover, the differentiability problem of L0-norm penalty function makes it unusable in the SSL framework, while L0-norm can be incorporated as a mean of sparsity penalty terms in our proposed method.
[Effectiveness of ADMM]: The ADMM method breaks the minimization of the augmented Lagrangian function into two parts which brings the differentiability and separability advantages to the proposed method (claims 1 and 2 in the Discussion section).
Although during the training phase ADMM seems to be more demanding in terms of memory (2 copies of the parameters need to be stored), after the convergence only one copy of parameters with sparse structure is retrieved and used in the test phase and the final model benefits from a more compact size (smaller memory footprint).
[Confusion over claimed contributions]: We agree with the reviewer's comment, implying that the statement 1, 2, and 3 may induce some confusion to the readers. Actually, compared against the standard L1 regularization, which is somehow differentiable, the major point is that due to the decoupling property of ADMM, the proposed algorithm makes it possible to use the non-differentiable L0 regularization (differentiability advantage).
Furthermore, the decoupling capability of ADMM algorithm along with the separability of the penalty functions provide analytical solutions to the sparsity-promoting problems (separability advantage). In order to clarify this, we revised the statement in the discussion part to elucidate the notions of the /differentiability/ and the /separability/ of the proposed method.
[Discussion on why sparsity helps improve performance]: The advantage of the complicated models is that they can capture highly non-linear relationship between features and output. The drawback of such large models is that they are prone to capture the noise that does not generalize to new datasets, leading to over-fitting and a high variance. The recently presented compression methods of dense networks without losing accuracy show significant redundancy in the trained model and inadequacy of current training methods to find the existing sparse and compact solutions with better generalization properties. We propose ADMM-based training method as a post-processing step, once a good model has been learned, with the solution gradually moving from the original dense network to the sparse structure of interest, as our emphasis on the sparsity-promoting penalty term is increased.
More specifically, since the target solution is likely to be sparse, we may think that enforcing sparsity right in the beginning, using our proposed method, will provide a way to avoid overfitting for achieving a better performance.
However, increasing more the sparsity strength of the solution may lead to over-smoothing the models and drops in the performance.
Therefore, in practical design of networks, we figured out that this regularization factor should be increased gradually, until the desired balance between performance and sparsity is achieved.
Therefore, from our preliminary experiments, we observed that sparsifying networks gradually and after a good dense network has been learned is better at improving performance. We conjuncture that the learning method may need to have enough room to first explore and find a good position in the search space of possible models, before applying gradual sparsification over it.
Otherwise, a strong sparsification penalty may break the smoothness of the search space, making it much harder to explore sparser solutions in the neighborhood of the current model. Verifying this may be the topic of a future work.
[Report speedup]: We reported speedup as we vary the parameter mu to the results of Tables 3, 4, and 5 in Appendix B. As the reviewer truly noticed, the number of parameters in the network is reduced by the proposed method and in practice, a speedup for all the networks is achieved.
[Cite Han et al. (2016)]: Thank you for suggesting the interesting and relevant paper by Han et al. (2016), which is cited in the latest revision of our paper.
[Correct Eq. 3]: We also edited the augmented Lagrangian equation (3).
We hope you find the modified paper as a better presentation of the work.
This paper reduces the computational complexity of CNNs by minimizing the recognition loss and a sparsity-promoting penalty term together. The authors use ADMM, which is widely used in optimization problems but not used with CNNs before.
The paper has a good motivation and well written. The experiments show that the proposed approach increases the sparsity in a network as well as increases the performance.
As the authors stated, ADMM approach is not guaranteed to converge in non-convex problems. The authors used pre-trained networks to mitigate the problem of trapping into a local optimum.
However, the datasets that are used are very small. It would be good to investigate how the proposed approach works on bigger datasets such as ImageNet.
Authors should compare their results with previous studies that use pruning or sparsity regularizers (Liu et al. (2015); Han et al. (2015); Collins & Kohli (2014)).
In the discussion section, authors stated that the proposed approach is efficient in training because of the separability property. Could you elaborate on that? Lets say this work uses two phases; phase 1 is pre-training a network, phase 2 is using sparsity and performance promoting steps.  Phase 2 also includes fine-tuning the network based on the new sparse structure. How long does phase 2 take compare to phase 1? How many epochs is needed to fine-tune the network?
Thank you for your review.
To answer this point, we added the following text to the revised version of the paper, starting in second paragraph of the Experimental Results section.
--- Begin quoted text ---
Since the regularization factor $\mu$ is selected from gradually increasing values, for the first small values of $\mu$ the selection of long epochs for performance-promoting step (inner loop) and fine-tuning steps is computationally prohibitive and would result in over-fitting.
Instead, we start with one epoch for the first $\mu$ and increase the number of epochs by $\delta$ for the next $\mu$ values up to the $\nu$-th $\mu$ value, after which the number of epochs is limited to $\delta\nu$. We found that $\delta=1$ and $\nu=15$ generally work well in our experiments. We already incorporated the number of training epochs at tables 3, 4, and 5 of Appendix B. If the maximum limiting number of iterations of inner loop is $\xi$ (suggested value of $\xi$=10), the training time of the $\nu$-th $\mu$ value takes a total of $\delta\nu\xi+\delta\nu$ epochs ($\delta\nu\xi$ for performance-promoting step and $\delta\nu$ for fine-tuning) under the worst-case assumption, where the inner loop has not converged and completes only at the $\xi$-th iteration.
--- End quoted text ---
We hope this will answer well the questions of your last paragraph.
This paper presents a framework to use sparsity to reduce the parameters and computations of a pre-trained CNN.
The results are only reported for small datasets and networks, while it is now imperative to be able to report results on larger datasets and production-size networks.
The biggest problem with this paper is that it does not report numbers of inference time and gains, which is very important in production. And similarly for disk pace. Parameters reduction is useful only if it leads to a large decrease in space or inference time.
We thank the reviewer for the comments.
We have tried to evaluate the proposed sparse CNN approach, extensively, which was not an easy task. The proposed scheme has been validated using a variety of network architectures and on three well-known datasets, the CIFAR-10, CIFAR-100, and SVHN datasets, under two different sparsity penalty term L0-norm and L1-norm and a variety of values assigned to the sparsity strength mu. These configurations are coherent to what many other current work on sparsifying deep neural networks are reporting. Realistically, we think that such an extensive experimental assessment can hardly be done on much more complex datasets and models in a timely manner. We prefer to be more systematic in validating the method than trying to report results on only a handful set of configurations with larger models and datasets, where we would encounter the risk of reporting only anecdotal results.
We also added more results in terms of speedup changes as we vary the parameter mu to the results of Tables 3, 4, and 5 in Appendix B. As the reviewer noticed, in practice, when the network becomes sparser with fewer parameters, its inference time is reduced.
Could the authors elaborate more on point (3) in their conclusion?
Like Reviewer3, I'd like some more intuition as to how this particular point is supposed to be compared against  the more standard L-1 regularization, which is likewise differentiable.  Is the claim that this algorithm is sort of like a differentiable version of L-0 regularization?
Also, could the authors speculate more about why some of the models become better with more sparsity?  Are there enough experiments to determine if this is just a statistical fluke, or is this more of a general observation?  Or perhaps this is just an artifact of the sparsity allowing for longer training times?  Numerics would be nice here, but really I'm just curious.
Edit: Because I was added late as a reviewer, if the authors could provide an answer by Thursday 1/5/2017, I will incorporate that into my review.
Otherwise, I'll post a full review on Thursday (but will still, of course, consider further comments by the authors in updates to that review before the deadline).
[Elaborate more on point (3) of the conclusion]: Indeed, the ADMM method breaks the minimization of the augmented Lagrangian function into two parts, which brings the differentiability advantage to the proposed method for loss minimization, while making possible the use of L0 regularization.
[Statistical significance of generalization improvements with more sparsity]: In order to demonstrate that the ADMM training performance is statistically significant, we did a t-test by repeating the experiment 15 times on CIFAR-10 using the NIN model. The results demonstrate that ADMM training achieves significant improvements from both the baseline model (t-test result with p<0.001) and standard fine tuning (t-test result with p<0.001). The t-test results are given in the new appendix A.
The authors use the Alternating Direction Method of Multipliers (ADMM) algorithm for the first time on CNN models, allowing them to perform model compression without any appreciable loss on the CIFAR-10, CIFAR-100, and SVHN tasks.  The algorithmic details and the intuition behind their algorithm are generally well presented, (although there are occasional typos).
Pros:
1) Put an old algorithm to good use in a new setting
2) The algorithm has the nice property that it is partially analytically solvable (due to the separability property the authors mention).  This contributes to the efficient trainability of the model
3) Seems to dovetail nicely with other results to encourage sparsity--that is, it can be used simultaneously--and is quite generalizable.
Cons:
1) It would be nice to see a more thorough analysis of the performance gains for using this method, beyond raw data about % sparsity--some sort of comparison involving training time would be great, and is currently lacking. EDIT: Authors addressed this by addition of results in Appendix B.
2) I would very much like to see some discussion about why the sparsity seems to *improve* the test performance, as mentioned in my previous comment.  Is this a general feature?  Is this a statistical fluke? etc.
Even if the answer is "it is not obvious, and determining why goes outside the scope of this work", I would like to know it!  EDIT: Authors addressed this by addition of statistical significance tests in the new Appendix A.
3) Based on the current text, and some of the other reviewer comments, I would appreciate an expanded discussion on how this work compares with other methods in the field.  I don't think a full numerical comparison is necessary, but some additional text discussing some of the other papers mentioned in the other reviews would greatly benefit the paper.  EDIT: Authors addressed this by followup to question and additional text in the paper.
Additional comments: If my Cons are addressed, I would definitely raise my score to a 6 or even a 7.  The core of this paper is quite solid, it just needs a little bit more polishing.
EDIT: Score has been updated.
Note: the authors probably meant "In order to verify" in the first sentence of Appendix A.
Thank you very much for your comments.
[More information on training time (cons 1)]: See answers to reviewer 1.
[Why sparsity improves performance (cons 2)]: See the newly added Appendix A, which is reported statistical test values over 15 runs of the approach on NIN model on CIFAR-10. The statistical tests demonstrate that results are statistically significant. We hope these empirical results clarify the question.
[Comparison with other methods (cons 3)]: Standard L1 regularization methods in the literature, like Structural Sparsity Learning (SSL) using group Lasso, suffer from two limitations compared to our proposed method.
First, they rely on a rigid framework that avoids incorporation of non-differentiable penalty functions (e.g., not usable with L0-norm).
Second, they require training the original full model for performance and sparsity with the same optimization method, while our proposed method allows to decompose the corresponding optimization problems into two sub-problems and exploit the separability of the sparsity-promoting penalty functions to find analytical solutions for one of the sub-problems, while using SGD for solving the other.
These two properties are highlighted in the statements 1, 2 at the Discussion section and the following text is also added to that section in order to clarify the distinction between our proposed method and closely related SSL approach (text beginning with "Some methods such as SSL..." up to the end of the Discussion section).
We hope the latest version of the paper will provide you with material that will convince you of the soundness of our proposal.
I greatly appreciate the effort the authors have spent in polishing their manuscript.  I have updated my review in response.
A revision of the paper has been uploaded, following comments and reviews received.

Meta-review:
role: PC
data: comment: This paper studies a sparsification method for pre-trained CNNs based on the alternating direction method of multipliers. 
 The reviewers agreed that the paper is well-written and easy to follow. The authors were responsive during the rebuttal phase and addressed most of the reviewers questions. 
 
 The reviewers, however, disagree with the significance of this work. Whereas R4 is happy to see an established method (ADMM) applied to a nowadays very popular architecture, R1-R3 were skeptical about the scope of the experiments and the usefulness of the sparse approximation. 
 
 Pros:
 - Simple algorithmic description using well-known ADMM method. 
 - Consistent performance gains in small and mid-scale object classification problems. 
 
 Cons:
 - Lack of significance in light of current literature on the topic. 
 - Lack of numerical experiments on large-scale classification problems and/or other tasks.
 - Lack of clarity when reporting speedup gains. 
 
 Based on these assessments, the AC recommends rejection. Since this decision is not 100% aligned 
 with the reviewers, let me expand on the reasons why I recommend rejection. 
 
 This paper presents a sound algorithm that sparsifies the weights of a trained-CNN, and it shows that 
 the resulting pruned network works well, better than the original one. A priori, this is a solid result. 
 My main problem is that this contribution has to be taken in the context of the already large body of 
 literature addressing this same question, starting from the seminal 'Predicting Parameters in Deep Learning', by Misha Denil et al., NIPS 2013, which is surprisingly ignored in the bibliography here; and onwards with 'Exploiting linear structure within convolutional networks for efficient evaluation', Denton et al, NIPS'14 (also ignored) and 'Speeding up convolutional neural networks with low rank expansions', Jadergerg et al., '14. These and many other works culminated in two recent papers, 'Deep Compression', by Han et al, ICLR'16 and 'Learning Structured Sparsity in Deep Neural Networks' by Wen et al, NIPS'16.
 Several reviewers pointed out that comparisons with Wen et al were needed; the authors of the present submission now do cite this work, but do not compare their results quantitatively and do not present the reader with compelling reasons to choose their acceleration instead of Wen et al.'s. Moreover, they mention in the rebuttal that the paper was published after the submission, but the work was available on the arxiv well before the ICLR deadline (august 2016). Wen's paper presents (i) source code, (ii) experiments on imagenet, (iii) 3x to 5x speedups in both CPU and GPU, and (iv) accuracy improvements on Cifar of ~1.5%. As far as I can see, the present submission presents none of these. Further compression factors were obtained in 'Deep Compression', also in large-scale models. 
 
 In light of these results, the authors should present more compelling evidence (in this case, since this is an experimental paper, empirical evidence in the form of higher accuracy and/or larger speedups/gains) as to why should a practitioner use their model, rather than simply presenting the differences between the approaches.
role: Author
data: value: comment: Well, the decision comment is a review on its own! Although unfortunate to us, we accept the decision. Thank you for the well informed advices and pointers to relevant litterature, it will certainly help us to make the paper stronger and to relate it better with existing work.

Decision: Reject