Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper reduces the computational complexity of CNNs by minimizing the recognition loss and a sparsity-promoting penalty term together. The authors use ADMM, which is widely used in optimization problems but not used with CNNs before.
The paper has a good motivation and well written. The experiments show that the proposed approach increases the sparsity in a network as well as increases the performance.

As the authors stated, ADMM approach is not guaranteed to converge in non-convex problems. The authors used pre-trained networks to mitigate the problem of trapping into a local optimum. However, the datasets that are used are very small. It would be good to investigate how the proposed approach works on bigger datasets such as ImageNet.

Authors should compare their results with previous studies that use pruning or sparsity regularizers (Liu et al. (2015); Han et al. (2015); Collins & Kohli (2014)).

In the discussion section, authors stated that the proposed approach is efficient in training because of the separability property. Could you elaborate on that? Lets say this work uses two phases; phase 1 is pre-training a network, phase 2 is using sparsity and performance promoting steps.  Phase 2 also includes fine-tuning the network based on the new sparse structure. How long does phase 2 take compare to phase 1? How many epochs is needed to fine-tune the network?


review_title: null

final_score: rating: 7: Good paper, accept
review_content: The authors use the Alternating Direction Method of Multipliers (ADMM) algorithm for the first time on CNN models, allowing them to perform model compression without any appreciable loss on the CIFAR-10, CIFAR-100, and SVHN tasks.  The algorithmic details and the intuition behind their algorithm are generally well presented, (although there are occasional typos).

Pros: 
1) Put an old algorithm to good use in a new setting
2) The algorithm has the nice property that it is partially analytically solvable (due to the separability property the authors mention).  This contributes to the efficient trainability of the model
3) Seems to dovetail nicely with other results to encourage sparsity--that is, it can be used simultaneously--and is quite generalizable.

Cons:
1) It would be nice to see a more thorough analysis of the performance gains for using this method, beyond raw data about % sparsity--some sort of comparison involving training time would be great, and is currently lacking. EDIT: Authors addressed this by addition of results in Appendix B.
2) I would very much like to see some discussion about why the sparsity seems to *improve* the test performance, as mentioned in my previous comment.  Is this a general feature?  Is this a statistical fluke? etc.  Even if the answer is "it is not obvious, and determining why goes outside the scope of this work", I would like to know it!  EDIT: Authors addressed this by addition of statistical significance tests in the new Appendix A.
3) Based on the current text, and some of the other reviewer comments, I would appreciate an expanded discussion on how this work compares with other methods in the field.  I don't think a full numerical comparison is necessary, but some additional text discussing some of the other papers mentioned in the other reviews would greatly benefit the paper.  EDIT: Authors addressed this by followup to question and additional text in the paper.

Additional comments: If my Cons are addressed, I would definitely raise my score to a 6 or even a 7.  The core of this paper is quite solid, it just needs a little bit more polishing.  


EDIT: Score has been updated.  

Note: the authors probably meant "In order to verify" in the first sentence of Appendix A.
review_title: Review

final_score: rating: null
review_content: Could the authors elaborate more on point (3) in their conclusion?  Like Reviewer3, I'd like some more intuition as to how this particular point is supposed to be compared against  the more standard L-1 regularization, which is likewise differentiable.  Is the claim that this algorithm is sort of like a differentiable version of L-0 regularization?

Also, could the authors speculate more about why some of the models become better with more sparsity?  Are there enough experiments to determine if this is just a statistical fluke, or is this more of a general observation?  Or perhaps this is just an artifact of the sparsity allowing for longer training times?  Numerics would be nice here, but really I'm just curious.

Edit: Because I was added late as a reviewer, if the authors could provide an answer by Thursday 1/5/2017, I will incorporate that into my review.  Otherwise, I'll post a full review on Thursday (but will still, of course, consider further comments by the authors in updates to that review before the deadline).
review_title: Additional review questions

final_score: rating: null
review_content: 1) Can you please add a table that includes datasets/models/methods vs accuracies/sparsities? It would be easier to compare the results. 
2)  Does the proposed sparsity-promoting penalty term provide with a more sparse network than by just using L1,L2 regularizer during training?
review_title: Table for the results

final_score: rating: 7: Good paper, accept
review_content: This paper presents a framework to use sparsity to reduce the parameters and computations of a pre-trained CNN. 
The results are only reported for small datasets and networks, while it is now imperative to be able to report results on larger datasets and production-size networks.
The biggest problem with this paper is that it does not report numbers of inference time and gains, which is very important in production. And similarly for disk pace. Parameters reduction is useful only if it leads to a large decrease in space or inference time.
review_title: Alternating Direction Method of Multipliers for Sparse Convolutional Neural Networks

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
The paper presents a method for sparsifying the weights of the convolutional
filters and fully-connected layers of a CNN without loss of performance. Sparsification is achieved by using augmenting the CNN objective function with a regularization term promoting sparisty on groups of weights. The authors use ADMM for solving this optimization task, which allows decoupling of the two terms. The method alternates between promoting the sparsity of the network and optimizing the recognition performance

The method is technically sound and clearly explained. The paper is well organised and the ideas presented in a structured manner. I believe that sometimes the wording could be improved. 

The proposed method is simple and effective. Using it in combination with other CNN compression techniques such as quantization/encoding is a promising direction for future research. The experimental evaluation is convincing in the sense that the method seems to work well. The authors do not use state-of-the-art CNNs architectures, but I don't see this a requirement to deliver the message. 

On the other hand, the proposed method is closely related to recent works (as shown in the references posted in the public comment titled "Comparison with structurally-sparse DNNs using group Lasso), that should be cited and compared against (at least at a conceptual level). I will of course consider the authors rebuttal on this matter.

It would be very important for the authors to comment on the differences between these works and the proposed approach. It seems that both of these references use sparsity regularized training for neural networks, with a very similar formulation. 

The authors choose to optimize the proposed objective function using ADMM. It is not clear to me, why this approach should be more effective than proximal gradient descent methods. Could you please elaborate on this? ADMM is more demanding in terms of memory (2 copies of the parameters need to be stored). 

The claimed contributions (Section 5) seem a bit misleading in my opinion. Using sparsity promoting regularization in the parameters of the model has been used by many works, in particular in the dictionary learning literature but also in the neural network community (as stated above).  Claims 1,2 and 3, are well understood properties of L1-type regularizers. As written in the current version of the manuscript, it seems that these are claimed contributions of this particular work. 

A discussion on why sparsity sometimes helps improve performance could be interesting. 

In the experimental section, the authors mainly concentrate on comparing accuracy vs parameter reduction. While this is naturally very relevant property to report, it would be also interesting to show more results in terms of speed-up. which should also be improved by the proposed approach.

Other minor issues:
- In (3): I think a "^2" is missing in the augmented term (the rightmost term).

- The authors could cite the approach by Han et all for compressing DNNS:
Han, ICLR 2016 "Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding"


review_title: Review 

Meta-review: This paper studies a sparsification method for pre-trained CNNs based on the alternating direction method of multipliers. 
 The reviewers agreed that the paper is well-written and easy to follow. The authors were responsive during the rebuttal phase and addressed most of the reviewers questions. 
 
 The reviewers, however, disagree with the significance of this work. Whereas R4 is happy to see an established method (ADMM) applied to a nowadays very popular architecture, R1-R3 were skeptical about the scope of the experiments and the usefulness of the sparse approximation. 
 
 Pros:
 - Simple algorithmic description using well-known ADMM method. 
 - Consistent performance gains in small and mid-scale object classification problems. 
 
 Cons:
 - Lack of significance in light of current literature on the topic. 
 - Lack of numerical experiments on large-scale classification problems and/or other tasks.
 - Lack of clarity when reporting speedup gains. 
 
 Based on these assessments, the AC recommends rejection. Since this decision is not 100% aligned 
 with the reviewers, let me expand on the reasons why I recommend rejection. 
 
 This paper presents a sound algorithm that sparsifies the weights of a trained-CNN, and it shows that 
 the resulting pruned network works well, better than the original one. A priori, this is a solid result. 
 My main problem is that this contribution has to be taken in the context of the already large body of 
 literature addressing this same question, starting from the seminal 'Predicting Parameters in Deep Learning', by Misha Denil et al., NIPS 2013, which is surprisingly ignored in the bibliography here; and onwards with 'Exploiting linear structure within convolutional networks for efficient evaluation', Denton et al, NIPS'14 (also ignored) and 'Speeding up convolutional neural networks with low rank expansions', Jadergerg et al., '14. These and many other works culminated in two recent papers, 'Deep Compression', by Han et al, ICLR'16 and 'Learning Structured Sparsity in Deep Neural Networks' by Wen et al, NIPS'16.
 Several reviewers pointed out that comparisons with Wen et al were needed; the authors of the present submission now do cite this work, but do not compare their results quantitatively and do not present the reader with compelling reasons to choose their acceleration instead of Wen et al.'s. Moreover, they mention in the rebuttal that the paper was published after the submission, but the work was available on the arxiv well before the ICLR deadline (august 2016). Wen's paper presents (i) source code, (ii) experiments on imagenet, (iii) 3x to 5x speedups in both CPU and GPU, and (iv) accuracy improvements on Cifar of ~1.5%. As far as I can see, the present submission presents none of these. Further compression factors were obtained in 'Deep Compression', also in large-scale models. 
 
 In light of these results, the authors should present more compelling evidence (in this case, since this is an experimental paper, empirical evidence in the form of higher accuracy and/or larger speedups/gains) as to why should a practitioner use their model, rather than simply presenting the differences between the approaches.
Decision: Reject