Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This is yet another paper on parameter tying and compression of DNNs/CNNs.  The key idea here is a soft parameter tying under the K-means regularization on top of which an L1 regularization is further imposed for promoting sparsity. This strategy seems to help the hard tying in a later stage while keeping decent performance.  The idea is sort of interesting and the reported experimental results appear to be supportive.  However, I have following concerns/comments. 

1.  The roles played by K-means and L1 regularization are a little confusing from the paper.  In Eq.3, it appears that the L1 regularization is always used in optimization. However, in Eq.4,  the L1 norm is not included.  So the question is,  in the soft-tying step, is L1 regularization always used?  Or a more general question,  how important is it to regularize the cross-entropy with both K-means and L1? 

2. A follow-up question on K-means and L1.   If no L1 regularization, does the K-means soft-tying followed by a hard-tying work as well as using the L1 regularization throughout? 

3. It would be helpful to say a few words on the storage of the model parameters. 

4. It would be helpful to show if the proposed technique work well on sequential models like LSTMs.
review_title: comments on K-means and L1 regularization

final_score: rating: 6: Marginally above acceptance threshold
review_content: As the authors mentioned, weight-sharing and pruning are not new to neural network compression. The proposed method resembles a lot with the deep compression work (Han et. al. 2016), with the distinction of clustering across different layers and a Lasso regularizer to encourage sparsity of the weights. Even though the change seems minimal, the authors has demonstrated the effectiveness on the benchmark.

But the description of the optimization strategy in Section 3 needs some refinement. In the soft-tying stage, why only the regularizer (1) is considered, not the sparsity one? In the hard-tying stage, would the clustering change in each iteration? If not, this has reduced to the constrained problem as in the Hashed Compression work (Chen et. al. 2015) where the regularizer (1) has no effect since the clustering is fixed and all the weights in the same cluster are equal. Even though it is claimed that the proposed method does not require a pre-trained model to initialize, the soft-tying stage seems to take the responsibility to "pre-train" the model.

The experiment section is a weak point. It is much less convincing with no comparison result of compression on large neural networks and large datasets. The only compression result on large neural network (VGG-11) comes with no baseline comparisons. But it already tells something: 1) what is the classification result for reference network without compression? 2) the compression ratio has significantly reduced comparing with those for MNIST. It is hard to say if the compression performance could generalize to large networks.

Also, it would be good to have an ablation test on different parts of the objective function and the two optimization stages to show the importance of each part, especially the removal of the soft-tying stage and the L1 regularizer versus a simple pruning technique after each iteration. This maybe a minor issue, but would be interesting to know: what would the compression performance be if the classification accuracy maintains the same level as that of the deep compression. As discussed in the paper, it is a trade-off between accuracy and compression. The network could be compressed to very small size but with significant accuracy loss.

Some minor issues:
- In Section 1, the authors discussed a bunch of pitfalls of existing compression techniques, such as large number of parameters, local minimum issues and layer-wise approaches. It would be clearer if the authors could explicitly and succinctly discuss which pitfalls are resolved and how by the proposed method towards the end of the Introduction section. 
- In Section 4.2, the authors discussed the insensitivity of the proposed method to switching frequency. But there is no quantitative results shown to support the claims.
- What is the threshold for pruning zero weight used in Table 2?
- There are many references and comparisons missing: Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations in NIPS 17 for instance. This paper also considers quantization for compression which is related to this work.
review_title: Simple and effective compression method, but needs refinement and large-scale experiments

final_score: rating: 6: Marginally above acceptance threshold
review_content: Approach is interesting however my main reservation is with the data set used for experiments and making general (!) conclusions. MNIST, CIFAR-10 are too simple tasks perhaps suitable for debugging but not for a comprehensive validation of quantization/compression techniques. Looking at the results, I see a horrific degradation of 25-43% relative to DC baseline despite being told about only a minimal loss in accuracy. A number of general statements is made based on MNIST data, such as on page 3 when comparing GMM and k-means priors, on page 7 and 8 when claiming that parameter tying and sparsity do not act strongly to improve generalization. In addition, by making a list of all hyper parameters you tuned I am not confident that your claim that this approach requires less tuning. 

Additional comments:

(a) you did not mention student-teacher training
(b) reference to previously not introduced K-means prior at the end of section 1
(c) what is that special version of 1-D K-means?
(d) Beginning of section 4.1 is hard to follow as you are referring to some experiments not shown in the paper.
(e) Where is 8th cluster hiding in Figure 1b?
(f) Any comparison to a classic compression technique would be beneficial.
(g) You are referring to a sparsity at the end of page 8 without formally defining it. 
(h) Can you label each subfigure in Figure 3 so I do not need to refer to the caption? Can you discuss this diagram in the main text, otherwise what is the point of dumping it in the appendix?
(i) I do not understand Figure 4 without explanation. 
review_title: Automatic Parameter Tying in Neural Networks

Meta-review: This paper presents yet another scheme for weight tying for compressing neural networks, which looks a lot like a less Bayesian version of recent related work, and gets good empirical results on realistic problems.

This paper is well-executed and is a good contribution, but falls below the bar on
1) Discovering something new and surprising, except that this particular method (which is nice and simple and sensible) works well.  That is, it doesn't advance the conversation or open up new directions.
2) Potential impact (although it might be industrially relevant)

Also, the title is a bit overly broad given the amount of similar existing work.
Decision: Reject