Title: Training Compressed Fully-Connected Networks with a Density-Diversity Penalty
Authors: Shengjie Wang
Haoran Cai
Jeff Bilmes
William Noble
Venue: ICLR 2017
Keywords: Deep learning

Abstract: Deep models have achieved great success on a variety of challenging tasks. How- ever, the models that achieve great performance often have an enormous number of parameters, leading to correspondingly great demands on both computational and memory resources, especially for fully-connected layers. In this work, we propose a new “density-diversity penalty” regularizer that can be applied to fully-connected layers of neural networks during training. We show that using this regularizer results in significantly fewer parameters (i.e., high sparsity), and also significantly fewer distinct values (i.e., low diversity), so that the trained weight matrices can be highly compressed without any appreciable loss in performance. The resulting trained models can hence reside on computational platforms (e.g., portables, Internet-of-Things devices) where it otherwise would be prohibitive.

TL;DR: We propose a new ''density-diversity penalty'' to fully-connected layers to get significantly high sparsity and low diversity trained matrices, while keeping the performance the same.

Review discussion:
Good results. The technique looks a little bit complicate and obfuscated. Do you have results on more competitive tasks?
For instance CIFAR, SVHN... and more... MNIST use to have always room for improvement in many directions...
Thank you for pointing out our strong empirical results. We will be running more experiments on datasets such as CIFAR. We also note that there are TIMIT dataset results in the paper, and the MNIST data is actually quite difficult for the task of compression, since the model considered (the LeNet network) is very small and hard to compress.
But indeed, one can always add more experimental results.
The results have not been compared to a recent NIPS paper by  Guo et al., 2016.
Furthermore, there is a very similar paper in spirit submitted to ICLR that claims to achieve even better results (Soft Weight-Sharing for Neural Network Compression by
Ullrich et.al. ).
For example Guo et al., 2016 prunes 99.1% of weights in LeNet-5.
Whereas you report 92%.
The final compression rate could, however, be competitive nevertheless if you can quantize more than others.
Could you please also provide compression rates as proposed by Han et al., 2015.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient DNNs. In Advances In Neural Information Processing Systems, pp. 1379–1387, 2016.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. CoRR, abs/1510.00149, 2, 2015
Thank you for pointing out the most recent work on the compression task. Our proposed regularizer enforces high sparsity and low diversity simultaneously, and thus we should compare the final compression rate instead of sparsity alone. We compare our compression rates to Han's work using the same compression rate calculation, which considers the sparse matrix representation as well as bits for encoding, and we report the numbers in paragraphs 3-4 of section 4.1 and paragraph 3 of section 4.2.
The compression rate reported in Guo's work is not the same compression rate shown in our work or Han's work.
Indeed, Guo et al. use the inverse of the overall sparsity. As mentioned by the reviewer, the final compression rate could be comparable, and especially for the LeNet-300-100 model, which is a smaller fully-connected network and therefore harder to compress, our compression rate should be significantly better.
Furthermore, the reported accuracy in Guo's work is quite worse than that of the uncompressed model.
For Ullrich's work, comparing the LeNet-300-100 model as reported in paragraph 1 of section 6.1 of Ullrich's work, our method gives better sparsity as well as significantly better accuracy.
However, it is hard to compare the final results, because for Ullrich's work, a more aggressive encoding method of the sparse matrix is used and Huffman encoding is also applied.
Your paper only considers the target problem of compression and does not pretend to apply to regularisation in the sense that the point is usually to improve generalization performance.
However given the known links between the two, (i.e. Occam's razor, Minimum Description Length, and more generally smaller models being known to generalize better), have you investigated the impact on generalisation ? Do you think that your method definitely won't help with respect to generalization (and then why shouldn't it), or do you think it should help w.r.t generalisation (but then why didn't you talk about it in the paper) ?
Thank you for pointing out the possible improvement on generalization power of the proposed regularizer. We think the proposed regularization method could also improve the performance of the model, if the certain pattern in the data is learned as enforced by the regularization.
However, the proposed regularization is a natural fit for the model compression task, so we focus on the compression results in the present paper.
For deep neural networks, it is not clear whether smaller models will always generate better results, because 1) empirically, we often find that a deeper and wider model gives better performance, and 2) theoretically, there is recent work showing that the local optima of the network are close to the global optimum if the network is big enough.
So in general, we don't think that the compressed models are necessarily better in terms of performance. Our primary motivation in this work is to enable the good performance offered by deep models to be achievable on lower-resource devices.
There are a number of different things going on here:
- spare initialization
- the proposed Density-Diversity Penalty applied with some probability
- mode subtraction after the Density-Diversity Penalty is applied
- weight tying (weights with the same value after the above steps accumulate and share gradient updates)
Can you speak to the importance of each of these to your overall results? Have you done any sensitivity analysis with respect to the probability of applying the penalty or the fraction of sparsity in the initialization?
Thank you for asking about the importance of different steps as well as suggesting the sensitivity test. The four parts mentioned are all required to get the desired compressed model performance with both low density and low diversity. The sparse initialization is more important for the low density objective, and the regularizer is more important for the low diversity objective. The weight tying in the end focuses on obtaining performance as good as that of the original, uncompressed model.
Applying the penalty with a certain probability is for training efficiency. Ideally, we would like to apply the penalty at every mini-batch.
However, we find that even with a certain small probability of applying the penalty, we can obtain the desired compression result with much greater training speed.
For sparse initialization, we vary the initialization sparsity fraction from 5% to 20%, and when comparing the highest and lowest values on the TIMIT dataset, we observe 0.2% absolute difference for phone accuracy, 1.17%, 2.48% and 2.72% absolute differences for learned sparsity of the first three weight matrices, respectively, and almost no difference for learned diversity.
Overall, we believe the performance is not very sensitive to the fraction of sparse initialization (this answers your question about sensitivity). Given the opportunity, we will add this information to the final paper.
The paper shows promising results but it is difficult to read and follow. It presents different things closely related and it is difficult to asses the performance of each one. Diversity, sparsity, regularization term, tying weights. Anyway results are good.
Thanks for pointing out our strong empirical results. We are working on making the paper clearer. The diversity, sparsity, and tying weights are the three key components of the proposed method. Applying any of the component individually does not yield nearly as good results. Please see the discussion with AnonReviewer3 for more details on this point.
This work introduces a number of techniques to compress fully-connected neural networks while maintaining similar performance, including a density-diversity penalty and associated training algorithm. The core technique of this paper is to explicitly penalize both the overall magnitude of the weights as well as diversity between weights. This approach results in sparse weight matrices comprised of relatively few unique values.
Despite introducing a more efficient means of computing the gradient with respect to the diversity penalty, the authors still find it necessary to apply the penalty with some low probability (1-5%) per mini-batch.
The approach achieves impressive compression of fully connected layers with relatively little loss of accuracy. I wonder if the cost of having to sort weights (even for only 1 or 2 out of 100 mini-batches) might make this method intractable for larger networks. Perhaps the sparsity could help remove some of this cost?
I think the biggest fault this paper has is the number of different things going on in the approach that are not well explored independently. Sparse initialization, weight tying, probabilistic application of density-diversity penalty and setting the mode to 0, and alternating schedule between weight tied standard training and diversity penalty training. The authors don't provide enough discussion of the relative importance of these parts.
Furthermore, the only quantitative metric shown is the compression rate which is a function of both sparsity and diversity such that they cannot be compared on their own. I would really like to see how each component of the algorithm affects diversity, sparsity, and overall compression.
A quick verification: Section 3.1 claims the density-diversity penalty is applied with a fixed probability per batch while 3.4 implies structured phases alternating between application of density-diversity and weight tied standard cross entropy. Is this scheme in 3.4 only applying the density-diversity penalty probabilistically when it is in the density-diversity phase?
Preliminary rating:
I think this is an interesting paper but lacks sufficient empirical evaluation of its many components.
As a result, the algorithm has the appearance of a collection of tricks that in the end result in good performance without fully explaining why it is effective.
Minor notes:
Please resize equation 4 to fit within the margins (\resizebox{\columnwidth}{!}{ blah } works well in latex for this)
Thanks for your questions about applying the components of the proposed method independently. Applying regularization alone gives diversity similar to what we reported in the paper but at a cost of much higher sparsity.
Also, even worse, performance degrades as well in this case.
In fact, we began our investigation without using sparse initialization and weight tying, and we found that performance is far worse in this case.  Applying sparse initialization or weight tying independently would have no effect on the compression task.
Overall, we achieve good compression results only when sparse initialization, density-diversity penalty, and weight tying are all applied simultaneously. A good compression requires low diversity (which is achieved by applying the density-diversity penalty), high sparsity (which is achieved by applying both the density-diversity penalty and the sparse initialization) and no loss of performance (which is achieved by training with weight tying). The other components of the algorithm proposed in the paper (e.g. applying the density-diversity penalty with a certain small probability), are included to make the method more computationally efficient.  We have added a paragraph to the end of section 3.4 in the latest version of the paper, and we will be happy to add more discussions into the final version of the paper, should we be given the opportunity.
We are also optimistic about the applicability of our method to large networks.
First, the experiments on TIMIT uses a network with matrices of size 2048 * 2048, i.e., reasonably representative of modern big networks.
Moreover, sorting can be greatly accelerated by: 1) sorting in parallel; 2) using the sparse structure of the weight matrices, as also pointed out by the other reviewers; and 3) using the sorting results of previous batches.
Regarding the "quick verification", the penalty is not applied at all during the weight tying phase, and the weight tying phase would not change the diversity or sparsity of the trained weight matrices. The weight tying phase is solely for the purpose of getting the compressed network performance similar to the original, uncompressed performance.
The method proposes to compress the weight matrices of deep networks using a new density-diversity penalty together with a computing trick (sorting weights) to make computation affordable and a strategy of tying weights.
This density-diversity penalty consists of an added cost corresponding to the l2-norm of the weights (density) and the l1-norm of all the pairwise differences in a layer.
Regularly, the most frequent value in the weight matrix is set to zero to encourage sparsity.
As weights collapse to the same values with the diversity penalty, they are tied together and then updated using the averaged gradient.
The training process then alternates between training with 1. the density-diversity penalty and untied weights, and 2. training without this penalty but with tied weights.
The experiments on two datasets (MNIST for vision and TIMIT for speech) shows that the method achieves very good compression rates without loss of performance.
The paper is presented very clearly,  presents very interesting ideas and seems to be state of the art for compression. The approach opens many new avenues of research and the strategy of weight-tying may be of great interest outside of the compression domain to learn regularities in data.
The result tables are a bit confusing unfortunately.
minor issues:
p1
english mistake: “while networks *that* consist of convolutional layers”.
p6-p7
Table 1,2,3 are confusing. Compared to the baseline (DC), your method (DP) seems to perform worse:
In Table 1 overall, Table 2 overall FC, Table 3 overall, DP is less sparse and more diverse than the DC baseline. This would suggest a worse compression rate for DP and is inconsistent with the text which says they should be similar or better.
I assume the sparsity value is inverted and that you in fact report the number of non-modal values as a fraction of the total.
Thanks for pointing out that our paper is strong.
Regarding the confusion about the results tables, 1-sparsity (or density, as we will now call it based on your comments) is presented instead of sparsity.  Sparsity represents the fraction of weights that are zero, while we report percentage of non-zero entries.
Thus, our methods achieve very sparse weight matrices. We choose to report density instead of sparsity because density (sometimes known as the percentage of non-zero entries) is used in related work (e.g., the "deep compression" paper, Han et al., 2015a);
also, density is often a small number and, hence, is easier to read in plots;
finally, and perhaps most importantly, for both density and diversity, lower is better meaning they are more mutually consistent.

Meta-review:
role: PC
data: comment: The reviewers unanimously recommended accepting the paper.

Decision: Accept (Poster)