Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes to use an empirical Bayesian approach to learn the parameters of a neural network, and their priors.
A mixture model prior over the weights leads to a clustering effect in the weight posterior distributions (which are approximated with delta peaks). 
This clustering effect can exploited for parameter quantisation and compression of the network parameters.
The authors show that this leads to compression rates and predictive accuracy comparable to related approaches. 

Earlier work [Han et al. 2015] is based on a three-stage process of pruning small magnitude weights, clustering the remaining ones, and updating the cluster centres to optimise performance. The current work provides a more principled approach that does not have such an ad-hoc multi-stage structure, but a single iterative optimisation process.

A first experiment, described in section 6.1 shows that an empirical Bayes’ approach, without the use of hyper priors, already leads to a pronounced clustering effect and to setting many weights to zero. 
In particular a compression rate of 64.2 is obtained on the LeNet300-100 model.
In section 6.1 the text refers to figure C, I suppose this should be figure 1.

Section 6.2 describes an experiment where hyper-priors are used, and the parameters of these distributions, as well as other hyper-parameters such as the learning rates, are being optimised using Spearmint (Snoek et al., 2012). Figure 2 shows the performance of the  different points in the hyper-parameter space that have been evaluated (each trained network gives an accuracy-compressionrate point in the graph). The text claims that best results lie on a line, this seems a little opportunistic interpretation given the limited data. Moreover, it would be useful to add a small discussion on whether such a linear relationship would be expected or not. Currently the results of this experiment lack interpretation.

Section 6.3 describes results obtained for both CNN models and compares results to the recent results of (Han et al., 2015) and (Guo et al., 2016).
Comparable results are obtained in terms of compression rate and accuracy. 
The authors state that their current algorithm is too slow to be useful for larger models such as VGG-19, but they do briefly report some results obtained for this model (but do not compare to related work). It would be useful here to explain what slows the training down with respect to standard training without the weight clustering approach, and how the proposed algorithm scales in terms of the relevant quantities of the data and the model.

The contribution of this paper is mostly experimental, leveraging fairly standard ideas from empirical Bayesian learning to introduce weight clustering effects in CNN training.
This being said, it is an interesting result that such a relatively straightforward approach leads to results that are on par with state-of-the-art, but more ad-hoc, network compression techniques.
The paper could be improved by clearly describing the algorithm used for training, and how it scales to large networks and datasets.
Another point that would deserve further discussion is how the hyper-parameter search is performed ( not using test data I assume), and how the compared methods dealt with the search over hyper-parameters to determine the accuracy-compression tradeoff. Ideally, I think, methods should be evaluated across different points on this trade-off.


review_title: Empirical Bayesian learning applied for neural net parameter compression

final_score: rating: null
review_content: In 4.5 you write that "96% of the first layer (235K parameter), 90% of the second (30K) and only 18% of the
final layer (10K) are being pruned.". Especially for the first two layers it would be very interesting to know how the zeros are distributed. E.g. it could be whole filters set to zero which would suggest that not as many convolutional filters are needed. Or the filters themselves could be much sparser in space or in the input channels. Maybe the convolutions are even reduced to 1x1 convolutions which would basically be a "pixelwise" nonlinearity.

Also it would be interesting to know whether these results (most zeros in early layers) translate to the VGG network's compression.
review_title: Question regarding the zero weights after compression

final_score: rating: null
review_content: I am also interesting in seeing (or getting a description) of any filter visualizations you might be able to compute. This could give us additional ideas for variants of weight sharing that are influenced by old ideas involving filterbanks and sparsity.
review_title: filter visualizations

final_score: rating: 7: Good paper, accept
review_content: The authors propose a method to compress neural networks by retraining them while putting a mixture of Gaussians prior on the weights with learned means and variances which then can be used to compress the neural network by first setting all weights to the mean of their infered mixture component (resulting in a possible loss of precision) and storing the network in a format which saves only the fixture index and exploits the sparseness of the weights that was enforced in training.

Quality:
Of course it is a serious drawback that the method doesn't seem to work on VGG which would render the method unusable for production (as it is right now, maybe this can be improved). I guess AlexNet takes too long to process, too, otherwise this might be a very valuable addition.
In Figure 2 I am noticing two things: On the left, there is a large number of points with improved accuracy which is not the case for LeNet5-Caffe. Is there any intuition for why that's the case? Additionally regarding the spearmint optimization: Do they authors have found any clues about which hyperparameter settings worked well? This might be helpful for other people trying to apply this method.
I really like Figure 7 in it's latest version.

Clarity:
Especially section 2 on MDL is written very well and gives a nice theoretic introduction. Sections 4, 5 and 6 are very short but seem to contain most relevant information. It might be helpful to have at least some more details about the used models in the paper (maybe the number of layers and the number of parameters).
In 6.1 the authors claim "Even though most variances seem to be reasonable small there are some that are large". From figure 1 this is very hard to assess, especially as the vertical histogram essentially shows only the zero component. It might be helpful to have either a log histogram or separate histograms for each componenent. What are the large points in Figure 2 as opposed to the smaller ones? They seem to have a very good compression/accuracy loss ratio, is that it?
Some other points are listed below

originality: While there has been some work on compressing neural networks by using a reduced number of bits to store the parameters and exploiting sparsity structure, I like the idea to directly learn the quantization by means of a gaussian mixture prior in retraining which seems to be more principled than other approaches

significance: The method achievs state-of-the-art performance on the two shown examples on MNIST, however these networks are far from the deep networks used in state-of-the-art models. This obviously is a drawback for the practical usability of the methods and therefor it's significance. If the method could be made to work on more state-of-the-art networks like VGG or ResNet, I would consider this a contribution of high significance.

Minor issues:

page 1: There seems to be a space in front of the first author's name
page 3: "in this scenario, pi_0 may be fixed...". Missing backslash in TeX?
page 6: 6.2: two wrong blanks in "the number of components_, \tau_."
page 6, 6.3: "in experiences with VGG": In experiments?
page 12: "Figure C": Figure 7?

review_title: Nice idea, some minor issues

final_score: rating: 7: Good paper, accept
review_content: This paper revives a classic idea involving regularization for purposes of compression for modern CNN models on resource constrained devices. Model compression is hot and we're in the midst of lots of people rediscovering old ideas in this area so it is nice to have a paper that explicitly draws upon classic approaches from the early 90s to obtain competitive results on standard benchmarks.

There's not too much to say here: this study is an instance of a simple idea applied effectively to an important problem, written up in an illuminating manner with appropriate references to classic approaches. The addition of the filter visualizations enhances the contribution.
review_title: nice tie-in to classic NN literature through lens of modern engineering needs

Meta-review: This paper provides a principled and practical formulation for weight-sharing and quantization, using a simple mixture of Guassians on the weights, and stochastic variational inference. The main idea and results are presented clearly, along with illustrative side-experiments showing the properties of this method in practice. Also, the method is illustrated on non-toy problems.
Decision: Accept (Poster)