Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes to use a mixture of continuous spikes propto 1/abs(w_ij-c_k) as prior for a Bayesian neural network and demonstrates good performance with relatively sparsified convnets for minist and cifar-10. The paper is building quite a lot upon Kingma et al 2015 and  Molchanov et al 2017. 

The paper is of good quality, clearly written with an ok level of originality and significance.

Pros:
1. Demonstrates a sparse Bayesian approach that scales.
2. Really a relevant research area for being able to make more efficient and compact deployment.
Cons:
1. Somewhat incremental relative to the papers mentioned above.
2. Could have taken the experimental part further. For example can we learn something about what part of the network has  the biggest potential for being pruned and use that to come up with modifications of the architecture?    
review_title: Sparsity prior in variational Bayesian deep learning

final_score: rating: 7: Good paper, accept
review_content: 
The goal of this work is to infer weights of a neural network, constrained to a discrete set, where each weight can be represented by a few bits. This is a quite important and hot topic in deep learning. As a direct optimization would lead to a highly nontrivial combinatorial optimization problem, the authors propose a so-called 'quantizing prior' (actually a relaxed spike and slab prior to induce a sparsity enforcing heavy tail prior) over weights and derive a differentiable variational KL approximation. One important advantage of the current method is that this approach does not require fine-tuning after quantization. The paper presents ternary quantization for LeNet-5 (MNIST) and DenseNet-121 (CIFAR-10).

The paper is mostly well written and cites carefully the recent relevant literature. While there are a few glitches here and there in the writing, overall the paper is easy to follow. One exception is that in section 2, many ideas are presented in a sequence without providing any guidance where all this will lead.
The idea is closely related to sparse Bayesian learning but the variational approximation is achieved via the local reparametrization trick of Kingma 2015, with the key idea presented in section 3.3.



Minor

In the introduction, the authors write "... weights with a large variance can be pruned as they do not contribute much to the overall computation". What does this mean? Is this the marginal posterior variance as in ARD? 

The authors write: "Additionally, variational Bayesian inference  is known to automatically reduce parameter redundancy by penalizing overly complex models." I would argue that 
it is Bayesian inference; variational inference sometimes retains this property, but not always.

In Eq (10), z needs also subscripts, as otherwise the notation may suggest parameter tying. Alternatively, drop the indices entirely, as later in the paper.

Sec. 3.2. is not very well written. This seems to be the MAP of the product of the marginals,
or the mode of the variational distribution, not the true MAP configuration of the weight posterior. Please be more precise. 

The abbreviation P&Q (probably Post-training Quantization) seems to be not defined in the paper.

review_title: A modern sparse Bayesian learning approach to weight quantization

final_score: rating: 7: Good paper, accept
review_content: This paper presents Variational Network Quantization; a variational Bayesian approach for quantising neural network weights to ternary values post-training in a principled way. This is achieved by a straightforward extension of the scale mixture of Gaussians perspective of the log-uniform prior proposed at [1]. The authors posit a mixture of delta peaks hyperprior over the locations of the Gaussian distribution, where each peak can be seen as the specific target value for quantisation (including zero to induce sparsity). They then further propose an approximation for the KL-divergence, necessary for the variational objective, from this multimodal prior to a factorized Gaussian posterior by appropriately combining the approximation given at [2] for each of the modes. At test-time, the variational posterior for each weight is replaced by the target quantisation value that is closest, w.r.t. the squared distance, to the mean of the Gaussian variational posterior. Encouraging experimental results are shown with performance comparable to the state-of-the-art for ternary weight neural networks.

This paper presented a straightforward extension of the work done at [1, 2] for ternary networks through a multimodal quantising prior. It is generally well-written, with extensive preliminaries and clear equations. The visualizations also serve as a nice way to convey the behaviour of the proposed approach. The idea is interesting and well executed so I propose for acceptance. I only have a couple of minor questions: 
- For the KL-divergence approximation you report a maximum difference of 1 nat per weight that seems a bit high; did you experiment with the `naive` Monte Carlo approximation of the bound (e.g. as done at Bayes By Backprop) during optimization? If yes, was there a big difference in performance?
- Was pre-training necessary to obtain the current results for MNIST? As far as I know, [1] and [2] did not need pre-training for the MNIST results (but did employ pre-training for CIFAR 10).
- How necessary was each one of the constraints during optimization (and what did they prevent)? 
- Did you ever observe posterior means that do not settle at one of the prior modes but rather stay in between? Or did you ever had issues of the variance growing large enough, so that q(w) captures multiple modes of the prior (maybe the constraints prevent this)? How sensitive is the quantisation scheme?

Other minor comments / typos:
(1) 7th line of section 2.1 page 2, ‘a unstructured data’ -> ‘unstructured data’
(2) 5th line on page 3, remove ‘compare Eq. (1)’ (or rephrase it appropriately).
(3) Section 2.2, ’Kullback-Leibler divergence between the true and the approximate posterior’; between implies symmetry (and the KL isn’t symmetric) so I suggest to change it to e.g. ‘from the true to the approximate posterior’ to avoid confusion. Same for the first line of Section 3.3.
(4) Footnote 2, the distribution of the noise depends on the random variable so I would suggest to change it to a general \epsilon \sim p(\epsilon).
(5) Equation 4 is confusing.

[1] Louizos, Ullrich & Welling, Bayesian Compression for Deep Learning.
[2] Molchanov, Ashukha & Vetrov, Variational Dropout Sparsifies Deep Neural Networks.
review_title: Good paper extending on previous work on variational compression for neural networks.

Meta-review: The paper presents a variational Bayesian approach for quantising neural network weights and makes interesting and useful steps in this increasingly popular area of deep learning.
Decision: Accept (Poster)