Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This work proposes a set of efficient algorithms for learning and prediction in Bayesian quantized networks, which allows for differentiable learning without the need to sampling. From this point of view, this work targets at solving a real and interesting problem in BNN. I am not an expert in this area, and I can only comment on broad picture.

Some questions:
-	The proposed method is claimed to work on large-scale datasets. However, all datasets selected in experiment are black and white images with low resolution (28 x 28 ?). A ‘real’ large-scale dataset is expected. The current experiment results are far from sufficient.
-	The competitive methods are limited. If the author claims BNN in general can have well-calibrated uncertainties, non-BNN methods should also be considered. Further, when the data has heavy tail or is not Gaussian-distributed, will the model still give well calibrated results?
-	Does this proposed sampling free BQN have faster computational speed than standard BNN? If yes, it is also desired to have a comparison.
-	I am totally lost at section 4.3. It seems to give a fast way of computing Eq.(12). A general introduction at the beginning or the end of section 4.3 could be really helpful. Further, I’m not clear why the approximation made in the model is good enough. A pseudo-algorithm is desired for clarification.  

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposed Bayesian Quantized Neural Networks (BQN), which is a rough Bayesian treatment for quantized neural networks (QNN). The advantages the BQN over QNN is that it provides better uncertainty estimates as well as accuracy. The disadvantage is that BQN loses the memory efficient property of QNN, that is, BQN in general still needs to store real-value parameters, which parameterize the binary weights. The authors compare BQN to EQNN and show that BQN outperforms the latter.

In general, the paper is well organized and fairly readable, though the first 2 sections could have been more concise. I appreciate the careful description of notations. 

The topic of Bayesian formulation and inference in the context of QNN is interesting. I like the analysis provided for handling small/medium/large fan-in for Bayesian QNN.

One of my concerns is that some description in the provided theorems (as well as the proof) is not very rigorous and sometimes misleading. For example, Theorem 3.2 claims that there is an analytic form for Bayesian inference in softmax layers. One would expect that it is either an exact integration over the distribution of h, or a valid bound. Unfortunately, it is neither. Looking at the proof, it seems it is a rather coarse approximation. Note that in the Appendix, inequality in Equation (42) and Equation (47) takes different direction, meaning that this is not a valid bound for the error term L. The authors may want to adjust the description to make it more rigorous and avoid confusion.

The paper describes BQN in very general terms. It is unclear what distributions are used to parameterize the weights in the experiments. Do you use Bernoulli distributions to parameterize the binary weights {-1, 1}, and learn the real-valued parameters for the distributions during training?

Theorem 3.1 is straightforward. The rough approximation provided in Theorem 3.2 is interesting. It would be interesting to compare this with Gaussian natural-parameter networks, which bypass the softmax problem using multi-logistic regression for classification, where the convolution of Gaussian and sigmoid functions has a closed form.

In terms of related work, references such as natural-parameter networks (NPN) and its variants (one of which specifically handles the softmax case) are missing [1, 2]. Note that, similarly to BQN, NPN also provides a general probabilistic (Bayesian) framework for sampling-free learning/prediction. Theorem 3.3 is also covered in Section 3.2 of [1].

This brings me to another of my concerns. The only baseline provided is E-QNN from three years ago. There are various state-of-the-art baselines, either on QNN or BNN. It would certainly make the paper much stronger if such baselines are included, especially when the authors decide to take up the full ten-page space in this submission.


Minor:

P2: bf Bayesian -> Bayesian

P2: theta that predicted -> theta that predict

P6: Lemma 4.2 -> Theorem 4.2

P6: by with -> with

Assuming most of my concerns can be addressed during the feedback period, I tend to give a ‘weak accept’ to this submission.

Natural-Parameter Networks: A Class of Probabilistic Neural Networks, NIPS 2016
Feed-forward Propagation in Probabilistic Neural Networks with Categorical and Max Layers, ICLR 2018
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: (1) Contributions:
The paper proposes an algorithm for training quantized Bayesian neural networks (BQN). The method optimizes a lower bound to the ELBO that can be analytically computed for BQNs.

The paper shows that propagation in BQNs is a sequence of tensor contractions, for which there exist fast and efficient approximate algorithms.

BQNs show good performance on set of image classification benchmarks (MNIST, KMNIST, Fashion MNIST).

(2) Originality:
I am not aware of any prior works that attempts to do inference in quantized networks. The idea of BQNs and propagation as tensor contractions is interesting and novel. It is a cool result that in quantized networks, one can exactly calculate the distribution of the activations (h^(l)) and therefore get an accurate estimate of the ELBO.

It is also interesting that this algorithm can be implemented as a series of tensor contractions. This leads to a fast and efficient algorithm for training.

Regarding the novelty in the lower bound to the ELBO, variational Bayes makes the same approximation by sampling the layer activations (h^(1) .. h^(L-1)). In VB, when we sample the layer activations, we approximate the logarithm of the expectation by the logarithm of a Monte Carlo sample, which is a lower bound due to Jensen’s inequality. The only difference here is that this approximation is only done in the final hidden layer instead of every layer.

(2) Clarity:
I want to point out some issues with the writing.

Overall, I believe that the number of equations and theorems hinders the readability of the paper. It is very confusing to try and navigate the various notations and assess the significance of each result. I would recommend including a few key equations and theorems in the paper and leaving the rest to the appendix. The ideas and concepts in the paper could be easily communicated in writing.

I do not think this paper should be 9.5 pages. The work could be comfortably explained in the standard 8 pages. Namely, in Section 3.1 and 3.2, it is unnecessary to provide analytic gradients for the expressions (in the age of automatic differentiation). Section 4.3 should be left to the appendix. I found this section very confusing and it does not seem to be important in the narrative.

(4) Significance:
I think this work is significant, although I felt that the experiments section could have included more regression benchmarks as well as more difficult datasets such as CIFAR10.

(5) Impact:
This work would certainly be interesting to researchers and anyone working on quantized neural networks.

Overall assessment:
I think this paper presents interesting and novel ideas, but it has shortcomings in the presentation.
review_title: Official Blind Review #3

Meta-review: This paper proposes Bayesian quantized networks and efficient algorithms for learning and prediction of these networks. The reviewers generally thought that this was a novel and interesting paper.  There were a few concerns about the clarity of parts of the paper and the experimental results. These concerns were addressed during the discussion phase, and the reviewers agree that the paper should be accepted.
Decision: Accept (Poster)