Sampling-Free Learning of Bayesian Quantized Neural Networks
Authors: Jiahao Su
Milan Cvitkovic
Furong Huang
Venue:  (2020)
Keywords: Bayesian neural networks
Quantized neural networks

Abstract: Bayesian learning of model parameters in neural networks is important in scenarios where estimates with well-calibrated uncertainty are important. In this paper, we propose Bayesian quantized networks (BQNs), quantized neural networks (QNNs) for which we learn a posterior distribution over their discrete parameters. We provide a set of efficient algorithms for learning and prediction in BQNs without the need to sample from their parameters or activations, which not only allows for differentiable learning in quantized models but also reduces the variance in gradients estimation. We evaluate BQNs on MNIST, Fashion-MNIST and KMNIST classification datasets compared against bootstrap ensemble of QNNs (E-QNN). We demonstrate BQNs achieve both lower predictive errors and better-calibrated uncertainties than E-QNN (with less than 20% of the negative log-likelihood).

Published as a conference paper at ICLR 2020
SAMPLING-FREE LEARNING OF BAYESIAN QUANTIZED NEURAL NETWORKS
Jiahao Su
Milan Cvitkovic
Department of Electrical and Computer Engineering University of Maryland College Park, MD 20740 j ahaosuu umd edu
Amazon Web Services
Seattle, WA, USA cvit amazon. com
Furong Huang
Department of Computer Science University of Maryland College Park, MD 20740 furonghêcs umd. edu
ABSTRACT
Bayesian learning of model parameters in neural networks is important in sce- narios where estimates with well-calibrated uncertainty are important. In this pa- per, propose Bayesian quantized networks (BQNs), quantized neural networks (QNNs) for which we learn a posterior distribution over their discrete parameters. We provide of efficient algorithms for learning and prediction in BQNs with- out the need to sample from their parameters or activations, which not only allows for differentiable learning in QNNs. but also reduces the variance in gradients. We evaluate BQNs MNIST, Fashion-MNIST, KMNIST and CIFAR10 10 image classification datasets. compared against bootstrap ensemble of QNNs (E-QNN), We demonstrate BQNs achieve both lower predictive errors and better-calibrated uncertainties than E- E-QNN (with less than 20% of the negative lig-liielihood).
INTRODUCTION
A Bayesian approach deep learning considers the network's parameters to be random variables and seeks to infer their posterior distribution given the training data. Models trained this way, called Bayesian neural networks (BNNs) (Wang Yeung 2016), in principle have well-calibrated uncer- tainties when they make predictions, which is important in scenarios such as active learning and reinforcement learning (Gal 2016). Furthermore, the posterior distribution over the model parame- ters provides valuable information for evaluation and compression of neural networks.
There are three main challenges using BNNs: (1) Intractable posterior: Computing and storing the exact posterior distribution over the network weights is intractable due to the complexity and high-dimensionallity of deep networks.
(2) Prediction: Performing forward pass (a.k.a. as prob- abilistic propagation) in BNN to compute a prediction for an input cannot be performed exactly, since the distribution of hidden activations at each layer is intractable to compute.
(3) Learning: The classic evidence lower bound (ELBO) learning objective for training BNNs is not amenable to backpropagation as the ELBO is not an explicit function of the output probabilistic propagation.
These challenges are typically addressed either by making simplifying assumptions about the distri- butions of the parameters and activations, or by using sampling- based approaches, which are expen- sive and unreliable (likely to overestimate the uncertainties in predictions). Our goal is to propose sampling-free method which uses probabilistic propagation to deterministically learn BNNs.
A seemingly unrelated area of deep learning research is that of quantized neural networks (QNNs), which offer advantages of computational and memory efficiency compared to continuous-valuee models. QNNs, like BNNs, face challenges in training, though for different reasons: (4.1) The non-

---

Published as a conference paper at ICLR 2020
differentiable activation function is not amenable to backpropagation. (4.2) Gradient updates cease to be meaningful, since the model parameters in QNNs are coarsely quantized.
In this work, combine the ideas of BNNs and QNNs in novel way that addresses the aforemen- tioned challenges (4) in training both models. We propose Bayesian quantized networks (BQNs), models that (like QNNs) have quantized parameters and activations over which they learn (like BNNs) categorical posterior distributions. BQNs have several appealing properties:
BQNs solve challenge (1) due their use of categorical distributions for their model parameters.
BQNs can be trained via sampling-fre frre backpropagation and stochastic gradient ascent of a dif- ferentiable lower bound to ELBO, which addresses challenges (2), (3) and (4) above.
BQNs leverage efficient tensor operations for probabilistic propagation, further addressing chal- lenge (2). We show the equivalence between probabilistic propagation in BQNs and tensor con- tractions (Kolda & Bader, 2009), and introduce a rank-1 CP tensor decomposition (mean-fiell) approximation) that speeds up the forward pass in BQNs.
BQNs provide a tunable trade-off between computational resource and model complexity: using refined quantization allows for more complex distribution at the cost of more computation.
Sampling from learned BQN provides alternative way obtain deterministic QNNs.
In our experiments, we demonstrate the expressive power BQNs. We show that BQNs trained using our sampling-free method have much better-calibrated uncertainty compared with the state- of-the-art Bootstrap ensemble of quantized neural networks (E-QNN) trained by Courbariaux et al.) (2016). More impressively, our trained BQNs achieve comparable log-likelihood against Gaussian Bayesian neural network (BNN) trained with stochastic gradient variational Bayes (SGVB) (Shrid- har et al. 2019) (the performance of Gaussian BNNs are expected to be better than BQNs since they allows for continuous random variables). We further verify that BQNs can be easily used to compress (Bayesian) neural networks and obtain determinstic QNNs. Finally, we evaluate the effect of mean-field approximation in BQN, by comparing with its Monte- Carlo realizations, where no approximation used. We show that our sampling-fre probabilistie propagation achieves similar accuracy and log- le-likkelihodd - justifying the use of mean-field approximation in BQNs.
Related Works. In Appendix A we survey different approaches for training Bayesian neu- ral networks including sampling-free assumed density filtering (Minka 2001; Soudry et al.)
2014 Beenánddez-Lobtoo & Adams 2015:
Ghosh et al., 2016). sampling -haii- variational infer-
Graves 2011] Blundell et al.
2015 Shridhar al. | 2019)), well as sampling varia-
(0166) Shekhovtsov & Flach 2018 Gast & Roth 2018). quantized neural network (Han al.
tional inference (Wu et al.] 2018), probabilistic neural networks (Wang et al.
2015 Courbariaux
et al. 2015) Zhu et 2016 Kim Smaragdis]] 2016) Zhou et al.] 2016]]
Rastegari et al.] [016]]
Hubara et al. [017 Esser
[2015]]Peerr] & Welling 2018: Shayer al.
2017 and tensor net-
works and tensorial neural networks (Grasedyck et al. |013 Orús 2014
Cichocki et al., 2016
2017
Su et
2018 Newman
2018 Robeva & Seigal 2017).
Contributions:
We propose an alternative evidence lower bound (ELBO) for Bayesian neural networks such that optimization of the variational objective is compatible with the backpropagation algorithm.
We introduce Bayesian quantized networks (BQNs), establish a duality between BQNs and hierar- chical tensor networks, and show prediction a BQN is equivalent to series of tensor contractions.
We derive sampling- frren approach for both learning and inference in BQNs using probabilistic propagation (analytical inference), achieving better-calibrated uncertainty for the learned models.
We develop set of fast algorithms to enable efficient learning and prediction for BQNs.
2 BAYESIAN NEURAL NETWORKS
Notation. We use bold letters such as 0 to denote random variables, and non non-bold letters such as 0 to denote their realizations. We abbreviate Pr(0 0) as Pr[0] and use bold letters in an equation the equality holds for arbitrary realizations. For example, Pr(æ, y] = Pr(y xx] Pr[z] means Pr(æ x. y Prly y/ aj Pr(æ ss. Var E X, y E )
2

---

Published as a conference paper at ICLR 2020
2.1
PROBLEM SETTING
Given a dataset D = (In: fn)]n= of N data points, we aim to learn a neural network with model
parameters 0 that predict the output y E y based on the input E X. (1) We first solve the learning problem to find an approximate posterior distribution Q(0; () over 0 with parameters such that Q(0: () R Pr(0 (2) We then solve the prediction problem to compute the predictive distribution Pr y (z, DJ for arbitrary input
given (00; 6). For notational simplicity, we will omit the conditioning on D and write Pr(y|z, D] as Pr(y [r] in what follows.
In order to address the prediction and learning problems in BNNs, we analyze these models in their general form of probabilistic graphical models (shown in Figure 3b in Appendix B). Let h 0(0)
and h (1+1) denote the inputs, model parameters, and (hidden) outputs of the /-th layer respectively.
We assume that 00'ss are layer-wise independent, i.e. Q(0; 6) 1= Q(0(; d) and h())
follow the Markovian property, i.c. Pr[h((+]) (((()))))()))))))))))))))))))(() ,0!:") Pr|h(t11) 000)
2.2 THE PREDICTION PROBLEM
Computing the predictive distribution 2r D] with BNN requires marginalizing over the ran- dom variable O. The hierarchical structure of BNNs allows this marginalization to be performed in multiple steps sequentially. In Appendix we show that the predictive distribution of (1+1) given input æ = x can be obtained from its preceding layer (!) by
/ k00 8())
Pr(h(++1|a]]
=
Pr(h0+1))h 000 Q(0() Príh [x) dh do())
(1)
P(h (1110+))000))))
P(h(0)p(0))
This iterative process to compute the predictive distributions layer- sequentially is known as probabilistic propagation (Soudry et al., 2014 Hernández-Lobate & Adams 2015 Ghosh et al. 2016). With this approach, we need to explicitly compute and store each intermediate result Pr(h (1) ar] in its parameterized form (( () (the conditioning on x is hidden in wtt) i.c. ei(()
function æ).. Therefore, probabilistic propagation is deterministic process that computes (1+1) as a function of vi((!) and (000, which we denote wi(1+1) nd (1b101) (00))
Challenge in Sampling-Free Probabilistic Propagation. If the hidden variables h are tinuous, Equation (T) generally can not be evaluated in closed form as it is difficult to find family under the operations of
of parameterized distributions P for (() such that (1+1) remains neural network layer. Therefore most existing methods consider approximations at each layer of probabilistic propagation. In Section 4 we will show that this issue can be (partly) addressed if we consider the h (1'ss to be discrete random variables, as BQN.
2.3 THE LEARNING PROBLEM
Objective Function. A standard approach to finding good approximation Q(0: Φ) is variational from Q(0: Φ) to Pr[O 'D is minimized. In Appendix B we prove that to minimizing the KL-divergence is equivalent maximizing an objective function known as the evidence lower bound (ELBO), denoted as L(Ø).
inference, which finds 6² such that the KL-divergence ( +) PP]
max L(Ø) = - ())L(((())))))))
Σ L. R()))
n=1
where (x( = EQQ [log Pr(ynlen. 0]] and R(Ø)= Eq [log (Pr (Q))
Probabilistic Backpropagation. Optimization in neural networks heavily relies on the gradient- based methods, where the partial derivatives @L(e)/a of the objective L(Φ) w.r.t. the parameters ø are obtained by buckpropagation. Formally, if the output produced by neural network is given by a (sub-) eibbeeiifereet function g()), and the objective L(g(o)) is an explicit function of g()) (and not just explicit function of Φ). then the partial derivatives can be computed by chain rule:
aL(g())) aL(g()))/gg( 0g(0)/86
(3)
3

---

Published as a conference paper at ICLR 2020
The learning problem can then be (approximately) solved by first-order methods, typically stochastic gradient descent/ascent. Notice that (1) For classification, the function g( returns the probabilities after the softmax function, not the categorical label; (2) An additional regularizer R(Ø) the parameters will not cause difficulty in hackpropagation, given 0R(6)/āá is easily computed.
Challenge in Sampling-Fres Free Probabilistic Backpropagation. Learning BNNs is not amenable to standard backpropagation because the ELBO objective function (()) in (bb) is not an explicit (i.e.
implicit) function of the predictive distribution ((6) in (4ap:
In (6) = Eq [Pr[yn]<x 0]] /. Pr(yn| (]Q(0:))d)
(4a)
(6) Eq [log (Pr[yn| 0)) log (Prlyn In, 0)) (0: Φ)
(4b)
Although (o) is function of o, it is not explicit function of gn (o). Consequently, the chain rule Equation (3) on which backpropagation based is not directly applicable.
3 PROPOSED LEARNING METHOD FOR BAYESIAN NEURAL NETWORKS
Alternative Evidence Lower Bound. We make learning in BNNs amenable to backpropagation be obtained by chain rule
by developing lower bound (Ø) S (o) such that ( (i.e. (Ø) is an explicit function of the results from the forward pass.) With (o) in hand, we can (approximately) find ø* by maximizing the alternative objective via gradient-based method:
e = arg max E(Ø) arg max &
R(0) + Σ Enn(d)
(5)
n=1
In Appendix C.I we proved one feasible Cn (Ø) which only depends on second last output h (4-1).
Theorem 3.1 (Alternative Evidence Lower Bound). Define each term (,( in L(Ø) as
(Ø) = E u [log (Pr(ymlhh (1(
(6)
then (o) is lower bound of Ln( i.e. In(o) Ln( The equality (o( Ln(d) holds if (L-1) is deterministic given input and all parameters before the last layer L-))-
Analytic Forms of Co). While the lower bound in Theorem 3.1 applies to BNNs with arbitrary distributions P on hidden variables h. Q on model parameters O, and any problem setting (e.g. clas- sification or regression), in practice sampling- can be analytically evaluated (or further lower bounded) in terms of (4--1) and ((L-1), This task nontrivial since it requires redesign of the output layer, i.e. the function of Pr (L-1), g(L-1)).
probabilistic backpropagation requires that (g)
In this paper, we develop two layers for classification and regression tasks, and present the classifi- cation case in this section due to space limit. Since (T) involves the last layer only, we omit the superscripts/subrripss of h (4-1), 25(L-1), g(L--), In. Da. and denote them as h, t, d. x. y-
Theorem 3.2 (Analytic Form of L.( for Classification). Let E RRK (with K the number of classes) be the pre-activations of a softmax layer (a.ka. logits), and ②) = = E R+ + be a scaling factor that adjusts its scale such that Pr(y c/h, exp(he/s) /b/Ek= exp(hx/s). Suppose the logits {hx } 1 are pairwise independent (which holds under mean-field approximation) and h2 follows Gaussian distribution ha (NR, vk) (therefore vi {uz, 1k and is deterministic log (EK= exp (EE 23
parameter. Then En(0) is further lower bounded as (d)
The regression case and proofs for both layers are deferred to Appendix C
4 BAYESIAN QUANTIZED NETWORKS (BQNs)
While Section 3 provides a general solution to learning in BNNs, the solution relies on the ability to perform probabilistic propagation efficiently. To address this, we introduce Bayesian quantized
4

---

Published as a conference paper at ICLR 2020
networks (BQNs)- BNNs where both hidden units hs and model parameters 000s 's take discrete values - along with a set of novel algorithms for efficient sampling- free probabilistic propagation assume activations and model parameters take values
in BQNs. For simplicity of exposition, from the same set Q, and denote the degree of quantization as D = QQ). = (-1,1), 2).
4.1 PROBABILISTIC PROPAGATION TENSOR CONTRACTIONS
Lemma 4.1 (Probabilistie Propagation in BQNs). After quantization, the iterative step of proba- bilistic propagation in Equation I is computed with finite sum instead of an integral:
(+1)) ;
E
h00 Pr(h 1 Q(00) ((0) p (b(1))
(7)
and a categorically distributed h0) results in h (1+1) being categorical as well. The equation holds without any assumption on the operation Pr(h (+D)h! ,00] performed in the neural network.
Notice all distributions in Equation 7 are represented in high-order tensors: Suppose there are input units, J output units, and model parameters at the 1-th layer, then and (1+1) E Q², and their distributions are characterized by (110)) E RDD Q000; ((0)
h1))) Q O(I) QK
RD* (1+11))); 2i(++1) IR"", and Pr[h(((1))6) 001 E I D x xD* respectively. There- fore, each step in probabilistic propagation is a tensor contraction of three tensors, which establishes the duality between BQNs and hierarchical tensor networks (Robeva & Seigal (017).
Since tensor contractions are differentiable w.r.t all inputs, BQNs thus circumvent the difficulties in training QNNs (Courbariaux et al. 2015 Rastegari al. 2016), whose outputs are differentiable w.r.t. the discrete parameters. This result is not surprising: if we consider learning in QNNs as an integer progranming (IP) problem, solving its Bayesian counterpart equivalent the approach to relaxing the problem into continuous optimization problem (Williamson & Shmoys] 2011).
Complexity of Exact Propagation. The computational complexity evaluate Equation (7) is exponential in the number of random variables O(DTJK), which is intractable for quantized neural network of any reasonable size. We thus turn approximations.
4.2 APPROXIMATE PROPAGATION VIA RANK-1 TENSOR CP DECOMPOSITION
We propose a principled approximation to reduce the computational complexity in probabilistic propagation in BQNs using tensor CP decomposition, which factors an intractable high-order prob- ability tensor into tractable lower-order factors (Grasedyck al. 2013). In this paper, we consider the simplest rank-1 1 tensor CP decomposition. where the joint distributions of P and Q are fully factorized into products of their marginal distributions, thus equivalent the mean-field approxima- tion (Wainwright et al. 2008). With rank-1 CP decomposition on (()()), (f ), Vi E [L]. the tensor contraction in reduces to a standard Tucker contraction (Kolda & Bader 2009)
P(h o ((tt))
Σ
k0, h0,,0m gm Pr(h 100, hh II, Q000 o II, P(n(:.0) (8)
where each term of (1 k parameterizes single categorical variable. In our implementation, we store the parameters their log-space, i.e. Q(0(") Q(d)) = exp(u (d)/E=- exp(#) (g)).
Fan-in Number In a practical model, for the 1-th layer, an output unit h ((+1) only (condition- ally) depends on a subset of all input units {h } and model parameters (0(²) according to the con- nectivity pattern in the layer. We denote the set of dependent input units and parameters for h ((+1)
as I ((++1 and M (I+1) and define the fan-in number for the layer as max, Z (((+1) + |
Complexity of Approximate Propagation. The approximate propagation reduces the computa- tional complexity from O(DiJK) to (() which is linear in the number of output units J if we assume the fan-in number E. to be constant (i.e. F. is not proportional to T).
5

---

Published as a conference paper at ICLR 2020
4.3 FAST ALGORITHMS FOR APPROXIMATE PROPAGATION
Different types of network layers have different fan-in numbers E, and for those layers with E greater than small constant, Equation (8) is inefficient since the complexity grows exponential in E. Therefore in this part, we devise fast(er) algorithms to further lower the complexity.
Small Fan-in Layers: Direct Tensor Contraction. If E is small, we implement the approximate propagation through tensor contraction in Equation (8). The computational complexity is O() as discussed previously. See Appendix for a detailed discussion.
Medium Fan-in Layers: Discrete Fourier Transform. If E is medium, we implement approx- imate propagation through fast Fourier transform since summation of discrete random variables is equivalent to convolution between their probability mass function. See Appendix D.2 for details. With the fast Fourier transform, the computational complexity is reduced to O Tog
Large Fan-in Layers: Lyapunov Central Limit Theorem. In typical linear layer, the fan-in E is large, and a super-quadratic algorithm using fast Fourier transform is still computational expensive. Therefore, we derive a faster algorithm based on the Lyapunov central limit theorem (See App D.3 With CLT, the computational complexity is further reduced to O(J
Depending on the fan-in numbers E,, we adopt CLT for linear layers with sufficiently large E such as fully connected layers and convolutional layers; DFT for those with medium E such as average pooling layers and depth-wise layers; and direct tensor contraction for those with small E such as shortcut layers and nonlinear layers.
Remarks:
5 EXPERIMENTS
In this section, we demonstrate the effectiveness of BQNs on the MNIST, Fashion-MNIST, KM- NIST and CIFARIO classification datasets. We evaluate our BQNs with both multi-layer perceptron (MLP) and convolutional neural network (CNN) models. In training, each image is augmented by a random shift within 2 pixels (with an additional random flipping for CIFAR1Ó), and no augmen- tation is used in test. In the experiments, we consider class of quantized neural networks, with both binary weights and activations (i.e. Q = (-1,1}) with sign activations a-) = sign(.). For BQNs, the distribution parameters é are initialized by Xavier's 's uniform initializer, and all models are trained by ADAM optimizer (Kingma & Ba 2014) for 100 epochs (and 300 epochs for CIFARIO) with batch size 100 and initial learning rate 10- which decays by 0.98 per epoch.
MNIST
KMNIST
Rashion-MNIST
CIFARIO
Methods
NLL (10-3)
% Em
NLL (107))
% Ere.
NLL (10-)
% Em
N1L ")
Er.
E-ONN on MLP
546.6=137.9
3.30 40.65
23856644323
17.8818866
2529
13.02-0881
N/A
N/A
BQN on MLP
130.0=+3.5
2.49::0.08
457.7+13.88
13.41:10.12
417.3=8.1
9.99-1-.20
NA
NIA
E-ONN on CNN
425.3-16.88
0.5:10113
3755.714465.1
11.49:11166
1610.74158.4
3.02:10037
7989.7 1 600.2
15.92 I 0.72
BQN CNN
1.6
0.85.10.06
1.5
9.95
209.5
4.65.10.15
530.6
1. 23.0
13.74 10.47
Table 1: Comparison of performance of BQNs against the baseline E-QNN. Each E-QNN is an ensemble of 10 networks, which are trained individually and but make predictions jointly. We report both NLL (which accounts for prediction uncertainty) and 0-1 test error (which doesn't account for prediction uncertainty). All the numbers are averages over 10 runs with different seeds, the standard deviation are exhibited following the + sign.
Training Objective of BQNs. To allow for customized level of uncertainty in the learned Bayesian models, we introduce a regularization coefficient X in the alternative ELBO proposed in Equation (5) (i.e. a lower bound of the likelihood), and train the BQNs by maximizing the following objective:
N
N
Z = Σ (n) + XR(Φ) = 1/A Σ (z) + (
where À controls the uncertainty level, i.e. the importance weight of the prior over the training set.
Baselines. (1) We compare our BQN against the baseline Bootstrap ensemble of quantized neural networks (E-Qu B--NN). Each member in the ensemble is trained in a non-Bo -ayessin way (Courbariaux et al. 2016), and jointly make the prediction by averaging over the logits from all members. Note

---

Published as a conference paper at ICLR 2020
301
127
FONN
KON
RON
1P
Afencied enandd excralar
asoomaiiyy
(a) NLL MNIST
(b) NLL FMNIST
(c) NLL KMNIST
(d) NLL CIFARIO
1-00%
LIG
DN3 MON
BNN PON
00-
Almeted
(e) Error MNIST
( Error FMNIST
gg) Error KMNIST
(h) Error CIFARIO
Figure I: Comparison of the predictive performance of our BQNs against the E-QNN as well as the non-quantized BNN trained by SGVB CNN. Negative log-likelihood (NLL) which accounts for uncertainty and 0-1 test error which doesn't account for uncertainty are displayed.
that Courbariaux al. (2066) is chosen over other QNN training methods as the baseline since it trains QNN from random initialization, thus a fair comparison to our approach. Details are discussed in Appendix A (2) To exhibit the effectiveness of BQN, further compare against continuous- valued Bayesian neural network (abbreviated as BNN) with Gaussian parameters. The model is trained with stochastic gradient variational Bayes (SGVB) augmented by local re-parameterization trick (Shridhar et al., 2019). Since the BNN allows for continuous parameters (different from BQN with quantized parameters). the predictive error expected to be lower than BQN.
Evaluation of BQNs. While 0-1 test error is popular metric to measure the predictive perfor- mance, too coarse a metric to assess the uncertainty in decision making (for example does not account for how badly the wrong predictions are). Therefore, we will mainly use the negative log-likelihood (NL..) to measure the predictive performance in the experiments.
Once BQN is trained (i.e. an approximate posterior Q(0) is learned), we consider three modes to evaluate the behavior of the model: (1) analytic inference (AI), (2) Monte Carlo (MC) sampling and (3) Maximum Posterior (MAP) estimation:
In analytic inference (AI, i.e. our proposed method), we analytically integrate over (()) to obtain the predictive distribution in the training phase. Notice that the exact NLL is not accessible with probabilistic propagation (which is why we propose an alternative ELBO in Equation (5)) will report an upper bound of the NLL in this mode.
In MC sampling, S sets of model parameters are drawn independently from the posterior pos- terior 0g Q(0),
Vs Vs [S]. and the forward propagation is performed in (non- moon-Bayeiian) quantized neural network for each set 0s. followed by an average over the model outputs. The difference between analytic inference and MC sampling will be used to evaluate (a) the effect of mean-field approximation and (b) the tightness of the our proposed alternative ELBO.
MAP estimation is similar to MC sampling, except that only one set of model parameters 0* is obtained 0* arg maxę Q(0). We will exhibit our model's ability to compress a Bayesian neural network by comparing MAP estimation of our BQN with oon-aassian QNN.
5.1 ANALYSIS OF RESULTS
Expressive Power and Uncertainty Calibration in BQNs. We report the performance via all evaluations our BQN models against the Ensemble- ONN Table | and Figure [ (1) Com- pared to E-QNNs, our BQNs have significantly lower NI.L and smaller predictive emror (except for

---

Published as a conference paper at ICLR 2020
60-
More Cade Sumpplnng
930
Aralytici Infenmec Uniferenee
900
60-
Masie Carlo Samplny
Moric Carla Samplay
Asalyccal inforrace
Asalyical Inferesez
Dillerowee
100
Dillocwee
30
10
107
10"1
10-1
10
10"
10
10"3
maddll uncorsintty
uncersintt
model uncorsiatt
(a) NLL MNIST
(b) NLL FMNIST
(C) NLL KMNIST
0.5
Sampling
Rosanage
Sampling Dillacesee
Sampling Dillizzaaee
0.4
02-
2-
10-1 Aeve. modd uncertanty
Jovelll medel uncertantt
Aeve model uncerianty
(d) Error MNIST
(e) Error FMNIST
(f) Error KMNIST
Figure 2: Illustration of mean- field approximation and tightness of alternative ELBO on a CNN. The performance gap between our analytical inference and the Monte Carlo Sampling is displayed.
Fashion- MNIST with architecture CNN). (2) As we can observe in Figure I BQNs impressively achieve comparable NLL continuous-valuee BNN, with slightly higher test error. As our model parameters only take values {-1, 1}. small degradation in predictive accuracy is expected.
Evaluations of Mean-field Approximation and Tightness of the Alternative ELBO. If analytic inference (by probabilistic propagation) were computed exactly, the evaluation metrics would have been equal to the ones with MC sampling (with infinite samples). Therefore we can evaluate the approximations in probabilistic propagation, namely mean-field approximation in Equation (8) and relaxation of the original ELBO in Equation (5). by measuring the gap between analytic inference and MC sampling. As shown in Figure 2] such gaps small for all scenarios, which justifies the approximations we use in BQNs.
To further decouple these factors of mean-field approximation and relaxation of the original ELBO, we vary the regularization coefficient À in the learning objective. (1) For X = 0 (where the prior term is removed), the models are forced to become deterministic during training. Since the deterministic models do not have mean-field approximation in the forward pass, the gap between an- alytic inference and MC-sampling reflects the tightness of our alternative ELBO. (2) As À increases, the gaps increases slightly as well, which shows that the mean- mean-field approximation becomes slightly less accurate with higher learned uncertainty in the model.
MNIST
KMNIST
Fashiso-MNIST
Methods
NLI(10-3)
%
NLL(10-3)
S Err.
NLI,110"))
% Eir.
QNN on MLP
522.4=4222
4.14:1::5
2019.1
28122
19.56:11.97
193.5
15.67=1.19
MAP BQN MLP
137.60
3,69 1.009
464.60
12,80
14.79
461,30 13.40
12.89
QNN on CNN
139.4
1.08::0.2
4734.5
1697.2
14.2422.9
1878.3=223.8
3.881:0,33
MAP of BQN on CNN
3.6
0.92:10.07
293.6.14.4
10.82.10.37
179.1:14.4
5.00110.11
Table 2: Deterministic model compression through direct training of QNN (Courbariaux et al.
2016) v.s. MAP estimation our proposed BQN. All the numbers are averages over 10 runs with different seeds, the standard deviation are exhibited following the + sign.
Compression of Neural Networks via BQNs. One advantage of BQNs over continuous-valued
BNNs is that deterministic QNNs can be obtained for free, since BQN can be interpreted as an ensemble of infinite QNNs (each of which is realization of posterior distribution). (1) One simple approach is to set the model parameters to their MAP estimates, which compresses a given BQN to 1 1/64 of its original size (and has the same number of bits single QNN). (2) MC sampling be
8

---

Published as a conference paper at ICLR 2020
MINSTT
KMNIST
Fashioo-MMNSS
Methods
NLL(10
% Err.
NLL(11
% Etr.
NLL )
% Err.
E-QNN
146.60
2385.60-1-432.30
1.88
2061276.70
13.02.10.81
MC of BQN on MLP
108.9422.6
2.73:20009
429.50:
13.83=0112
385.30=55.10
10.81=0.444
E. QNNN on CNN
425.3:161.88
0.85
3755, 70.1 465.10
1.14
1610.70.1158.44
3.02.10.37
MC of BQN CNN
29.2:1:066
0.87::004
286.142.77
10.56=0.14
174.5:::66
4.82::0.13
Table 3: Bayesian Model compression through direct training of Ensemble-QNN vs a Monte-Carlo sampling on our proposed BQN. Each ensemble consists of 5 quantized neural networks, and for fair comparison we use samples for Monte- Carlo evaluation. All the numbers are averages over 10 runs with different seeds, the standard deviation are exhibited following the + sign.
interpreted another approach compress a BQN, which reduces the original size its S/64 (with
the same number of bits as an ensemble of S QNNs). In Tables 2 and 3 we compare the models by both approaches to their counterparts (a single QNN for MAP, and E- E-NN for MC sampling) trained from scratch as in Courbariaux et al.] (2016). For both approaches, our compressed models outperform their counterparts (in NLL) We attribute this to two factors: (a) QNNs are not trained in Bayesian way, therefore the uncertainty is not well calibrated; and (b) Non-differentiable QNNs are unstable to train. Our compression approaches via BQNs simultaneously solve both problems.
CONCLUSION
based approach for learning Bayesian quantized neural networks (BQNs). We develop suite of algorithms for efficient inference in BQNs such that our approach scales to large problems. We evaluate our BQNs by Monte-Carlo sampling, which proves that our approach is able to learn proper posterior distribution on QNNs. Furthermore, we show that our approach can also be used learn (ensemble) QNNs by taking maximum a posterior (or sampling from) the posterior distribution.
We present sampling-free, lkkkproppgatiioncoomaaiiiii
REFERENCES
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint /3XXiv:300.... 3432, 2013.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv preprint arXiv: 2015.
Andrzej Cichocki, Namgil Lee, Ivan V Oseledets, Anh Huy Phan, Qibin Zhao, and Mandic. Low-rank tensor networks for dimensionality reduction and large-scale optimization problems: Perspectives and challenges part 1. arXiv preprint 1609 00893, 2016.
Andrzej Cichocki, Anh-Huy Phan, Qibin Zhao, Namgil Lee, Ivan Oseledets, Masashi Sugiyama, Danilo P Mandic, et al. Tensor networks for dimensionality reduction and large-scale optimiza- tion: Part 2 applications and future perspectives. Foundations and Trends in Machine Learning, 9(6):331-67. -733, 2017.
Matthieu Courbariaux, Yoshua Bengio, and Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems, pp. 3123-3131, 2015.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniy, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ or-1. arXiv preprint arxiv:1602.022300 2016.
Steve K Esser, Rathinakumar Appuswamy, Paul Merolla, John V Arthur, and Dharmendra S Modha. Backpropagation for energy-efficieet neuromorphic computing. Advances in Neural Informa- fion Processing Systems, pp. 11 1117-1123 125, 2015.
Yarin Gal. Uncertainty in Deep Learning, PhD thesis, University of Cambridge, 2016.
Jochen Gast and Stefan Roth. Lightweight probabilistic deep networks. In Proceedings of the IEEE Conference on Computer Vision and Patter Recognition, pp. 3369-3378, 2018

---

Published as a conference paper at ICLR 2020
Soumya Ghosh, Francesco Maria Delle Fave, and Jonathan S Yedidia. Assumed density filtering methods for learning bayesian neural networks. In AAAI, pp. 1589-1595, 2016.
Lars Grasedyck, Daniel Kressner, and Christine Tobler. A literature survey of low-rank tensor 36(1):53-78, 2013.
approximation techniques.
Alex Graves. Practical variational inference for neural networks. In Advances in neural information processing systems, pp. 2356, 2011.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.0014 2015.
José Miguel Hernández-Lobato and Ryan Adams. Probabilistic backpropagation for scalable learn- ing of bayesian neural networks. In International Conference on Machine Learning, pp. 1861- 1869, 2015.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran EI- Va-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. Journal of Machine Learning Research, 18:187-1,2011
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparamcterizaation with gumbel-softuax, arXiv preprint arXiv:1611.0114 2016
Mohammad Khan. Variational learning for latent Gaussian model of discrete data. PhD thesis, University of British Columbia, 2012.
Minje Kim and Paris Smaragdis. Bitwise neural networks. arXiv preprint arXiv:1601.00071, 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412:::412.6980. 2014.
Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 51()): 455-500, 2009.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv: 1.007 2016.
Thomas Peter Minka. A family of algorithms for approximate Bayesian inference. PhD thesis, Massachusetts Institute of Technology, 2001.
Elizabeth Newman, Lior Horesh, Haim Avron, and Misha Kilmer. Stable tensor neural networks for rapid deep learning, 2018
Román Orús. A practical introduction to tensor networks: Matrix product states and projected entangled pair states. Annals of Physics, 349:117-158, 2014.
Jorn WT Peters and Max Welling. Probabilistic binary neural networks. arXiv preprint 18X.008800,,336,, 2018.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. European Conference on Computer Vision, pp. 522- Springer, 2016.
Elina Robeva and Anna Seigal. Duality of graphical models and tensor networks. Information and Inference: A Journal of the IMA, 2017.
Oran Shayer, Dan Levi, and Ethan Fetaya. Learning discrete weights using the local reparameteri- zation trick. arXiv preprint arXiv: 1710.0777, 2017.
Alexander Shekhovtsov and Boris Flach. Feed-forward propagation in probabilistic neural networks with categorical and max layers. 2018.
Kumar Shridhar, Felix Laumann, and Marcus Liwicki. A comprehensive guide to bayesian convo- lutional neural network with variational inference. arXiv preprint arXiv: 1901.0231, 2019.
10

---

Published as a conference paper at ICLR 2020
Daniel Soudry, Itay Hubara, and Ron Meir. Expectation backpropagation: Parameter-free train- ing of multilayer neural networks with continuous or discrete weights. In Advances in Neural Information Processing Systems, pp. 963-971, 2014.
Jiahao Su, Jingling Li, Bobby Bhattacharjee, and Furong Huang. Tensorized spectrum preserving compression for neural networks. arXiv preprinl arXiv: 10352, 2018.
Martin Wainwright, Michael I Jordan, et al. Graphical models, exponential families, and variational inference. Foundations and Trends()) in Machine Learning, 1(1-2):1-305, 2008.
Hao Wang and Dan Yeung. Towards bayesian deep learning: A survey. arXiv preprint arXiv: 16.4. 1604.0166,, 2016.
Hao Wang, SHI Xingjian, and Dit- Yan Yeung. Natural-parameter networks: A class of probabilistic neural networks. In Advances in Neural Information Processing Systems, pp. 118-126, 2016.
David P Williamson and David B Shmoys. The design of approximation algorithms. Cambridge university press, 2011.
Anqi Wu, Sebastian Nowozin, Edward Meeds, Richard Turner, José Miguel Hernández-Lobato, and Alexander L Gaunt. Fixing variational bayes: Deterministic variational inference for bayesian neural networks. arXiv preprint arXiv:1810.039958 2018.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06166, 2016.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. arXiv preprint arXiv:1612.01064 2016.

---

Published as a conference paper at ICLR 2020
Appendix: Sampling-Free Learning of Bayesian Quantized Neural Networks
A RELATED WORK
Probabilistic Neural Networks and Bayesian Neural Networks These models consider weights to be random variables and aim to learn their distributions. To further distinguish two families of such models, call model Bayesian neural network if the distributions
learned using prior-posterior framework (i.c. via Bayesian inference) (Soudry et al. 2014 Hernándezz Lobato &
Adams 2013 Ghosh et al. 2016 Graves 2011 Blundell et al. 2015| [Shridhar et al.] (2199), and
otherwise probabilistic neural network (Wang et al. 2016; Shekhovtsov & Flach] 2018 Gast &
Roth 2018). In particular, our work is closely related antural-parameter networks (NPN) (Wang
et al. 2016), which consider both weights and activations to be random variables from exponential
family. Since categorical distribution (over quantized values) belongs to exponential family, our
BQN can be interpreted as categorical NPN, but we learn the distributions via Bayesian inference.
For Bayesian neural networks, various types of approaches have been proposed learn the posterior distribution over model parameters.
(1) Sampling free Assumed Density Filtering (ADF), including EBP (Soudry et al., 2014) and PBP Biernandez-Lobato & Adams 2015). is an online algorithm which (approximately) updates the posterior distribution by Bayes' rule for each observation. the model parameters 0 are Gaus- sian distributed, Minka (2001) shows that the Bayes' rule can be computed in analytic form based on (+)) ( and EBP EBP [oudry al. (2014) derives a similar rule for Bernoulli parameters in binary classification. Notice that ADF is compatible to backpropagation:
Blog(g.(0))
1
(Φ)
(10)
06
In (0)
06
assuming gn (6) can be (approximately) computed by sampling-free probabilistic propagation as in Section 2 However, this approach has two major limitations: (a) the Bayes' rule needed to be derived case by case, and analytic rule for most common cases are not known yet. (b) it is not
compatible to modern optimization methods (such SGD or ADAM) as the optimization is solved analytically for each data point, therefore difficult to cope with large-scale models.
(2) Sampling- -based Variational inference (SVI), formulates an optimization problem and solves it approximately via stochastic gradient descent (SGD). The most popular method among all is, Stochastic Gradient Variational Bayes (SGVB), which approximates L,, by the average of multi- ple samples (Graves)) 2011 Blundell et al. 2015 Shridhar et al. 2019). Before each step of learning or prediction, number of independent samples of the model parameters [0,)
drawn accord- ing to the current estimate of Q, i.e. 0g ~ Q, by which the predictive function gn (Ø) and the loss L, (Ø) can be approximated by
S Pr(ym I2n, 0z] = 5
s
(d)
~
Σ
fr.).)
(11a)
1=1
==1
1 Σ
(0) ~
log (Pr(yn 2n 0,))))
log (f.))
(11b)
s
where f0) Pr(yn Fn, denotes the predictive function given specific realization 0 of the model parameters. The gradients of L, (Ø) can now be approximated as
(C) 0
1 s (c) (f.)
O..
00. 0
(12)
00,
This approach has multiple drawbacks: (a) Repeated sampling suffers from high variance, besides being computationally expensive in both learning and prediction phases; (b) While g. (0) is differ- entiable w.n.t o. (f) may not be differentiable w.r.t. 8. One such example quantized neural
networks, whose backpropagation is approximated by straight through estimator (Bengio et al.
12

---

Published as a conference paper at ICLR 2020
2013]. (3) The partial derivatives 00./0g are difficult to compute with complicated reparametert- zation tricks (Maddison et al. 2016; Jang et al. 2016).
(3) Deterministic Variational inference (DVI) Our approach is most similar to Wu et al. 2018 which observes that if the underlying model is deterministic, i.e. Pr(h(++))) (n " 09 is a dirac function
Ln Ep D.P; alc n~Q log (Pr(y, (1(-1)) 0 11]
(13)
Our approach considers a wider scope of problem settings, where the model could be stochastic, i.e. Pr(h (1+1) (r(h(++) (00) ( is an arbitrary function. Furthermore, Wu et al. (2018) considers the case that all parameters θ are Gaussian distributed, whose sampling-free probabilistic propagation requires complicated approximation (Shekhovtsov & Flach 2018).
Quantized Neural Networks These models can be categorized into two classes: (1) Partially
quantized networks, where only weights are discretized (Han et al. 2013) Zhu al. 2016): (2)
Fully quantized networks, where both weights and hidden units are quantized (Courbariaux et al.) 2013 Kim & Smaragdis, 2016, /Zhou er al. 2016 Rastegari et al. 2016 Hubara et al.] 2017). While both classes provide compact size, low-precision neural network models, fully quantized networks further enjoy fast computation provided by specialized bit-wise operations. In general, quantized neural networks are difficult to train due to their non-dilferentiaailityy Gradient descent by back- propagation is approximated by either straighttthrouggh estimators (Bengio et al. 2013) or prob- abilistic methods (Esser et al., 2013; [haaeer et al., 2017 Peters & Welling 2018). Unlike these papers, we focus on Bayesian learning of fully quantized networks in this paper. Optimization of quantized neural networks typically requires dedicated loss function, learning scheduling and initial- ization. For example, |etters & Welling (2018) considers pre-training of a continuous-val valuud neural network as the initialization. Since our approach considers learning from scratch (with an uniform initialization), the performance could be inferior to prior works in terms of absolute accuracy.
Tensor Networks and Tensorial Neural Networks Tensor networks (TNs) are widely used in numerical analysis (Grasedyck et al,, (013). quantum physises (Orús) 2014), and recently machine learning (Cichocki et al. 2016 2017) to model interactions among multi-dimensional random ob- jects. Vañooss tensorial neural networks (TNNs) (Su et al. 2018: Newman et al., 2018) have been proposed that reduce the size of neural networks by replacing the linear layers with TNs. Recently, (Robeva & Seigal (017)) points out the duality between probabilistic graphical models (PGMs) and TNs. Lee there exists bijection between PGMs and TNs. Our paper advances this line of thinking by connecting hierarchical Bayesian models (e.g. Bayesian neural networks) and hierarchical TNs.
B SUPERVISED LEARNING WITH BAYESIAN NEURAL NETWORKS (BNNs)
The problem settings general Bayesian model and Bayesian neural networks for supervised learn- ing are illustrated in Figures 3a and 3b using graphical models.
2/2
JUN
a
RRI
Ø
aA
F
JN
Graphical model depiction of Bayesian neural network as a hierarchical model, where predicting y from æ can be performed iteratively through the hidden variables (!'ss.
(a) Graphical model depiction of the problem setting in Bayesian neural networks.
Figure 3: Graphical models.
General Bayesian model Formally, the graphical model in Figure BauureBalinne imiplls the joint distribution of the model parameters 0, the observed dataset D { Dn) ) and any unseen data point
13

---

Published as a conference paper at ICLR 2020
(a, y) is factorized as follows:
Pr(æ, y. D, 0 = (Priylæ, Pr[[[ (Pr[D[o]) Pr(0]
(14)
Priy|æ 0 Pr(z]
I Pr(vn En 0|pr(zm)
Pr(0]
(15)
where Pr[z;]'s and Pr(æ] are identically distributed, and so are the conditional distributions Prfy 0J's and Pr(y (z, 0). In other words, assume that (1) the samples (xn, Vn)'s (and unseen data point (æ, y)) are are identical and independent distributed according to the same data distribution; and (2) In (or æ) and 0 together predict the output yn. (or y) according the same conditional distribution. Notice that the factorization above also implies the following equations:
Pr x. D, 0] Pr(y at, 0)
(16a)
Pr D] = Pr
(16b)
With these implications, the posterior predictive distribution Pr(y a D) can now expanded as:
Pr(y/æ, D) = Prly ,r D, Did0 Priylæ, 0] Pr[ do
(17)
=Q
where we approximate the posterior distribution Pr[ by parameterized distribution Q(0; ()-
Variational Learning The reason we are learning an approximate posterior Q and not the exact distribution Pr(0 D is that for complex models the latter is intractable to compute. The exact posterior Pr(O generally does not take the form Q(0; Φ) even its prior Pr(0) does.
A standard approach to finding a good approximation (0; o) is variational inference, which finds 6* such that the Kl-divergence (L(Q((; +)] ))[(Pr(() of ((0; () from Pr[0[D] is minimized (or alternatively the negative KL-divergence is maximized.)
Φ² arg max +
[LL(QQ(0 +) (r((D)))
(18)
Q(6:0)
arg max +
log
dB
(19)
Pr
where Pr is obtained via standard Bayes' rule, i.e. Pr[0[D] Pr Pr(0//P(() Now
we are able to decompose the maximization objective into two terms by plugging the rule into (19]
((0)= 1 Q(0; o) log (((::))
Pr(D)
do
(20)
Pr(0
N
Q(0;
log (Pr[ynlz xn, Q(0; d0 + Q(0 6) log
d0 + const.
(21)
Pr[0]
==1
N
Eg [log (Pr(ynlen. 01) KL Q(0 Φ) Pr(o)) - log (Pr(D))
(22)
(
const.
where (1) (o) is the expected log-likelihood, which reflects the predictive performance of the Bayesian model on the data point (In, yn); and (2) R(6) is the KL-divergence between Q(0; and its prior Pr(O), which reduces to entropy H (Q)) if the prior of 0 follows a uniform distribution.
Hierarchical Bayesian Model Bayesian neural network can be considered as hierarchical Bayesian model depicted in Figure 3b which further satisfies the following two assumptions:
Assumption B.1 (Independence of Model Parameters (())) The approximate posterior Q(0; Φ) over the model parameters are partitioned into disjoint and statistically independent layers (Q L-1 (where each () parameterizes ( the 1-th layer) such that:
L-1
Q(0:0) = IIQ ((00))00)
(23)
1=0
14

---

Published as a conference paper at ICLR 2020
Assumption B.2 (Markovianity of Hidden Units h )). The hidden variables h (hx0)/ li-o sat- isfy the Markov property that h(++) depends the input a: only through its previous layer h(;;
Pr(h (+ G00 00:01 Prih (r (100 80)
(24)
where we use short-hand notations (:1) and ((:1)
1) to represent the sets of previous layers (h( y 1k=0
and (0 For consistency, we denote h (0) x and h( y.
Based on the two assumptions above, we provide a proof for probabilistic propagation in Equation ① as follows:
Proof of probabilistic prorogation
P(1 (041)) (404))
Pr|h0t11|a/
Pr(h 01:01 (ot dol.1)
(25)
Jac.n
= -Soo Uno Prih no 000 Prip) 0c: ( agc.))
(26)
So Pr(m ( 100000000 )
(27)
Vor.. Prih lr 0 - 10(00 ,); ((1-11))))) (d00111)) dh a00
= -I
1,00000 Pr|h ,0010(000; Pr(l((1)) dl (dow)
(28)
P(NSt); wit)
C ALTERNATIVE EVIDENCE LOWER BOUND AND ITS ANALYTIC FORMS
C.I ALTERNATIVE EVIDENCE LOWER BOUND (PROOF FOR THEOREM 3.T
The steps prove the inequality
6 almost follow the ones for probabilistic propagation above:
(d) EQ [log( Pr(yy (x,,, 0)]
(29)
J. log (Pr(ym F ());
(30)
= J log (Ar Priyn, h (1) 1n,,
()dh((-t) (2n
Q(0: o)d0
(31)
| log (Arc- Priyn 11-1 101-110111111 Pr (1- p glo. 2)1a @)
(32)
(nO-- log Prryni/h (0 Pr In glu.L (1ah4-1))) Q(0: 6)d0
(33)
Sac---.0 ((0-1)00111)) log Prryy,hh (111)/1001))0001111 ((- o((-1))
(34)
( ( Pr(m((-1) In, geu.L -0:22-2)1110000000100 2(00001 r0l -2) 27) dh (L dh 111444111
JK log (Pr(yn/h ,00--11) Q(0(4-1))Pr(M (
2)
(35)
= =ENG-->>P, u log (Priy.ln(ee o ( =Zn(o)
(36)
where the key is the Jensen's inequality Reg [log(-)] > log (EQ (-]) in Equation 33 Notice that if ((4-1) is not random variable (typical for output layer), Cn (0) can be simplified as:
(Ø) (0))==)[(a))
SNa. log (Pr(unl1((-1)) ((L--11)) ((b(
() (6(1)1)) )a1 (Lh( (L-1)
(37)
15

---

Published as a conference paper at ICLR 2020
where write Pr[h(4-11/a)) in its parameterized form P(h!lt"); Now, the gradient (Cn( (0) can be obtained by differentiating over Equation 37 while other gradients 2Cn (0) (0))//(12))) L-2) further obtained by chain rule:
(0) = dxbl--1) :o(:
(0) 0xbil-1)
(38)
06(:1-2)
which requires us to compute ( /Obit and (ei(!--1) /00l /0 1--2), While aI, (o)
can be derived from Equation (37), 0i.(1-1 (1) L-)) can be obtained by backpropagating outputs of the (L - 2)th layer obtained from probabilisticc propagation in Equation (1) In other words: since P(h(T-1); 1); 2 (((-1)) is an intermediate step of the forward pass, eb(l--1) is a function of all
parameters from previous layers of: and each step ei(+1) = (x>(1)>) (0))) is differentiable w.r.t. w)(() and Φ()), the partial derivatives 0y((L-1)
(0gl1 /0gl:
can be obtained by iterative chain rule.
C.2 SOFTMAX LAYER FOR CLASSIFICATION PROBLEM
In this part, we first prove the alternative evidence lower bound (ELBO) for Bayesian neural net- works with softmax function as their last layers. Subsequently, we derive the corresponding back- propagation rule for the softmax layer. Finally, we show a method based on Taylor's 's expansion to approximately evaluate a softmax layer without Monte Carlo sampling.
Theorem C.1 (Analytic Form of (Ø) for Classification). Let h E RK (with K the number of classes) be the pre-activations of a softmax layer (a.ka. logits), and á = s E R be a scaling factor that adjusts its scale such that Pr(y = hhh s) = exp(ha/s) >>//>kn k= exp(ha//s). Suppose the logits {he are pairwise independent (which holds under mean-field approximation) and h2 follows a Gaussian distribution h& ~ N(r,, uk) (therefore v/ = {uu,, Uk ) k=z)) and s is deterministic parameter. Then (o) can be further upper bound by the following analytic form:
K
(n(
2 Pe log
Σ
exp
MR
1/1
L(()
(39)
2s
Proof. The lower bound follows by plugging Pr(y (h, s) and Pr [2] into Equation (6
(o) = S. log (Prlyn = =c]h;,) s]) Pr[M|/]]dh
(40)
K = =/1 he -log Σ
K II Jm
hi
exp
Pr [hx]22]] dh
(41)
Aml
s )
1
Jhe h hePr(hic (a
log
exp
h+
II Pr
Pr[he|a]
dh
(42)
K
K
He
jn log
Σ
exp
h+k
Pr
[x]
dh
(43)
8
K
K
2
Me
log
exp
I k=1
Pr
Tha
[c]
dh
(44)
k=
He
log
Σ
exp
hx
Pr
(45)
8
s
k=
He
K
(he -1xx)
log
exp
exp
dhk
(46)
8
h/
8
2ix
K
He
log
exp
1/k
)
L(O)
(47)
2x
16

---

Published as a conference paper at ICLR 2020
where the last equation follows
hkk
1
(h+
ma)2
exp
exp
dhi
(48)
hx
5
V2TVK
20k
h
2(14x
V/k s///s + H4
exp
dhi
(49)
hx
20k
(h+ (HL + xx)))
[4k
1/2
exp
dhy
exp
+
(50)
hy.
V 12TV
2/1
2x
where the under-braced term is unity since it takes the form of Gaussian distribution.
From Equation (43) to (44), we use the Jensen's 's inequality to achieve a lower bound for integral of log-sum-exp. The bound can be tighten with advanced techniques in Khan (2012).
To use probabilistic backpropagation to obtain the gradients w.r.t. the parameters from previous layers, we first need to obtain the derivatives w.r.t. vi(4-1) {Uk, k
Derivatives of (o) in 39
(L)(#)
exp (Ja/s / + vz/25²²
1[k =cl
(51a)
0pk
Ek=1 exp (/K/ + vk/22s2)
(C)
exp (ML/ s //5 2 2s²))
(51b)
Ouk
28
ER=1 exp (1x/s - 1/k /2s2)
Furthermore, the scale s can be (optionally) updated along with other parameters using the gradient
aL,(o) Me
+ 2k=1 s2 va/s") exp (Ha/s vx/2s²)
(52)
Ōs
82
Σ exp (uu/s + Uk v
Prediction with Softmax Layer Once we learn the parameters for the Bayesian neural network, in principle we can compute the predictive distribution of y by evaluating the following equation:
Pr(y c/ao] = / Prly c/h, (/Pr(hla
S. (a(h)Pr(h)a)
(53)
K
(Mean-field assumption)
/ hx
(h(
II Pr(hu/l)) dhi dhk
(54)
exp(ha/s)] Unfortunately. the equation above can not be computed in closed form. The most straight- forward work-around is to approximate the integral by Monte Carlo sampling: for each he we draw S samples {hi]a independently and compute the prediction:
where we denote the softmax function as (c(h) exp(he/s)//ii
S Prly = c|x] = S Eee(h) Vc E [K]
(55)
Despite its conceptual simplicity, Monte Carlo method suffers from expensive computation and high variance in estimation. Instead, we propose economical estimate based Taylor's expansion. First, we expand the function (h) by Taylor's series at the point (up to the second order):
(ahh) = (c(m) +
Tle Oh
1 [m)] (h - (4) 2
10-11 T 20h-
82
(1) (h (h- + (l1h cl
(56)
(c( +
Σ
K
Ila
(1))
(h+ pk) +
K
K
8l
(m) (hr (hi- ms)(
(ha-mi)(hp - M3) + +0 (11h 1lll3)
ahv
Thihy
km
i=1
(57)
17

---

Published as a conference paper at ICLR 2020
Before we derive the forms of these derivatives, first show the terms of odd orders do not tribute to the expectation. For example, if (h( is upproximated by its first two terms (i.e. a linear function), Equation (54) can be written
(h.".Ao (alu)+
K
Ole (n)) (Ne: - me) Thu
[II Pr (rrhalla)
Pr(y - -cx]== 20
+
Σ k=1
dhi ... dhg
(58)
Ola
(m))
(1
= =c(()) +
(he - mee) Pr(ha Pr[hako]dhs = (c(())
(59)
Thg
1=1
where the second term zero by the symmetry of Pr(hg |a]] around ja (or simply the definition of jo s). Therefore, the first-order approximation results exactly in a (deterministic) softmax function of the mean vector μr. In order to incorporate the variance into the approximation, we will need to
derive the exact forms of the derivatives of (h). Specifically, the first-order derivatives are obtained from the definition of (h).
Oe, (h) The
1
exp (ha/8) - exp (2ha/s)
(e.(h) (
(60a)
Ek exp (ha/ss)
Olc (h) The
exp (ha/s) exp (1x/
1 (.))))))))) # #c
(60b)
=k=1 exp (ha/s)
and subsequently the second-order derivatives from the first ones:
8l (h)
1
/Ole,
Hle (1)
(h) (1)
(e.(h) 3l2(1) + 262
(61a)
ah? 8 The
The
1 1(0 Ole (m))x(h) + (h)
Olk The
(h)
(h)
s2 (-Lc(h)(a(M) + (0(())(()))) Vk #c
Oh
The
(61b)
with these derivatives we can compute the second-order approximation as
Prly c|x] R S Jnssh
Hle
K (I dii ... ..d
(o(N) +
(n) pi) ms)
ppipi
k=
(62)
1 16 (1) / (he Pr +
Σ Olee (11) / (ha- 1 1x) Pr | dhx
= Ec((m) +
2 0x2
2
Ouc
Aop
(63)
= -o((m) +
1
(Colu) 36 + 222
Σ
(-Rale)ea()) + 2x(( us
28²
2x
kgC
(64)
K
= ta(p)) + 28 (Cel)) 2(((m)) Ve Σ ex(u)ux
(65)
1=1
The equation above can be further written in vector form as:
Pr[y|æ] vo C(p) + 25 (E(p) (()))) o (()(
(66)
C.3 GAUSSIAN OUTPUT LAYER FOR REGRESSION PROBLEM
In this part, we develop an alternative evidence lower bound (ELBO) for Bayesian neural networks with Gaussian output layers, and derive the corresponding gradients for backpropagation. Despite the difficulty to obtain an analytical predictive distribution for the output, we show that its central moments can be easily computed given the learned parameters.
18

---

Published as a conference paper at ICLR 2020
Theorem C.2 (Analytic Form of (o) for Regression). Let h R be the output of last hidden layer (with I the number of hidden units), and @ = (w, 8) E R x Rt be the parameters that define the predictive distribution output y
1 exp
(y w h)2 2s
Prly s)
(67)
8
K Suppose the hidden units (hg 1==1 are pairwise independent (which holds under mean-field approx- imation), and each h; has mean μu and variance then (()) takes an analytic form:
w 1=) + 2
log(2rs)
(Ø)
(68)
2s
2
where (w = w and μ 1u,,... ur]] E IR! and 1.,..... vrl E R are vectors of mean and variance of the hidden units h.
Proof. The Equation
68 is obtained by plugging Pr hh; w. s) into Equation (6)
(n(
Σ
log (Prly/hi,
w)
II Pr(hslaal
(69)
h1
Ei=1 welts)
log (2T
Σ
I Pr (r]]h/p
(70)
28
2
h1
hr
I
log(2Ts)
y
wih,
(71)
2s
2
h1
by
where the long summation in the first term can be further simplified with notations of j and v:
i
Σ
wihi
(72)
h1
hi
i=
i=l
1
Σ
Σ
2y
wiht +
w wh
Σ
w wywkhgh
I
(73)
h1
=1²
2y
Wi Σ
h Pr[////0]]
+ Σ wa
1 hr Σ
(74)
WyWk
Σ h5
hy Pr/p//hl]
Σ
he Pr
jml
kiffjj
hu
= 2y
Will +
w (1L + vi) +
wy WyWkMyMK
(75)
i=1
i=1
J=1 knf
wiy; + {
-1² 2y
Will +
Σ
Σ wkPk
(76)
1=1
1=1
= 2y w 1 + (2) T + (w 11)
(77)
-Iy w T 2 + 22
(78)
where 02 denotes element-wiss square, i.e. 02 [uof... ... ,w7]
19

---

Published as a conference paper at ICLR 2020
Derivatives of (Ø) in Equation (68 It is not difficult to show that the gradient of (o))) can be backpropagated through the last layer. by computing derivatives of 2n (d) w.r.t μ and v:
OL, (Ø)
(y w p)w
(79a)
0p
0Ln (d)
w
(79b)
Óv 2s
Furthermore, the parameters {w, 8) can be updated along with other parameters with their gradients:
(y/ w p.) + (w v)
(80a)
Ow
0L (Φ)
1 + (y w 11) (w
(80b)
Ös
2s
2s²
Prediction with Gaussian Layer Once we determine the parameters for the last layer, in principle can compute the predictive distribution Pr[y|[æ] for the output y given the input according to
Pr Priy =
Σ
Prly w, Ses.w,,sPPPii. Pr[h]|]]
Σ h1
Σ
Pr h:; w. s]
II (rr[hslr)
hr
y
E/ i=1 wihss
( [Pr(haaa
exp
(81)
2
2s
Unfortunately, exact computation of the equation above for arbitrary output value y intractable in general. However, the central moments of the predictive distribution Pr(y (r]] are easily evaluated. Consider we interpret the prediction as y w + E, where ~N00, 8), mean and variance can be easily computed as
E y x = w T [h] w 1
(82a)
[ || = T [h] + [E] = (2 v +
(82b)
Furthermore, if we denote the (normalized) skewness and kurtosis of h, as and k;:
= EE [hh 1z)
/v2 3/2
E(h, (h. p4) ( Pr|[hs/=///
(83a)
h4
E [(h: [=]] /
> (h: Pr
(83b)
he
Then the (normalized) skewness and kurtosis of the prediction y are also easily computed with the vectors of y h .7rl"
E R and K = [K1, .. ,k]] R
[y w [1)
(w=3)
(7 203/2
[z]
(84a)
V [1) 3/2
[(w02) + s) 3/2/
(y w T [1)
(N (wod)
((oo))((A)))
202) + 2(W
(84b)
V [2]])]
( + ss²
D PROBABILISTIC PROPAGATION IN BAYESIAN QUANTIZED NETWORKS
In this section, we present fast(er) algorithms for sampling-fres probabilistic propagation (i.e. eval- uating Equation R). According to Section we divide this section into three parts, each part for specific range of fan-in numbers E.
D.1 SMALL FAN-IN LAYERS: DIRECT TENSOR CONTRACTION
E is small, tensor contraction Equation 8 immediately applicable. Representative layers of small E are shortcut layer (a.k.. skip-connection) and what name depth-wise layers.
20

---

Published as a conference paper at ICLR 2020
Shorteut Layer With skip connection, the output h ((++1) is an addition of two previous layers h and (mm) Therefore and the distribution of h (1+1) can be directly computed as
(1+1. P(h;
144 )
8(1 (1+1) h + h 1
P(h !1 ( P(h!T))
(m) ; )
(85)
Dipth-wise Layers In depth-wise layer, each output unit h (1+1) ized by 0 of its corresponding input h i..
is a transformation (parameter-
P(h 4/ ((+)))==))
(+1)
Σ Pr(h 200
I 1ihu+Dnn (1) 0 :0 P(h!):u!)
(86)
Depth-wise layers include dropout layers (where (()) are dropout rates), nonlinear layers (where g(t) are threshold values) or element-wise product layers (where O are the weights). For both shorteut and depth-wise layers, the time complexity is O(JD²) since E 2.
D.2 MEDIUM FAN-IN LAYERS: DISCRETE FOURIER TRANSFORM
In neural networks, representative layers with medium fan-in number E are pooling layers, where each output unit depends on medium number of input units. Typically, the special structure of pooling layers allows for faster algorithm than computing Equation (8) directly.
Max and Probabilistic Pooling For each output, (1) a max pooling layer picks the maximum value from corresponding inputs, i.. h (1+1) max h," while (2) probabilistic pooling
layer selects the value the inputs following a categorical distribution, i.c. Pr(h, ((+1) h 0,.
For both cases, the predictive distribution of h (1+1) can be computed as
Max: P(h. P(h [ = II P(h S
((+1)
(87)
iEI(s)
Prob:
P(h (1+1) 9) Σ
6,P(h ④)
(88)
(ETU)
where P(h (1) S 9) is the culminative mass function of P. Complexities for both layers are O(T)).
Average Pooling and Depth-wise Convolutional Layer Both layers require additions of medium number of inputs. We prove convolution theorem for discrete random variables and show that discrete Fourier transform (DFT) (with fast Fourier transform (FFT) can accelerate the additive computation. We also derive its backpropagation rule for compatibility of gradient- based learning.
Theorem D.1 (Fast summation via discrete Fourier transform). Suppose u, take values in {b,, b, + 1,..., B.....,B between integers b; and B,, then the summation v = _ u, takes values between b and B, where ΣE= b, and B Em Br. Let C", C be the discrete Fourier transforms of p", pw; respectively, i.e.
( Σ (V)))xp((j227 ( -B
(89a)
(a
pa - b, +1)
(89b)
m,=b.
Then C' (f) is the element-wise product of all Fourier transforms C (f). i.e.
E
(* = II C (f).ff
(90)
21

---

Published as a conference paper at ICLR 2020
Proof. We only prove the theorem for two discrete random variable, and the extension multiple variables can be proved using induction. Now consider 41 E [b1, B1], U2 E [bg. B2] and their sum U1 + U2 E [b, Bl. where b b, + b2 and B B + Bz. Denote the probability vectors of u1, U2 and v as P1 E B.-bi, P2 E A BB and P E A B-b respectively, then the entries in P are
computed with P1 and P2 by standard convolution follows:
B1
B2
P(v)
Σ P((w)) - w1) Σ
Pi(v - (2)2)) Ae E ( B}
(91)
N1 =1=
bg=
The relation above is usually denoted as P = Pi * P2, where * is the symbol for convolution. Now define the characteristic functions C, C1, and C2 as the discrete Fourier transform (DFT) of the probability vectors P, Pi and P2 respectively:
2x (v (o-)))) f [R]
c()) Σ P(v) exp
(-)
-
(92a)
B.
C( = Σ Px(u))ep exp
(-) (u, 1 bb)f) .f [R]
(92b)
-,=b;
where R controls the resolution of the Fourier transform (typically chosen as R = B - b + 1, ic. the range of possible values). In this case, the characteristic functions are complex vectors of same length R. i.c. C, C,C,,C C2 E CR, and we denote the (functional) mappings C' = F(P) and
C; = FFFF((P.) (P.). Given a characteristic function, its original probability vector can be recovered by inverse discrete Fourier transform (IDFT):
R-1 P(u) R
(12m 2T ( -))) Ve E (b. ,B}
C(f)
(93a)
f=
1 Σ
Vus (.2T (ui (12 bils): (u==b))f)
P.(u:)=
Ci(f) cxp
(b,...., BB}}
(93b)
R
/=0
(C.). Now we plug the con- volution in Equation (91) into the characteristic function C(f) in (92a) and rearrange accordingly:
which we denote the inverse mapping as P F- and P. F.
2T -j
c(
Σ
Σ
(1( mn)
exp
(u b)s)
(94)
B
(Let U2 = 01)
Pi P2(22) exp
(-)
2T R
(11 + u2 - 815)
(95)
=1=
Bi (u)) exp
2t -j - (u1 - -b))
R
Luz=b1
(Since b1 b2) =
(96)
B2
2n U22 b2) [
Ps (2 exp
R
Luzzb2
=(() C2(
(97)
The equation above can therefore be written as C = C₁ 02, where we use o to denote element- wise product. Thus, we have shown summation of discrete random variables corresponds to clement-wiss product their characteristic functions.
With the theorem, addition of E discrete random variables can be computed efficiently as follows
p p41 puz
Pug
(98)
f (F (P o F (P#2)
F F (PuR)
(99)
where FF denotes the Fourier transforms in Equations (93a) and 93b If FFT is used in computing
all DFT, the computational complexity of Equation (99) is O GERlog log R) = D(E (ED)
(since R O(FD)), compared to O (DD)) with direct tensor contraction.
22

---

Published as a conference paper at ICLR 2020
Backpropagation When fast Fourier transform is used to accelerate additions in Bayesian quan- tized network, we need to derive the corresponding backpropagation rule, i.c. equations that relate ac/OP to {ac/aP.) For this purpose, break the computation in Equation 999 into three steps, and compute the derivative for each of these steps.
C ===(((P)) = dL
0L ac,
R. R.F,
(100a)
aP;
C= 0 01 = 0L
C 0L
ac, = 0
(100b)
C, 0c
7C =R-1
0C
P P=F = 8c
(100c)
aP
where in (000)] we use C/C, to denote element-wise division. Since P. lies into real domain, we need to project the gradients back to real number (100c). Putting all steps together:
OC OP.
R Fr
C FF (c,
aC ViE [4]
(101)
OP
D.3 LARGE FAN-IN LAYERS: LYAPUNOV CENTRAL LIMIT APPROXIMATION
In this part, we show that Lyapunov central limit approximation (Lyapunov CLT) accelerates proba- bilistic propagation in linear layers. For simplicity. consider fully-connected layer in the deriva- tions, but the results can be casily extended to types of convolutional layers. We conclude this part by deriving the corresponding backpropagation rules for the algorithm.
Linear Layers Linear layers (followed by a nonlinear transformations o-)) are the most important building blocks in neural networks, which include fully-comnected and convolational layers. A linear layer is parameterized by set of vectors (() 's and maps h R to h (1-1 E R as
(+1) =a h h
Σ
= T
Σ KETU
= (1941)
(102)
TET())
where us (1) 10 h
from the ones of {u) (1) EiEIg) Uji The key difficulty here is to compute the distri- V i.c. addition of large number of random variables.
and vy
(+11)
bution of vjj (1+1)
Theorem D.2 (Fast summation via Lyapunov Central Limit Theorem). Let ti a(o;) o (Eietti) Oy.us) be an activation of a linear layer followed by nonlinearity a. Suppose both imputs {uijier and parameters heigy have bounded variance, then for sufficiently large (I(j)), the distribution of ú, converges to a Gaussian distribution N N(p D3) with mean and variance as
Pij Σ mjali
(103a)
8, Σ m + ezsti + Uzili
(103b)
where my = E00l. Uy = V [0,] and μ1 = E E[u,), vs = V [u.l And if the nonlinear transform
a is a sign function, each activation v, follows a Bernoulli distribution P(v, 1) = (f VTy), where T is the culminative probability function of a standard Gaussian distribution N 1).
Proof. The proof follows directly from the definitions of mean and variance:
Kg =
E
0 jz hg
E O0, h.]
(104)
1=1
E 10 [ [h.]]
mjalis
(105)
23

---

Published as a conference paper at ICLR 2020
1
Dj = V
Σ
0 h
= Σ 00 hi]
(106)
i=
t=1
(E [0 E [h]]
E
0 E [h
(107)
Σ [mm] + v.) (N, +vse) m3
(108)
Σ (mf.ys enlu + Us:ls)
(109)
For fully-connected layers, these two equations can be concisely written in matrix forms:
= M 5 (M + V (ne² +v) where M and n are element-wise square of M and y respectively.
(110a)
(110b)
Backpropagation With matrix forms, the backpropagation rules that relate aC/03(1+1) {0C/0ū, 0C/00) OC/060) (0C/OM, 0c/0V} and OC/0wll) {aC/0u, 0C 0C/0v] can be derived with matrix calculus.
Oc
0C
(
08 (02 26
T + +M
v
(11la)
0M
OL
0L
(1=2) T
(111b)
σV
OC
0C
VT
0c
M T
+ 2
(111c)
0p
0p
ai
ac
(M
T
OL
(111d)
Notice that these equations do not take into account the fact that V implicitly defined with M (i.e. Uy is defined upon m2j)) Therefore, we adjust the backpropagation rule for the probabilities: denote Qu(d) Q(0,; = Q(d); ( then the backpropagation rule can be written in matrix form as
ac
0L dc
0V
Oc
Ov
(112)
aQ(d)
0M
OV
0M
aQ(d)
OV
oQ(d)
dc
dL
Q(d)
((Q(d) M)
(113)
aV
Lastly, we derive the backpropagation rules for sign activations. Let p3 denote the probability that the hidden unit vy activated as p, Pr(v, 1|r), 0C/p, relates to ( as:
0p;
N
(114a)
0j, Viz
Opj Oū;
(114b)
2/
VDz
E SUPPLEMENTARY MATERIAL FOR EXPERIMENTS
E. NETWORK ARCHIECTURES
(1) For MNIST, Fashion-MNIS" and KMNIST, we evaluate our models on both MLP and CNN. For MLP. use 3-layers network with 512 units in the first layer and 256 units in the second; and
24

---

Published as a conference paper at ICLR 2020
for CNN, use 4-layers network with two 5 x 5 convolutional layers with 64 channels followed
by 2 x 2 average pooling, and two fully-connected layers with 1024 hidden units. (2) For 10, evaluate our models
smaller version of VGG (Peters & Welling 2018), which consists of
6 convolutional layers and 2 fully-connected layers: x 128C3 MP2 X 256C3 MP2 2 512C3 MP2 1024FC SM10.
E.2 MORE RESULTS FOR MULTI-LAYER PERCEPTRON (MLP)
1014
BNN T-ONN IION
1095
1025
10
Jovel of model cnncersiieyy
2 Jevel of model uncersiiity
lovel of model uncceraiiryy
(a) NLL MNIST
(b) NLL FMNIST
(C) NLL KMNIST
12-
Ponstige Time
8-
2-
Arad model
Aevel moded
Arael moded
(d) Error MNIST
(c) Error FMNIST
(f) Error KMNIST
Figure Comparison of the predictive performance of our BQNs against the E- E-QNN as well the non-quantized BNN trained by SGVB MI.P. Negative agellikelhoood (NLL) which accounts for uncertainty and (0- test error which doesn't 't account for uncertainty are displayed.
100
Morta Sumpling
Moria Carla Sumpling
Morta Sumpling Analytical Infemnner Defienunce
80
Aralytical Infernnze
Analytical Inferesoe
Drfienane
Deflerence
60
200
100
100
20
love of model mostr incornitty
lool motel incorointt
lovel of model incorrintt
(a) NLL MNIST
(b) NLL FMNIST
(c) NLL KMNIST
10-
Pancattgg Erne
Moste Cade Sacpplins Anabiial Infercses Diffcieaue
Moste Carlo Suetpling Anahtica Inforcece Difforeace
Poncnntag
G
Moste Cade Sazpplins Anabriaa Infercses Diffcieaee
2-
10
10
10033 sezerterey
10
incenneey
incerteery
(d) Error MNIST
(e) Error FMNIST
( Error KMNIST
Figure 5: Illustration of mean-field approximation and tightness of alternative ELBO on a MLP. The performance gap between our analytical inference and the Monte Carlo Sampling is displayed.
25

---

Published as a conference paper at ICLR 2020
E.3 REGRESSION ON BOSTON HOUSING DATASET
In this part, we evaluate our proposed BQN on Boston housing dataset, a regression benchmark widely used in testing Bayesian neural networks (Hernández-Loobtto & Adams 2015 Ghosh al. 2016) and Probabilistic neural networks (Wang et 2l.])016) The dataset consists of 456 training and 50 test samples, each sample has features as input and scalar (housing) price as output. Following #Heráández-Lobute & Adams (2015); Ghosh et al. (2016); Wang et al. (2016), we train two-layers network with hidden units, and report the performance in terms roo mean square error (RMSE) in Table 4 The results show that our BQN achieves lower RMSE compared to other
models trained a probabllitii/Bayesian way.
Dataset
BQN
PBP Ghosh
2016
EBP (Soudry et al. 2014
NPN Wang 2016
Boston
2.04 1 0.07
2.79 E U.16
3.14 0.93
2.57 NA
Table 4: Performance of different networks in of RMSE. The numbers for BQN averages over 10 runs with different seeds, the standard deviation are exhibited following the + sign. The results for PBP, EBP are from Ghosh et al. (2016), and the one for NPN is from (Wang al. 2016).
26


Reviews:
[{"review_id": "rygNYBzLFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "(1) Contributions:\nThe paper proposes an algorithm for training quantized Bayesian neural networks (BQN). The method optimizes a lower bound to the ELBO that can be analytically computed for BQNs.\n\nThe paper shows that propagation in BQNs is a sequence of tensor contractions, for which there exist fast and efficient approximate algorithms.\n\nBQNs show good performance on set of image classification benchmarks (MNIST, KMNIST, Fashion MNIST).\n\n(2) Originality:\nI am not aware of any prior works that attempts to do inference in quantized networks. The idea of BQNs and propagation as tensor contractions is interesting and novel. It is a cool result that in quantized networks, one can exactly calculate the distribution of the activations (h^(l)) and therefore get an accurate estimate of the ELBO.\n\nIt is also interesting that this algorithm can be implemented as a series of tensor contractions. This leads to a fast and efficient algorithm for training.\n\nRegarding the novelty in the lower bound to the ELBO, variational Bayes makes the same approximation by sampling the layer activations (h^(1) .. h^(L-1)). In VB, when we sample the layer activations, we approximate the logarithm of the expectation by the logarithm of a Monte Carlo sample, which is a lower bound due to Jensen’s inequality. The only difference here is that this approximation is only done in the final hidden layer instead of every layer.\n\n(2) Clarity:\nI want to point out some issues with the writing.\n\nOverall, I believe that the number of equations and theorems hinders the readability of the paper. It is very confusing to try and navigate the various notations and assess the significance of each result. I would recommend including a few key equations and theorems in the paper and leaving the rest to the appendix. The ideas and concepts in the paper could be easily communicated in writing.\n\nI do not think this paper should be 9.5 pages. The work could be comfortably explained in the standard 8 pages. Namely, in Section 3.1 and 3.2, it is unnecessary to provide analytic gradients for the expressions (in the age of automatic differentiation). Section 4.3 should be left to the appendix. I found this section very confusing and it does not seem to be important in the narrative.\n\n(4) Significance:\nI think this work is significant, although I felt that the experiments section could have included more regression benchmarks as well as more difficult datasets such as CIFAR10.\n\n(5) Impact:\nThis work would certainly be interesting to researchers and anyone working on quantized neural networks.\n\nOverall assessment:\nI think this paper presents interesting and novel ideas, but it has shortcomings in the presentation.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SygNIyraKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a set of efficient algorithms for learning and prediction in Bayesian quantized networks, which allows for differentiable learning without the need to sampling. From this point of view, this work targets at solving a real and interesting problem in BNN. I am not an expert in this area, and I can only comment on broad picture.\n\nSome questions:\n-\tThe proposed method is claimed to work on large-scale datasets. However, all datasets selected in experiment are black and white images with low resolution (28 x 28 ?). A ‘real’ large-scale dataset is expected. The current experiment results are far from sufficient.\n-\tThe competitive methods are limited. If the author claims BNN in general can have well-calibrated uncertainties, non-BNN methods should also be considered. Further, when the data has heavy tail or is not Gaussian-distributed, will the model still give well calibrated results?\n-\tDoes this proposed sampling free BQN have faster computational speed than standard BNN? If yes, it is also desired to have a comparison.\n-\tI am totally lost at section 4.3. It seems to give a fast way of computing Eq.(12). A general introduction at the beginning or the end of section 4.3 could be really helpful. Further, I’m not clear why the approximation made in the model is good enough. A pseudo-algorithm is desired for clarification.  \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeXozehKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed Bayesian Quantized Neural Networks (BQN), which is a rough Bayesian treatment for quantized neural networks (QNN). The advantages the BQN over QNN is that it provides better uncertainty estimates as well as accuracy. The disadvantage is that BQN loses the memory efficient property of QNN, that is, BQN in general still needs to store real-value parameters, which parameterize the binary weights. The authors compare BQN to EQNN and show that BQN outperforms the latter.\n\nIn general, the paper is well organized and fairly readable, though the first 2 sections could have been more concise. I appreciate the careful description of notations. \n\nThe topic of Bayesian formulation and inference in the context of QNN is interesting. I like the analysis provided for handling small/medium/large fan-in for Bayesian QNN.\n\nOne of my concerns is that some description in the provided theorems (as well as the proof) is not very rigorous and sometimes misleading. For example, Theorem 3.2 claims that there is an analytic form for Bayesian inference in softmax layers. One would expect that it is either an exact integration over the distribution of h, or a valid bound. Unfortunately, it is neither. Looking at the proof, it seems it is a rather coarse approximation. Note that in the Appendix, inequality in Equation (42) and Equation (47) takes different direction, meaning that this is not a valid bound for the error term L. The authors may want to adjust the description to make it more rigorous and avoid confusion.\n\nThe paper describes BQN in very general terms. It is unclear what distributions are used to parameterize the weights in the experiments. Do you use Bernoulli distributions to parameterize the binary weights {-1, 1}, and learn the real-valued parameters for the distributions during training?\n\nTheorem 3.1 is straightforward. The rough approximation provided in Theorem 3.2 is interesting. It would be interesting to compare this with Gaussian natural-parameter networks, which bypass the softmax problem using multi-logistic regression for classification, where the convolution of Gaussian and sigmoid functions has a closed form.\n\nIn terms of related work, references such as natural-parameter networks (NPN) and its variants (one of which specifically handles the softmax case) are missing [1, 2]. Note that, similarly to BQN, NPN also provides a general probabilistic (Bayesian) framework for sampling-free learning/prediction. Theorem 3.3 is also covered in Section 3.2 of [1].\n\nThis brings me to another of my concerns. The only baseline provided is E-QNN from three years ago. There are various state-of-the-art baselines, either on QNN or BNN. It would certainly make the paper much stronger if such baselines are included, especially when the authors decide to take up the full ten-page space in this submission.\n\n\nMinor:\n\nP2: bf Bayesian -> Bayesian\n\nP2: theta that predicted -> theta that predict\n\nP6: Lemma 4.2 -> Theorem 4.2\n\nP6: by with -> with\n\nAssuming most of my concerns can be addressed during the feedback period, I tend to give a ‘weak accept’ to this submission.\n\nNatural-Parameter Networks: A Class of Probabilistic Neural Networks, NIPS 2016\nFeed-forward Propagation in Probabilistic Neural Networks with Categorical and Max Layers, ICLR 2018", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes Bayesian quantized networks and efficient algorithms for learning and prediction of these networks. The reviewers generally thought that this was a novel and interesting paper.  There were a few concerns about the clarity of parts of the paper and the experimental results. These concerns were addressed during the discussion phase, and the reviewers agree that the paper should be accepted.
Decision: Accept (Poster)