Title: SNAPQUANT: A PROBABILISTIC AND NESTED PARAM-ETERIZATION FOR BINARY NETWORKS

Abstract: In this paper, we study the problem of training real binary weight networks (without layer-wise or filter-wise scaling factors) from scratch under the Bayesian deep learning perspective, meaning that the final objective is to approximate the posterior distribution of binary weights rather than reach a point estimation. The proposed method, named as SnapQuant, has two intriguing features: (1) The posterior distribution is parameterized as a policy network trained with a reinforcement learning scheme. During the training phase, we generate binary weights on-the-fly since what we actually maintain is the policy network, and all the binary weights are used in a burn-after-reading style. At the testing phase, we can sample binary weight instances for a given recognition architecture from the learnt policy network. (2) The policy network, which has a nested parameter structure consisting of layer-wise, filter-wise and kernel-wise parameter sharing designs, is applicable to any neural network architecture. Such a nested parameterization explicitly and hierarchically models the joint posterior distribution of binary weights. The performance of SnapQuant is evaluated with several visual recognition tasks including ImageNet. The code will be made publicly available.

1. INTRODUCTION:

Deep Neural Networks (DNNs) have achieved tremendous success in computer vision Krizhevsky et al. (2012) , natural language processing Collobert et al. (2011) and speech recognition Hinton et al. (2012) . Regardless of the availability of large-scale labeled data and powerful computational resources, the leading accuracies of DNNs are attributed to the huge number of learnable parameters, ranging from a couple of millions to even hundreds of millions. However, this also brings heavy consumption of memory, compute and power resources, which prohibits their use on the resourceconstrained platforms.

Binary Weight Network (BWN) is among the most promising way to ease the deployment of DNNs thanks to its advantages of much smaller storage space and cheaper bit-wise operations over the full-precision (32-bit floating-point) counterpart. Because of this, substantial research efforts have been invested on how to train BWNs either from the pre-trained full-precision models or from scratch. Existing solutions can be grouped into two basic families. The first solution family, such as Soudry et al. (2014) , BinaryConnect Courbariaux et al. (2015) and Binarized Neural Networks (BNNs) Courbariaux et al. (2016) , strictly forces the weight values to be either +1 or -1 from the perspective of approximating the posterior distribution of the weights. The second solution family, such as XNOR-Net Rastegari et al. (2016) , Zhou et al. (2018) and Wan et al. (2018) , directly considers the network binarization problem as a layer-wise approximation of the floatingpoint weight set by a binarized version (that is called a point estimation in this paper). To address notable accuracy degradation, they add filter-wise or layer-wise scaling factors over binarized weights, introducing additional multiplication operations. Our proposed method belongs to the first solution family, but we strive to explore the problem of training BWNs from scratch, by an association of Reinforcement Learning (RL) J William (1992) and Bayesian deep learning for approximating the posterior distribution of binary weights. Although there are some recent research attempts Lin et al. (2017) ; Ashok et al. (2017) ; He & Han (2018) that also use RL to address DNN compression, they focus on how to adapt policy network designs to reduce the complexity of pre-trained network models, mostly from the perspective of weights pruning. Our main contributions are summarized as follows:

• We propose SnapQuant, a reinforcement learning method for training BWNs from scratch under the Bayesian deep learning perspective, which approximates the posterior distribution of binary weights instead of a single point estimation. During the training phase, it generates binary weights in a burn-after-reading style by maintaining a policy network that parameterizes this posterior distribution. At the testing phase, it enables to sample binary weight instances for a given recognition architecture from the learnt policy network.

• We propose a nested parameter structure for the policy network, which explicitly and hierarchically models the joint posterior distribution of binary weights. Such a nested parameter structure has layer-wise, filter-wise and kernel-wise parameter sharing designs, thus it is applicable to any DNN architecture.

• Experiments conducted on the widely used image classification datasets including ImageNet show that SnapQuant has better performance in comparison to related probabilistic methods.

2. RELATED WORKS:

Here, we briefly summarize recent advancements in the related field. . As evaluating the generic posterior P (w|X, Y ) is intractable, we resort to the variational approximation P θ (w). Though there are other possible probabilistic parameterizations for θ, we propose an alternative that directly parameterizes P (w|X, Y ) as a policy network. More specifically, let's denote the parameters of the policy network as θ. As illustrated by the first panel of Fig 1, w is conditioned on θ, or say, P (w|X, Y ) . = P θ (w).

Following the common practice in modern variational approximation Gal & Ghahramani (2016) ; Blei (2016), the objective is to minimize the KL divergence D KL (P θ (w)||P (w|X, Y )). By applying the evidence lower bound (ELBO) theorem, we can get the equivalent objective to minimize -P θ (w) log P (Y |w, X)dw + D KL (P θ (w)||P (w)). Note that these two terms correspond to likelihood (denoted as ∆) and prior (denoted as Γ) respectively. Here a prior that all weights follow a 50%-50% Bernoulli distribution is imposed, thus the prior term Γ is actually an entropy maximization term (which is widely used in policy gradient to encourage exploration). We refer readers to Appendix A for a more detailed exposition for Γ. In the main text, we focus on the non-trivial part ∆. Specifically, a training scheme is shown in the second to fifth panel of Fig 1 . A complete forward-backward propagation iteration for optimizing ∆ is described as follows:

• In the first forward phase, we sample a set of concrete weights w from the distribution P (w|θ). It is worth mentioning that these concrete weights w only exist temporarily.

• In the second forward phase, we do a standard forward propagation on a batch of training data X using concrete weights w. Denoting the network output as Y * and the cross entropy metric as ∆( * , * ), we can evaluate ∆(Y * , Y ).

• In the first backward phase, we do a standard backward propagation, getting gradients w.r.t. concrete weights w. We denote these gradients as ∂∆ ∂w . • In the second backward phase, we update the policy network parameters θ. However, since the derivate ∂w ∂θ cannot be trivially evaluated, we propose to update θ using the REINFORCE algorithm J William (1992), treating µ( ∂∆ ∂w ) as a pseudo reward. The function µ( * ) will be elaborated later. Once θ updated, concrete weights w are discarded.

The network weights w used for the target task f : X → Y are sampled from P (w|θ) at the beginning of every single iteration and discarded at the end of the iteration. The de facto parameters we maintain during the whole training procedure are θ, which is a parameterization of P (w|X, Y ). Intuitively, we call this burn-after-reading style training scheme as SnapQuant. Note that the regularization term Γ is only involved in the second backward phase. Then we describe these four phases sequentially, for which we elaborate the nested parameter structure of the policy network as a necessary foundation. 

• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • (a) (b) Input Layer1 Layer2 Output

w liok , l ∈ [1, L], i =∈ [1, I], o ∈ [1, O], k ∈ [1, K 2 ].

There are various ways to construct the policy network. For example, the most straightforward choice is to condition every w liok on an independent θ liok . Adopting this formulation is similar to the probabilistic version of BinaryConnect Courbariaux et al. (2015) except that in SnapQuant the training is done with the REINFORCE algorithm J William (1992). However, this trivial formulation fails to capture the joint distribution of different weights, which is obviously not a realistic assumption.

To this end, we propose a nested parameter structure of θ, hierarchically modeling the dependency between weights across layers, filters and kernels.

We illustrate our nested parameter structure in Fig 2 , where the policy network is a four-layer neural network. Note that these four layers are not conventional fully connected layers as the input-output connections are separated into groups. Now we detail how these groups are organized. Let's denote the input to the policy network as s, and the hidden units of subsequent three layers as h 1 , h 2 and h 3 . Then we name the layer-wise parameters of the policy network according to the different colors used in Fig 2 . The blue connections are layer shared parameters θ 1 l . The orange connections are filter shared parameters θ 2 li . The green connections are kernel shared parameters θ 3 lio . The black connections are weight specific parameters θ 4 liok . For a clear exposition of the nested structure, here we formally describe the forward procedure of the policy network. We use f ( * ; θ) to represent two sequent operations: (1) feeding inputs/hidden units to these grouply-separated linear layers; (2) feeding outputs of (1) to a subsequent sigmoid nonlinear function that gives f ( * ; θ) a probabilistic interpretation. Meanwhile we denote the final outputs of the policy network as p liok .

The following equation represents calculating layer shared hidden units h 1 :

EQUATION

Similarly, filter shared/kernel shared hidden units h 2 /h 3 and final outputs p liok are calculated by:

EQUATION

As such, we get the weight specific outputs p liok . Each p liok characterizes a policy:

EQUATION

Following these policies, we can do the sampling mentioned in the first forward phase. Thanks to the nested parameter structure, the policies to generate different weights are connected by those shared weights. For clarity, we further give a concrete example to demonstrate the parameter sharing mechanism. Imagine there are two weights w liok1 and w liok2 which reside in the same kernel. They are sampled according to p liok1 and p liok2 . Formally these two quantities are calculated by:

p liok1 = f (f (f (f (s; θ 1 l ); θ 2 li ); θ 3 lio ); θ 4 liok1 ) p liok2 = f (f (f (f (s; θ 1 l ); θ 2 li ); θ 3 lio ); θ 4 liok2 )

As clearly shown by these two equations, among all quantities used to generated p liok1 and p liok2 , the only difference is the weight specific parameters θ 4 liok1 and θ 4 liok2 . Now that the first forward phase has been formally defined, we move on to the remaining three phases. Recall that the second forward phase and the first backward phase actually form a ordinary forward-backward routine of a typical CNN. Thus the only unclear part is the second backward phase, which will be elaborated in the next subsection.

3.3. OPTIMIZING THE EXPECTED LIKELIHOOD WITH REINFORCE:

Since the procedure of sampling w from P (w|θ) is not trivially differentiable, we propose to update the parameters θ of the policy network using the REINFORCE algorithm J William (1992).

We associate a total of L × I × O × K 2 Markov Decision Processes (MDPs) with all the weights w. All these MDPs share the same state s, which is the globally shared input vector into the policy network. For every policy characterized by p liok , there are two actions: (1) assigning value +1 to w liok (with probability p liok ); (2) assigning value -1 to w liok (with probability 1-p liok ). To this end, we have to define rewards w.r.t. these two actions.

Here, we propose an intuitive trick that assumes there is only one step in an episode of each MDP and an episode corresponds to a complete forward-backward iteration in Fig 1 . As such, we can define pseudo rewards r = µ( ∂∆ ∂w ), which is related to the gradients w.r.t. w obtained in the current iteration. In our implementation, the pseudo reward function µ( * ) takes this form:

g liok = ∂∆ ∂w liok r liok = µ(g liok ) = -β × g liok × w liok (4)

Note that w liok is either +1 or -1 so multiplying w liok only changes the sign of the reward and the magnitude of the reward equals βg liok . β is the scaling factor for the reward, which is a hyper-parameter. The reward gives a positive value if the gradient g liok and the sampled weight w liok take different signs. For intuitively understanding why we design rewards as such, we give a concrete example. Assume w liok is sampled as +1 during the second forward phase and the gradient on it g liok emerges as negative. According to the gradient descent rule, we should update w liok into w liok -g liok . Obviously, w liok -g liok > w liok , meanning that if this is not a binary network, w liok would take a value larger than +1 (e.g. +2) after this update. As w liok can only take the value +1 or -1, instead we increase the possibility of assigning +1 to w liok . In other words, we increase p liok , meaning that the action of assigning +1 to w liok is encouraged. In this case, the reward is positive. Now that the states, actions and rewards of these 1-step MDPs have already been defined, we can update θ using the standard REINFORCE algorithm J William (1992) . The expected reward is:

J(θ) = l,i,o,k E P (w liok |θ) (r liok )

According to the REINFORCE principle, we have:

J(θ) = l,i,o,k E P (w liok |θ) ( θ log P (w liok |s; θ) × r liok )

Via applying a Monte Carlo sampling, we get an unbiased estimator of this quantity:

J(θ) = l,i,o,k ( θ log P (w liok |s, θ) × r liok )

(5)

Now that we have J(θ) which optimizes the first term in the aforementioned objective ∆ + Γ, combining it with Γ(θ) (for details see Appendix A) drives the variational distribution P θ (w) to the minimum of D KL (P θ (w)||P (w|X, Y )). The whole training procedure of our SnapQuant is summarized as in Algorithm 1.

Algorithm 1 SnapQuant for training BWNs from scratch:

Require: X: natural images, Y : image labels, w: recognition network weights restricted to +1 or -1 (what we really need is w's structure), Y * : recognition network outputs, ∆( * , * ): cross entropy function, {θ : θ 1 , θ 2 , θ 3 , θ 4 }: randomly initialized policy network parameters, s: the globally shared state vector to the policy network, N : maximum iteration number, β: pseudo reward scaling factor. Ensure: P (w|θ) approximates the posterior distribution P (w|X, Y ).

1: for n = 1, 2, . . . , N do 2:

Calculate {p liok } according to Equation (1-2), using s and θ.

3::

Sample concrete weights w = {w liok } according to Equation (3).

4::

Sample a mini-batch B from the training set {X, Y }.

5::

Do a standard feed forward using concrete weights w and B, getting ∆(Y * , Y ).

6::

Do a standard backward propagation to get gradients g liok = ∂∆ ∂w liok .

7::

Calculate pseudo rewards according to Equation (4).

8::

Update θ according to J(θ) in Equation ( 5) and Γ(θ) in Appendix A.

9::

Discard concrete weights w. 10: end for

4. EXPERIMENTS:



4.1. EXPERIMENTAL SETTINGS:

We evaluate SnapQuant with standard MNIST Yann et al. (1998) , CIFAR-10/100 Krizhevsky & Hinton (2009) and ImageNet Russakovsky et al. (2015) datasets. At the inference phase, we use the trained policy network to sample concrete binary weights w for evaluation. Assuming the recognition network has |w| weights, the amount of possible binary networks is up bounded by 2 |w| , which can be considered as infinite. For easy implementation, we sample 100 binary networks from the policy network and pick the best one. This inference scheme demonstrates the versatility of SnapQuant's probabilistic nature. In the experiments, four different settings (namely non-sharing, kernel-wise sharing, filter-wise sharing and layer-wise sharing) are considered for SnapQuant, according to how parameters are shared in the nested policy network. For instance, SnapQuant (layer-wise sharing) means the complete nested parameter structure is exploited, while SnapQuant (non-sharing) represents the trivial solution of conditioning every w liok on an independent θ liok . Taking 32-bit floating-point network model as the baseline, we compare our method with the most related method BinaryConnect Courbariaux et al. (2015) . BinaryConnect has two versions: (1) the deterministic version binaries weights using a hard threshold; (2) the stochastic version also samples concrete weights at the inference phase like our method, thus we also report its best result out of 100 trials. All algorithms are implemented with Pytorch Paszke et al. (2017) .

4.2. RESULT COMPARISON ON MNIST:

MNIST is a digit recognition dataset consisting of 60000 training samples and 10000 testing samples. We adopt the LeNet-5 architecture Yann et al. (1998) and the results are summarized in Table 1 . It can be seen that SnapQuant reaches an error rate of 0.89%, which is better than the stochastic version of BinaryConnect, and is very close to the performance of the full-precision baseline. SnapQuant's top-1 performance gap against Binary-weight-network is 1.9%. As a reminder, Binaryweight-network exploits filter-wise scaling factors leading to +α i or -α i weights. Scaling involves multiplication operations, which costs additional hardware cycles. BinaryNet and XNOR-Net exploit 1-bit activations. Recall that the central scientific problem considered in this paper is to learn the posterior distribution of binary weights, learning the distribution of binary activations is another interesting direction yet out of our scope. 2 , all parameter sharing settings clearly outperform the non-sharing setting. For CIFAR-10, the filter-wise sharing setting gets an error rate of 12.74%, which outperforms the non-sharing baseline by 1.50%. For CIFAR-100, the layer-wise sharing setting performs best, surpassing the non-sharing baseline by 3.61%. A deeper analysis is presented in Figure 3 -a/b, in which the accuracy distribution of 100 randomly sampled binary networks is illustrated. Note that the accuracy variances of the sharing settings are obviously smaller than the non-sharing baseline. According to the mean accuracies of the different settings, we can see that the filter-wise sharing performs the best. This implies that modeling the joint distribution across layers is not that necessary, which is consistent to common senses. While kernels or filters may be statistically related, weights in different layers are less likely to correlated with each other. We further present the testing error curves in Figure 3-c/d , showing that the sharing settings enable a more stable training.

Variance reduction results and training details are provided in Appendix B and C.

5. CONCLUSIONS:

In this paper, we proposed SnapQuant, a probabilistic method for training binary weight neural networks from scratch under the Bayesian deep learning perspective. We approximate the posterior distribution of binary weights with a reinforcement learning scheme. A policy network with a novel nested parameter structure was presented to parameterize the posterior distribution of binary weights. We show that the proposed method performs well in several visual recognition tasks including ImageNet, as tested with different network architectures.

:



Figures and Tables:

Figure 2: An illustration of the nested parameter structure. The policy network (a) has four layers while the recognition network (b) is a 2-layer CNN. Input to the policy network is state s.Blue/orange/green/black connections are layer shared/filter shared/kernel shared/weight specific parameters of the policy network. The details of sampling the recognition network weights w from the policy network can be found in the main text.

Figure 3: (a/b) The accuracy distribution of 100 randomly sampled binary networks for different SnapQuant settings, on CIFAR-10/100. (c/d) Testing set error curves of different SnapQuant settings on CIFAR-10/100. Each data point corresponds to the performance of a randomly sampled network at that epoch. Note these four figures are better viewed electronically for a full resolution.

The leftmost panel illustrates the connection between probabilistic nodes. The remaining four panels show how a forward-backward iteration for the likelihood term ∆ is done.natural images and Y is the set of image labels taken from a pre-defined category taxonomy. Since Convolutional Neural Networks (CNNs) are the most prevalent model, we consider the setting of approximating f : X → Y with a CNN. More specifically, we restrict the weights of the recognition network (denoted as w) to take either value +1 or -1. We adopt the Bayesian deep learning

Table content: <table><tr><td>Graphical Model</td><td>Forward Phase I</td><td>Forward Phase II</td><td>Backward Phase I</td><td>Backward Phase II</td></tr><tr><td>: Policy Network Parameters : Recognition Network Weights</td><td>Sample from</td><td>: Input Data Points : Labels : Predictions : Loss Function</td><td>Calculate Gradients on Sampled</td><td>Update with REINFORCE w.r.t. Pseudo Reward</td></tr><tr><td>Figure 1:</td><td/><td/><td/><td/></tr><tr><td colspan="5">Reinforcement Learning. Recently, reinforcement learning J William (1992); Volodymyr et al.</td></tr><tr><td colspan="5">(2015) has been used to reduce the complexity of DNNs. Veniat &amp; Denoyer (2017) use RL to learn</td></tr><tr><td colspan="5">efficient DNN architecture with a budgeted objective function. Lin et al. (2017) propose to train a</td></tr><tr><td colspan="5">policy Recurrent Neural Network (RNN) which can dynamically prune DNNs according to the input</td></tr><tr><td colspan="5">data and the corresponding feature maps during the inference phase. Ashok et al. (2017) adopt two</td></tr><tr><td colspan="5">policy RNNs for learning reduced network architectures from the larger teachers in an incremental</td></tr><tr><td colspan="5">way. He &amp; Han (2018) utilize deep deterministic policy gradient agent to address DNN pruning.</td></tr><tr><td colspan="5">These RL based methods share a common feature: they mainly focus on how to adapt policy network</td></tr><tr><td colspan="5">designs for compressing pre-trained network models, usually from the perspective of weights pruning.</td></tr><tr><td colspan="5">Other Methods. Besides aforementioned approaches, there are also many other methods to improve</td></tr><tr><td colspan="5">the efficiency of DNNs. Network pruning is a promising way to transform dense DNN models into</td></tr><tr><td colspan="5">sparse versions without loss of predication accuracy. This line of research mainly includes network</td></tr><tr><td colspan="5">parameter pruning Yann et al. (1990); Hassibi &amp; Stork (1993); Han et al. (2015); Guo et al. (2016),</td></tr><tr><td colspan="5">filter pruning Li et al. (2017b) and channel pruning Hu et al. (2016); He et al. (2017). Knowledge</td></tr><tr><td colspan="5">distillation presents another way which allows training an efficient yet accurate student network</td></tr><tr><td colspan="5">distilled by the knowledge of a larger pre-trained teacher model Hinton et al. (2014); Romero et al.</td></tr><tr><td colspan="5">(2015); Mishra &amp; Marr (2018). For more neural network compression methods, we refer the reader</td></tr><tr><td colspan="4">to Sze et al. (2017) and Cheng et al. (2018) for comprehensive reviews.</td><td/></tr><tr><td colspan="3">3 PROPOSED METHOD: SNAPQUANT</td><td/><td/></tr><tr><td>3.1 OVERVIEW</td><td/><td/><td/><td/></tr><tr><td colspan="5">Here we consider the problem of training a binary neural network f for a supervised learning task</td></tr><tr><td colspan="5">f : X → Y , in which X is the training set of the input data points and Y is the corresponding</td></tr><tr><td colspan="5">objective. In this study, we focus on supervised image classification tasks, thus X represents</td></tr></table>

Quantitative results on MNIST. 4.3 RESULT COMPARISON ON CIFAR-10/100 CIFAR-10 Krizhevsky & Hinton (2009) consists of 50000 training images and 10000 testing images collected from 10 classes. CIFAR-100 has the same numbers of training/testing images containing 100 classes. For CIFAR-10, we firstly evaluate the VGG-like architecture proposed in BinaryConnect Courbariaux et al. (2015). It has 14.03 million parameters, causing severe over-fitting. The 32-bit floating-point baseline can only reach 10.64% error rate, which is worse than SnapQuant and BinaryConnect as binarization can be considered a regularization. Here, for BinaryConnect, we use its best result reported in the original paper. Then we evaluate the ResNet-20 architecture He et al. (2016) on both CIFAR-10 and CIFAR-100. The results are shown in Table 2. On both benchmarks, SnapQuant outperforms the stochastic version of BinaryConnect by large margins while performs comparably with its deterministic version. It is worth mentioning that the deterministic version cannot provide the versatility of Bayesian formulation, such as model selection and uncertainty measure.

Table content: <table><tr><td>Architecture</td><td>Method</td><td>Error rate (%)</td></tr><tr><td>LeNet-5</td><td>32-bit floating-point baseline</td><td>0.81</td></tr><tr><td/><td>BinaryConnect (deterministic)</td><td>0.88</td></tr><tr><td/><td>BinaryConnect (stochastic)</td><td>1.24</td></tr><tr><td/><td>SnapQuant (layer-wise sharing)</td><td>0.89</td></tr></table>

Quantitative results on CIFAR-10/100.

Table content: <table><tr><td>4.4 RESULT COMPARISON ON IMAGENET</td></tr><tr><td>ImageNet classification dataset is known as the most challenging image classification benchmark</td></tr><tr><td>so far. It has about 1.2 million training images and 50 thousand validation images. Each image is</td></tr></table>

Error rates (%) on ImageNet validation set.For a deep understanding of the nested parameter structure, we present ablation studies on CIFAR-10/100. According to the results shown in Table

Table content: <table><tr><td/><td>AlexNet</td><td/><td>ResNet-18</td><td/></tr><tr><td>Method</td><td>Top-1</td><td>Top-5</td><td>Top-1</td><td>Top-5</td></tr><tr><td>32-bit floating-point baseline</td><td>43.4</td><td>19.8</td><td>30.7</td><td>10.8</td></tr><tr><td>BinaryNet (1-bit activation)</td><td>72.1</td><td>49.6</td><td>-</td><td>-</td></tr><tr><td>XNOR-Net (1-bit activation)</td><td>55.8</td><td>30.8</td><td>48.8</td><td>26.8</td></tr><tr><td>Binary-weight-network (w/ scaling)</td><td>43.2</td><td>20.6</td><td>39.2</td><td>17.0</td></tr><tr><td>BinaryConnect (deterministic)</td><td>64.6</td><td>39.0</td><td>-</td><td>-</td></tr><tr><td>SnapQuant (layer-wise sharing)</td><td>47.7</td><td>25.0</td><td>41.1</td><td>18.3</td></tr><tr><td colspan="3">4.5 NESTED PARAMETER STRUCTURE: DEEP ANALYSIS</td><td/><td/></tr></table>

Peer review (iclr2019):
 Summary: The paper considers a variational inference strategy for learning neural networks with binary weights. In particular, the paper proposes using a structured recognition model to parameterise the variational distribution, which couples the weights in different layers/filters in a non-trivial way. The gradient of the expected likelihood term in the variational lower bound is estimated using the REINFORCE estimator. This paper adjusts this estimator to use the gradient of the log-likelihood wrt the samples. Experiments on several image classification tasks are provided. evaluation: pros: - the idea of the proposed approach is interesting: using variational inference for binary weight neural networks. While recent work on VI for discrete variables only focused on discrete latent variable models, this work shows how VI can be used for binary neural networks. cons: - the writing, in my opinion, needs to be improved [see my comments below]. The VI presentation is cluttered and the justification of using the pseudo-reward for reinforce is not clear. - the experimental results are mixed and it*s not clear to me how to interpret them/compare to the baselines -- what is the goal here: computational efficiency, compression or accuracy? Some specific questions/comments: + What is the input of the policy/recognition network? It*s not clear from the paper whether this includes the inputs of the current batch or outputs or both? If so, how are variable batch sizes handled? What is the input to this network at test time? In contrast to generative models/VAEs, the weights here are global parameters and it*s not clear to me these should be varied for different data batches. + related to the question above: how is prediction handled at test time? Say the parameters of the variational distribution over weights are generated using the recognition network, then 100 weights are sampled given these parameters which then give 100 predictions -- should these be then averaged out to get the final prediction? I*m not quite sure I understand why the paper chose to *pick the best one* out of 100 predictions and the justification/criterion for this procedure. + The writing is not very clear at places, and it does not help that the references being merged with the text. I*m also not sure about some of the technical jargons/terms used in the papers: - reinforcement learning: is this really a reinforcement learning problem? If you tackle this problem from a pure variational perspective, reinforce is used to obtain the gradient of the expected log-likelihood wrt the variational parameters. But instead of using the log likelihood, a learning signal that depends on the gradient of the log-likelihood is used. - concrete weights -- what are these? I assume they are just binary weights sampled from the variational approximation. - middle of page 3: p(w|X, Y) = p_.theta(w): this is not precise as p_.theta(w) is only an approximation to the exact posterior, which then allows us to lower bound the log marginal likelihood. "common practice in modern variational approximation": This is the standard way of deriving the lower bound and has been used for many years. + the reinforce estimator tends to have high variances since it does not make use of the gradient of the function in the expectation. This paper adjusts the vanilla estimator with a learning signal that involves the gradient. Could you comment on the bias/variance trade-off of the resulting estimator? Much of recent literature on learning discrete variables, as far as I understand, propose ways to not to have to use the vanilla reinforce, for example Concrete, Relax or rebar, albeit the focus on latent variable models. + model selection and uncertainty measure: the paper mentions these potential advantages of the proposed approach over deterministic binarisation schemes, but does not fully explore and test these.