Title: PROBABILISTIC BINARY NEURAL NETWORKS

Abstract: Low bit-width weights and activations are an effective way of combating the increasing need for both memory and compute power of Deep Neural Networks. In this work, we present a probabilistic training method for Neural Network with both binary weights and activations, called PBNet. By embracing stochasticity during training, we circumvent the need to approximate the gradient of functions for which the derivative is zero almost always, such as sign(•), while still obtaining a fully Binary Neural Network at test time. Moreover, it allows for anytime ensemble predictions for improved performance and uncertainty estimates by sampling from the weight distribution. Since all operations in a layer of the PBNet operate on random variables, we introduce stochastic versions of Batch Normalization and max pooling, which transfer well to a deterministic network at test time. We evaluate two related training methods for the PBNet: one in which activation distributions are propagated throughout the network, and one in which binary activations are sampled in each layer. Our experiments indicate that sampling the binary activations is an important element for stochastic training of binary Neural Networks.

1. INTRODUCTION:

Deep Neural Networks are notorious for having vast memory and computation requirements, both during training and test/prediction time. As such, Deep Neural Networks may be unfeasible in various environments such as battery powered devices, embedded devices (because of memory requirement), on body devices (due to heat dissipation), or environments in which constrains may be imposed by a limited economical budget. Hence, there is a clear need for Neural Networks that can operate in these resource limited environments.

One method for reducing the memory and computational requirements for Neural Networks is to reduce the bit-width of the parameters and activations of the Neural Network. This can be achieved either during training (e.g., Ullrich et al. (2017) ; Achterhold et al. (2018) ) or using post-training mechanisms (e.g., Louizos et al. (2017) , Han et al. (2015) ). By taking the reduction of the bit-width for weights and activations to the extreme, i.e., a single bit, one obtains a Binary Neural Network. Binary Neural Networks have several advantageous properties, i.e., a 32× reduction in memory requirements and the forward pass can be implemented using XNOR operations and bit-counting, which results in a 58× speedup on CPU (Rastegari et al., 2016) . Moreover, Binary Neural Networks are more robust to adversarial examples (Galloway et al., 2018) . Shayer et al. (2018) introduced a probabilistic training method for Neural Networks with binary weights, but allow for full precision activations. In this paper, we propose a probabilistic training method for Neural Networks with both binary weights and binary activations, which are even more memory and computation efficient. In short, obtain a closed form forward pass for probabilistic neural networks if we constrain the input and weights to binary (random) variables. The output of the Multiply and Accumulate (MAC) operations, or pre-activation, is approximated using a factorized Normal distribution. Subsequently, we introduce stochastic versions of Max-Pooling and Batch Normalization that allow us to propagate the pre-activatoins throughout a single layer. By applying the sign(•) activation function to the random pre-activation, we not only obtain a distribution over binary activations, it also allows for backpropagation through the sign(•) operation. This is especially convenient as this in a deterministic Neural Network all gradient information is zeroed out when using sign as activation. We explore two different methods for training this probabilistic binary neural network: In the first method the activation distribution of layer l is propagated to layer (l + 1), which means the MAC operation is performed on two binary random variables. In the second method the binary activation is sampled as the last operation in a layer using the concrete relaxation Maddison et al. (2016) . This can be thought of as a form of local reparametrization Kingma et al. (2015) . We call the networks obtained using these methods PBNet and PBNet-S, respectively.

At test time, we obtain a single deterministic Binary Neural Network, an ensemble of Binary Neural Networks by sampling from the parameter distribution, or a Ternary Neural Network based on the Binary weight distribution. An advantage of our method is that we can take samples from the parameter distribution indefinitely-without retraining. Hence, this method allows for anytime ensemble predictions and uncertainty estimates. Note that while in this work we only consider the binary case, our method supports any discrete distribution over weights and activations.

2. PROBABILISTIC BINARY NEURAL NETWORK:

Algorithm 1: Pseudo code for forward pass of single layer in PBNet(-S). a l-1 denotes the activation of the previous layer, B the random binary weight matrix, τ is the temperature used for the concrete distribution, f (•, •) the linear transformation used in the layer, > 0 a small constant for numerical stability, D the dimensionality of the inner product in f , and γ & β are the parameters for batch normalization.

Input: a l-1 , B ∼ p(B), τ , f (•, •), , γ, β Result: Binary activation a l // CLT approximation if a l-1 is a binary random variable then µ = f (E[B], E[a l-1 ]); σ 2 = D -f ((E[B]) 2 , (E[a l-1 ]) 2 ); else µ = f (E[B], a l-1 ); σ 2 = f (V[B], a 2 l-1 ); end // Batch normalization m = channel-wise-mean(µ); v = channel-wise-variance(µ, σ 2 , m); µ = γ(µ -m)/ √ v + + β; σ 2 = γ 2 σ 2 /(v + ); // Max pooling if max pooling required then n ∼ N (0, I); s = µ + σ n; ι = max-pooling-indices(s); µ, σ 2 = select-at-indices(µ, σ 2 , ι); end // Binarization and sampling p = 1 -Φ(0|µ, σ 2 ); if sample activation then a l ∼ BinaryConcrete(p, τ ); return a l ; else return Binary(p) end

In this section we introduce the probabilistic setting of the PBNet. Moreover, the approximation of the distribution on the pre-activations is introduced. For an explanation of the other operations in the PBNet, see Section 2.1 for the activation, Section 2.1.1 for the sampling of activations, and Section 2.2 for Pooling and Normalization.

We aim to train a probabilistic Binary Neural Network. As such, we pose a binary distribution over the weights of the network and optimize the parameters of this distribution instead of the parameters directly. This way, we obtain a distribution over parameters, but also deal with the inherent discreteness of a Binary Neural Network. Given an objective function L(•), this approach can be thought of in terms of the variational optimization framework Staines & Barber (2012) . Specifically, by optimizing the parameters of the weight distributions, we optimize a bound on the actual loss:

EQUATION

where B are the binary weights of the network and q θ (B) is a distribution over the binary weights. For q θ (B) a slight reparametrization of the Bernoulli distribution is used, which we will refer to as the Binary distribution. This distribution is parameterized by θ ∈ [-1, 1] and is defined by:

a ∼ Binary(θ) ⇐⇒ a + 1 2 ∼ Bernoulli( θ + 1 2 ). (2)

For the properties of this distribution, please refer to Appendix A.

We will now consider using the Binary distribution for both the weights and the activations in a Neural Network. Since the pre-activations in a Neural Network are computed using MAC operations, which are the same for each value in the pre-activation, we will only consider a single value in our discussion here. Let w ∼ Binary(θ) and h ∼ Binary(φ) be the weight and input random variable for a given layer. As such, the innerproduct between the weights and input is distributed according to a translated and scaled Poisson binomial distribution:

EQUATION

)

Where D is the dimensionality of h and w and denotes element-wise multiplication. See the picket fence on the top in Figure 1 for an illustration of the PMF of a Poisson binomial distribution.

Although the scaled and translated Poisson binomial distribution is the exact solution for the inner product between the weight and activation random variables, it is hard to work with in subsequent layers. For this reason, and the fact that the Poisson binomial distribution is well approximated by a Normal distribution (Wang & Manning, 2013) , we use a Normal approximation to the Poisson binomial distribution, which allows for easier manipulations. Using the properties of the Binary distribution and the Poisson binomial distribution, the approximation for the pre-activation a is given by:

EQUATION

Note that, this is exactly the approximation one would obtain by using the Lyapunov Central Limit Theorem (CLT), which was used by Shayer et al. (2018) . This allows us to obtain a close approximation to the pre-activation distribution, which we can propagate through the layer and/or network. So far, only the MAC operation in a given layer is discussed, in Section 2.1 application of the binary activation is discussed and in Section 2.1. The stochastic versions of Batch Normalization and Max Pooling are introduced in Section 2.2. For specifics on sampling the binary activation, see Section 2.1.1. The full forward pass for a single layer is given in detail in Algorithms 1.

2.1. STOCHASTIC BINARY ACTIVATION:

Since the output of a linear operation using binary inputs is not restricted to be binary, it is required to apply a binarization operation to the pre-activation in order to obtain binary activations. Various works -e.g., Hubara et al. (2016) (5)

In our case the pre-activations are random variables. Hence, applying a deterministic binarization function to a random pre-activations results in a stochastic binary activation. Specifically, let a i ∼ N (µ i , σ 2 i ) be a random pre-ctivation obtained using the normal approximation, as introduced in the previous section, then the activation (after binarization) is again given as a Binary random variable". Interestingly, the Binary probability can be computed in closed form by evaluating the probability density that lies above the binarization threshold:

EQUATION

where Φ(•|µ, σ 2 ) denotes the CDF of N (µ, σ 2 ). Applying the binarization function to a random pre-activation has two advantages. First, the derivatives ∂q i /∂µ i and ∂q i /∂σ i are not zero almost everywhere, in contrast to the derivatives of b det and b stoch when applied to a deterministic input. Second, the distribution over h i reflects the true uncertainty about the sign of the activation, given the stochastic weights, whereas b stoch uses the magnitude of the pre-activation as a substitute. For example, a pre-activation with a high positive magnitude and high variance will be deterministically mapped to 1 by b stoch . In contrast, our method takes the variance into account and correctly assigns some probability mass to -1. See Figure 1 for a graphical depiction of the stochastic binary activation.

2.1.1. SAMPLING THE BINARY ACTIVATIONS:

So far, we have discussed propagating distributions throughout the network. Alternatively, the binary activations can be sampled using the Concrete distribution (Maddison et al., 2016) during training. specifically, we use the hard sample method as discussed by Jang et al. (2016) . By sampling the activations, the input for subsequent layers will match the input that is observed at test time more closely.

As a consequence of sampling the activation, the input to a layer is no longer a distribution but a h ∈ {-1, +1} D vector instead. As such, the normal approximation to the pre-activation is computed slightly different. From the Lyapunov CLT it follows that the approximation to the distribution of the pre-activation is given by:

EQUATION

where w ∼ Binary(θ) is a random weight. Similarly, the pre-activation of the input layer is also computed using this approximation-given a real-valued input vector. We will refer to a PBNet that uses activation sampling as PBNet-S.

2.2. NORMALIZATION AND POOLING:

Other than a linear operation and an (non-linear) activation function, Batch Normalization (Ioffe & Szegedy, 2015) and pooling are two popular building blocks for Convolutional Neural Networks. For Binary Neural Networks, applying Batch Normalization to a binarized activation will result in a non-binary result. Moreover, the application of max pooling on a binary activation will result in a feature map containing mostly +1s. Hence, both operations must be applied before binarization. However, in the PBNet, the binarization operation is applied before sampling. As a consequence, the Batch Normalization and pooling operations can only be applied on random pre-activations. For this reason, we define these methods for random variables. Although there are various ways to define these operation in a stochastic fashion, our guiding principle is to only leverage stochasticity during training, i.e., at test time, the stochastic operations are replaced by their conventional implementations and parameters learned in the stochastic setting must be transferred to their deterministic counterparts.

2.2.1. STOCHASTIC BATCH NORMALIZATION:

Batch Normalization (BN) (Ioffe & Szegedy, 2015) -including an affine transformation -is defined as follows:

EQUATION

where a i denotes the pre-activation before BN, â the pre-activation after BN, and m & v denote the sample mean and variance of {a i } M i=1 , for an M -dimensional pre-activation. In essence, BN translates and scales the pre-activations such that they have approximately zero mean and unit variance, followed by an affine transformation. Hence, in the stochastic case, our aim is that samples from the pre-activation distribution after BN also have approximately zero mean and unit variance-to ensure that the stochastic batch normalization can be transfered to a deterministic binary neural network. This is achieved by subtracting the population mean from each pre-activation random variable and by dividing by the population variance. However, since a i is a random variable in the PBNet, simply using the population mean and variance equations will result in non-standardized output. Instead, to ensure a standardized distribution over activations, we compute the expected population mean and variance under the pre-activation distribution:

EQUATION

)

where M is the total number of activations and a i ∼ N (µ i , σ i ) are the random pre-activations. By substituting m and v in Equation 8 by Equation 9and 10, we obtain the following batch normalized Gaussian distributions for the pre-activations:

EQUATION

)

Note that this assumes a single channel, but is easily extended to 2d batch norm in a similar fashion as conventional Batch Normalization. At test time, Batch Normalization in a Binary Neural Network can be reduced to an addition and sign flip of the activation, see Appendix B for more details.

2.2.2. STOCHASTIC MAX POOLING:

In general, pooling applies an aggregation operation to a set of (spatially oriented) pre-activations.

Here we discuss max pooling for stochastic pre-activations, however, similar considerations apply for other types of aggregation functions.

In the case of max-pooling, given a spatial region containing stochastic pre-activations a 1 , . . . , a K , we aim to stochastically select one of the a i . Note that, although the distribution of max(a 1 , . . . , a K ) is well-defined (Nadarajah & Kotz, 2008) , its distribution is not Gaussian and thus does not match one of the input distributions. Instead, we sample one of the input random variables in every spatial region according to the probability of that variable being greater than all other variables, i.e., ρ i = p(a i > z \i ), where z \i = max({a j } j =i ). ρ i could be obtained by evaluating the CDF of (z \i -a i ) at 0, but to our knowledge this has no analytical form. Alternatively, we can use Monte-Carlo integration to obtain ρ:

EQUATION

where one-hot(i) returns a K-dimensional one-hot vector with the ith elements set to one. The pooling index ι is then sampled from Cat(ρ). However, more efficiently, we can sample s ∼ p(a 1 , . . . , a K ) and select the index of the maximum in s, which is equivalent sampling from Cat(ρ). Hence, for a given max pooling region, it is sufficient to obtain a single sample from each normal distribution associated with each pre-activation and keep the random variable for which this sample is maximum.

A graphical overview of this is given in Figure 2 .

Other forms of stochastic or probabilistic max pooling were introduced by Lee et al. ( 2009) and Zeiler & Fergus (2013), however, in both cases a single activation is sampled based on the magnitude of the activations. In contrast, in our procedure we stochastically propagate one of the input distributions over activations.

2.3. WEIGHT INITIALIZATION:

For the PBNet the parameters θ for q θ (B) are initialized from a uniform U (-1, 1) distribution.

Although the final parameter distribution more closely follows a Beta(α, α) distribution, for α < 1, we did not observe any significant impact choosing another initialization method for the PBNet.

In the case of the PBNet-S, we observed a significant improvement in training speed and performance by initializing the parameters based on the parameters of a pre-trained full precission Neural Network. This initializes the convolutional filters with more structure than a random initialization. This is desirable as in order to flip the value of a weight, the parameter governing the weight has to pass through a high variance regime, which can slow down convergence considerably.

Select maximum per region 2 Sample from input distributions 1:

Keep maximum distribution for each region 3

Figure 2 : Max pooling for random variables is performed by taking a single sample from each of the input distributions. The output random variable for each pooling region is the random variable that is associated with the maximum sample.

For the PBNet-S, We use the weight transfer method introduced by Shayer et al. ( 2018) in which the parameters of the weight distribution for each layer are initialized such that the expected value of the random weights equals the full precision weight divided by the standard deviation of the weights in the given layer. Since not all rescaled weights lay in the [-1, 1] range, all binary weight parameters are clipped between [-0.9, 0.9]. This transfer method transfers the structure present in the filters of the full precision network and ensures that a significant part of the parameter distributions is initialized with low variance.

2.4. DETERMINISTIC BINARY NEURAL NETWORK:

In our training procedure, a stochastic neural network is trained. However, at test time (or on hardware) we want to leverage all the advantages of a full binary Neural Network. Therefore, we obtain a deterministic binary Neural Network from the parameter distribution q θ (B) at test time. We consider three approaches for obtaining a deterministic network: a deterministic network based on the mode of q θ (B) called PBNET-MAP, an ensemble of binary Neural Networks sampled from q θ (B) named PBNET-x, and a ternary Neural Network (PBNET-TERNARY), in which a single parameter W i may be set to zero based on q θ , i.e.:

W i =    +1 if q θ (B i = +1) ≥ 3/4 -1 if q θ (B i = -1) ≥ 3/4 0 otherwise (13)

The ternary network can also be viewed as a sparse PBNet, however, sparse memory look-ups may slow down inference.

Note that, even when using multiple binary neural networks in an ensemble, the ensemble is still more efficient in terms of computation and memory when compared to a full precision alternative. Moreover, it allows for anytime ensemble predictions for improved performance and uncertainty estimates by sampling from the weight distribution.

Since the trained weight distribution is not fully deterministic, the sampling of individual weight instantiations will result in a shift of the batch statistics. As a consequence, the learned batch norm statistics no longer closely match the true statistics. This is alleviated by re-estimating the batch norm statistics based on (a subset of) the training set after weight sampling using a moving mean and variance estimator. We observed competitive results using as little as 20 batches from the training set.

3. RELATED WORK:

Binary and low precision neural networks have received significant interest in recent years. Most similar to our work, in terms of the final neural network, is the work on Binarized Neural Networks by Hubara et al. (2016) . in this work a real-valued shadow weight is used and binary weights are obtained by binarizing the shadow weights. Similarly the pre-activations are binarized using the same binarization function. In order to back-propagate through the binarization operation the straightthrough estimator (Hinton, 2012) is used. Several extensions to Binarized Neural Networks have been proposed which -more or less -qualify as binary neural networks: XNOR-net (Rastegari et al., 2016) in which the real-valued parameter tensor and activation tensor is approximated by a binary tensor and a scaling factor per channel. ABC-nets Lin et al. (2017) take this approach one step further and approximate the weight tensor by a linear combination of binary tensors. Both of these approaches perform the linear operations in the forward pass using binary weights and/or binary activations, followed by a scaling or linear combination of the pre-activations. 

4. EXPERIMENTS:

We evaluate the PBNet on the MNIST and CIFAR-10 benchmarks and compare the results to Binarized Neural Networks (Hubara et al., 2016) , since the architectures of the deterministic networks obtained by training the PBNet are equivalent.

4.1. EXPERIMENTAL DETAILS:

The PBNets are trained using either a cross-entropy (CE) loss or a binary cross entropy for each class (BCE). For the CE loss there is no binarization step in the final layer, instead the mean of the Gaussian approximation is used as the input to a softmax layer. For BCE, there is a binarization step, and we treat the probability of the ith output being +1 as the probability of the input belonging to the ith class. Specifically, for an output vector p ∈ [0, 1] C for C classes and the true class y, the BCE loss for a single sample is defined as

EQUATION

The weights for the PBNet-S networks are initialized using the transfer method described in Section 2.3 and the PBNets are initialized using a uniform initialization scheme. All models are optimized using Adam (Kingma & Ba, 2014) and a validation loss plateau learning rate decay scheme. We keep the temperature for the binary concrete distribution static at 1.0 during training. For all settings, we optimize model parameters until convergence, after which the best model is selected based on a validation set. Our code is implemented using PyTorch (Paszke et al., 2017) .

For Binarized Neural Networks we use the training procedure described by Hubara et al. (2016) , i.e., a squared hinge loss and layer specific learning rates that are determined based on the Glorot initialization method (Glorot & Bengio, 2010) .

Experimental details specific to datasets are given in Appendix C and the results are presented in Table 1 . We report both test set accuracy obtained after binarizing the network as well as the the test set accuracy obtained by the stochastic network during training (i.e., by propagating activation distributions).

4.2. ENSEMBLE BASED UNCERTAINTY ESTIMATION:

As presented in Table 1 the accuracy improves when using an ensemble. Moreover, the predictions of the ensemble members can be used to obtain an estimate of the certainty of the ensemble as a whole.

Table 1 : Test accuracy on MNIST and CIFAR-10 for Binarized NN (Hubara et al., 2016) , PBNet, and a full precission network (FPNet). PBNet-map refers to a deterministic PBNet using the map estimate, PBNet-Ternary is a ternary deterministic network obtained from q θ , and PBNet-X refers to an ensemble of X networks, each sampled from the same weight distribution. For the ensemble results both mean and standard deviation are presented. The propagate column contains results obtained using the stochastic network whereas results in the binarized column are obtained a deterministic binary Neural Network. To evaluate this, we plot an error-coverage curve (Geifman & El-Yaniv, 2017) in Figure 3a . This curve is obtained by sorting the samples according to a statistic and computing the error percentage in the top x% of the samples -according to the statistic. For the Binarized Neural Network and PBNet-MAP the highest softmax score is used, whereas for the ensembles the variance in the prediction of the top class is used. The figure suggests that the ensemble variance is a better estimator of network certainty, and moreover, the estimation improves as the ensemble sizes increases.

4.3. EFFECT OF BATCH STATISTICS RE-ESTIMATION:

As discussed in Section 2.4, after sampling the parameters of a deterministic network the batch statistics used by Batch Normalization must be re-estimated. Figure 3b shows the results obtained using a various number of batches from the training set to re-estimate the statistics. This shows that even a small number of samples is sufficient to estimate the statistics.

4.4. ABLATION STUDIES:

We perform an ablation study on both the use of (stochastic) Batch Normalization and the use of weight transfer for the PBNet-S on CIFAR-10. For Batch Normalization, we removed all batch normalization layers from the PBNet-S and retrained the model on CIFAR-10. This resulted in a test set accuracy of 79.21%. For the weight initialization experiment, the PBNet-S weights are initialized using a uniform initialization scheme and is trained on CIFAR-10, resulting in a test set accuracy of 83.61%. Moreover, the accuracy on the validation set during training is presented in Figure 3c . Note that these numbers are obtained without sampling a binarized network from the weight distribution, i.e., local reparametrization and binary activation samples are used. The PBNet-S that uses both weight transfer and stochastic Batch Normalization results in a significant performance improvement, indicating that both stochastic Batch Normalization and weight transfer are necessary components for the PBNet-S.

4.5. SAMPLING OF BINARY ACTIVATIONS:

The results of our experiments show that, following our training procedure, sampling of the binary activations is a necessary component. the stochastic PBNet. In contrast, this performance drop is not observed for PBNet-S. A potential explanation of this phenomenon is that by sampling the binary activation during training, the network is forced to become more robust to the inherent binarization noise that is present at test time of the binarized Neural Network. If this is the case, then sampling the binary activation can be thought of as a regularization strategy that prepares the network for a more noisy binary setting. However, other regularization strategies may also exist.

5. CONCLUSION:

We have presented a stochastic method for training Binary Neural Networks. The method is evaluated on multiple standardized benchmarks and reached competitive results. The PBNet has various advantageous properties as a result of the training method. The weight distribution allows one to generate ensembles online which results in improved accuracy and better uncertainty estimations. Moreover, the Bayesian formulation of the PBNet allows for further pruning of the network, which we leave as future work. 

A BINARY DISTRIBUTION:

For convenience, we have introduced the Binary distribution in this paper. In this appendix we list some of the properties used in the paper, which all follow direcly from the properties of the Bernoulli distribution. The Binary distribution is a reparametrization of the Bernoulli distribution such that:

EQUATION

This gives the following probability mass function:

EQUATION

where a ∈ {-1, +1} and θ ∈ [-1, 1]. From this, the mean and variance are easily computed:

EQUATION

)

Finally, let b ∼ Binary(φ), then ab ∼ Binary(θφ).

B BATCH NORMALIZATION IN A BINARY NEURAL NETWORK:

During training the PBNet is trained using stochastic Batch Normalization. At test time, the parameters learned using stochastic Batch Normalization can be transferred to a conventional Batch Normalization implementation. Alternatively, Batch Normalization can be reduced to an (integer) addition and multiplication by ±1 after applying the sign activation function. Given a pre-activation a, the application of Batch Normalization followed by a sign binarization function can be rewritten as: 

EQUATION

Note that we have used sign(0) = b det (0) = +1 here, as we have used everywhere in order to use sign as a binarization function.

:



Figures and Tables:

and Rastegari et al. (2016) -use either deterministic or stochastic binarization functions, i.e., b det (a) = +1 if a ≥ 0 -1 otherwise b stoch (a) = +1 with probability p = sigmoid(a) -1 with probability 1 -p .

Figure 1: The discrete Poisson binomial distribution (in green) is approximated by a continuous Gaussian distribution (in purple). By applying b det to a random pre-activation we obtain a binary activation distribution.

Figure 3: Error coverage curve, batch statistic re-estimation results and ablation study results for CIFAR-10.

20) when a ∈ Z, which is the case for all but the first layer γ) sign (a + b)

InMcDonnell (2018), similar methods toHubara et al. (2016) are used to binarize a wide resnet(Zagoruyko & Komodakis,  2016)  to obtain results on ImageNet very close to the full precision performance. Another method for training binary neural networks is Expectation Backpropagation(Soudry et al., 2014) in which the central limit theorem and online expectation propagation is used to find an approximate posterior. This method is similar in spirit to ours, but the training method is completely different. Most related to our work is the work byShayer et al. (2018) which use the local reparametrization trick to train a Neural Network with binary weights and the work byBaldassi et al. (2018) which also discuss a binary Neural Network in which the activation distribution propagated through the network.Moreover, in (Wang & Manning, 2013)  the CLT was used to approximate dropout noise during training in order to speed up training, however, there is no aim to learn binary (or discrete) weights or use binary activations in this work.

Table content: <table/>

Although the stochastic PBNet generalizes well to unseen data, there is a significant drop in test accuracy when a binary Neural Network is obtained from

Table content: <table><tr><td>error (%)</td><td>0.00 0.02 0.04 0.06 0.08 0.10 0.12</td><td>0.25 0.50 0.75 1.00 coverage (%) PBNet-2 PBNet-5 PBNet-16 PBNet Binarized Net</td><td>number of batches</td><td>1 2 5 10 15 20 all</td><td>86 test accuracy (%) 88</td><td>90</td><td>Validation accuracy (%)</td><td>90 70 75 80 85 60 65</td><td>0</td><td>Epochs 50 100 150 200 PBNet-MAP No Bnorm Uniform Init</td></tr><tr><td colspan="3">(a) coverage for CIFAR-10.</td><td colspan="4">(b) Test set performance with in-</td><td colspan="4">(c) Accuracy on validation set dur-</td></tr><tr><td/><td/><td/><td colspan="4">creasing number of batches used to</td><td colspan="4">ing training, i.e., using stochastic</td></tr><tr><td/><td/><td/><td colspan="4">re-estimate the batch statistics on</td><td colspan="4">weights, local reparametrization and</td></tr><tr><td/><td/><td/><td colspan="3">CIFAR-10.</td><td/><td colspan="4">binary activation sampling.</td></tr></table>

Karen Ullrich, Edward Meeds, and Max Welling. Soft weight-sharing for neural network compression. arXiv preprint arXiv:1702.04008, 2017. Sida Wang and Christopher Manning. Fast dropout training. In international conference on machine learning, pp. 118-126, 2013. Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016. Matthew D Zeiler and Rob Fergus. Stochastic pooling for regularization of deep convolutional neural networks. arXiv preprint arXiv:1301.3557, 2013.

Table content: <table/>

Peer review (iclr2019):
 ## Summary This work presents a probabilistic training method for binary Neural Network with stochastic versions of Batch Normalization and max pooling. By sampling from the weight distribution an ensemble of Binary Neural Networks could further improve the performance. In the experimental section, the authors compare proposed PBNet with Binarized NN (Hubara et al., 2016) in two image datasets (MNIST and CIFAR10). In general, the paper was written in poor quality and without enough details. The idea behind the paper is not novel. Stochastic binarization and the (local) reparametrization trick were used to training binary (quantized) neural networks in previous works. The empirical results are not significant. ## Detail comments Issues with the training algorithm of stochastic neural network The authors did not give details of the training method and vaguely mentioned that the variational optimization framework (Staines & Barber, 2012). I do not understand equation 1. Since B is binary, the left part of equation 2 is a combination optimization problem. If B is sampled during the training, the gradient would suffer from high variance. Issues with propagating distributions throughout the network Equation 3 is based on the assumption of that the activations are random variables from Bernoulli distribution. In equation 4, the activations of the current layer become random variables from Gaussian distribution. How the activations to further propagate? Issues with ternary Neural Networks in section 2.4 For a ternary NN, the weight will be from a multinomial distribution, I think it will break the assumption used by equation 3. Issues with empirical evidences Since the activations are sampled in PBNET-S, a more appropriate baseline should be BNN with stochastic binarization (Hubara et al., 2016) which achieved 89.85% accuracy on CIFAR-10. It means that the proposed methods did not show any significant improvements. By the way BNN with stochastic binarization (Hubara et al., 2016) can also allow for ensemble predictions to improve performance.