Probabilistic Binary Neural Networks

1 INTRODUCTION . Deep Neural Networks are notorious for having vast memory and computation requirements , both during training and test/prediction time . As such , Deep Neural Networks may be unfeasible in various environments such as battery powered devices , embedded devices ( because of memory requirement ) , on body devices ( due to heat dissipation ) , or environments in which constrains may be imposed by a limited economical budget . Hence , there is a clear need for Neural Networks that can operate in these resource limited environments . One method for reducing the memory and computational requirements for Neural Networks is to reduce the bit-width of the parameters and activations of the Neural Network . This can be achieved either during training ( e.g. , Ullrich et al . ( 2017 ) ; Achterhold et al . ( 2018 ) ) or using post-training mechanisms ( e.g. , Louizos et al . ( 2017 ) , Han et al . ( 2015 ) ) . By taking the reduction of the bit-width for weights and activations to the extreme , i.e. , a single bit , one obtains a Binary Neural Network . Binary Neural Networks have several advantageous properties , i.e. , a 32× reduction in memory requirements and the forward pass can be implemented using XNOR operations and bit-counting , which results in a 58× speedup on CPU ( Rastegari et al. , 2016 ) . Moreover , Binary Neural Networks are more robust to adversarial examples ( Galloway et al. , 2018 ) . Shayer et al . ( 2018 ) introduced a probabilistic training method for Neural Networks with binary weights , but allow for full precision activations . In this paper , we propose a probabilistic training method for Neural Networks with both binary weights and binary activations , which are even more memory and computation efficient . In short , obtain a closed form forward pass for probabilistic neural networks if we constrain the input and weights to binary ( random ) variables . The output of the Multiply and Accumulate ( MAC ) operations , or pre-activation , is approximated using a factorized Normal distribution . Subsequently , we introduce stochastic versions of Max-Pooling and Batch Normalization that allow us to propagate the pre-activatoins throughout a single layer . By applying the sign ( · ) activation function to the random pre-activation , we not only obtain a distribution over binary activations , it also allows for backpropagation through the sign ( · ) operation . This is especially convenient as this in a deterministic Neural Network all gradient information is zeroed out when using sign as activation . We explore two different methods for training this probabilistic binary neural network : In the first method the activation distribution of layer l is propagated to layer ( l + 1 ) , which means the MAC operation is performed on two binary random variables . In the second method the binary activation is sampled as the last operation in a layer using the concrete relaxation Maddison et al . ( 2016 ) . This can be thought of as a form of local reparametrization Kingma et al . ( 2015 ) . We call the networks obtained using these methods PBNet and PBNet-S , respectively . At test time , we obtain a single deterministic Binary Neural Network , an ensemble of Binary Neural Networks by sampling from the parameter distribution , or a Ternary Neural Network based on the Binary weight distribution . An advantage of our method is that we can take samples from the parameter distribution indefinitely—without retraining . Hence , this method allows for anytime ensemble predictions and uncertainty estimates . Note that while in this work we only consider the binary case , our method supports any discrete distribution over weights and activations . 2 PROBABILISTIC BINARY NEURAL NETWORK . Algorithm 1 : Pseudo code for forward pass of single layer in PBNet ( -S ) . al−1 denotes the activation of the previous layer , B the random binary weight matrix , τ is the temperature used for the concrete distribution , f ( · , · ) the linear transformation used in the layer , > 0 a small constant for numerical stability , D the dimensionality of the inner product in f , and γ & β are the parameters for batch normalization . Input : al−1 , B ∼ p ( B ) , τ , f ( · , · ) , , γ , β Result : Binary activation al // CLT approximation if al−1 is a binary random variable then µ = f ( E [ B ] , E [ al−1 ] ) ; σ2 = D − f ( ( E [ B ] ) 2 , ( E [ al−1 ] ) 2 ) ; else µ = f ( E [ B ] , al−1 ) ; σ2 = f ( V [ B ] , a2l−1 ) ; end // Batch normalization m = channel-wise-mean ( µ ) ; v = channel-wise-variance ( µ , σ2 , m ) ; µ = γ ( µ−m ) / √ v + + β ; σ2 = γ2σ2/ ( v + ) ; // Max pooling if max pooling required then n ∼ N ( 0 , I ) ; s = µ + σ n ; ι = max-pooling-indices ( s ) ; µ , σ2 = select-at-indices ( µ , σ2 , ι ) ; end // Binarization and sampling p = 1− Φ ( 0|µ , σ2 ) ; if sample activation then al ∼ BinaryConcrete ( p , τ ) ; return al ; else return Binary ( p ) end In this section we introduce the probabilistic setting of the PBNet . Moreover , the approximation of the distribution on the pre-activations is introduced . For an explanation of the other operations in the PBNet , see Section 2.1 for the activation , Section 2.1.1 for the sampling of activations , and Section 2.2 for Pooling and Normalization . We aim to train a probabilistic Binary Neural Network . As such , we pose a binary distribution over the weights of the network and optimize the parameters of this distribution instead of the parameters directly . This way , we obtain a distribution over parameters , but also deal with the inherent discreteness of a Binary Neural Network . Given an objective function L ( · ) , this approach can be thought of in terms of the variational optimization framework Staines & Barber ( 2012 ) . Specifically , by optimizing the parameters of the weight distributions , we optimize a bound on the actual loss : min B L ( B ) ≤ Eqθ ( B ) [ L ( B ) ] , ( 1 ) where B are the binary weights of the network and qθ ( B ) is a distribution over the binary weights . For qθ ( B ) a slight reparametrization of the Bernoulli distribution is used , which we will refer to as the Binary distribution . This distribution is parameterized by θ ∈ [ −1 , 1 ] and is defined by : a ∼ Binary ( θ ) ⇐⇒ a+ 1 2 ∼ Bernoulli ( θ + 1 2 ) . ( 2 ) For the properties of this distribution , please refer to Appendix A . We will now consider using the Binary distribution for both the weights and the activations in a Neural Network . Since the pre-activations in a Neural Network are computed using MAC operations , which are the same for each value in the pre-activation , we will only consider a single value in our discussion here . Let w ∼ Binary ( θ ) and h ∼ Binary ( φ ) be the weight and input random variable for a given layer . As such , the innerproduct between the weights and input is distributed according to a translated and scaled Poisson binomial distribution : w · h +D 2 ∼ PoiBin ( 2 [ θ φ ] − 1 ) . ( 3 ) Where D is the dimensionality of h and w and denotes element-wise multiplication . See the picket fence on the top in Figure 1 for an illustration of the PMF of a Poisson binomial distribution . Although the scaled and translated Poisson binomial distribution is the exact solution for the inner product between the weight and activation random variables , it is hard to work with in subsequent layers . For this reason , and the fact that the Poisson binomial distribution is well approximated by a Normal distribution ( Wang & Manning , 2013 ) , we use a Normal approximation to the Poisson binomial distribution , which allows for easier manipulations . Using the properties of the Binary distribution and the Poisson binomial distribution , the approximation for the pre-activation a is given by : a = w · h ∼̇ N ( D∑ d=1 θdφd , D − D∑ d=1 θ2dφ 2 d ) . ( 4 ) Note that , this is exactly the approximation one would obtain by using the Lyapunov Central Limit Theorem ( CLT ) , which was used by Shayer et al . ( 2018 ) . This allows us to obtain a close approximation to the pre-activation distribution , which we can propagate through the layer and/or network . So far , only the MAC operation in a given layer is discussed , in Section 2.1 application of the binary activation is discussed and in Section 2.1 . The stochastic versions of Batch Normalization and Max Pooling are introduced in Section 2.2 . For specifics on sampling the binary activation , see Section 2.1.1 . The full forward pass for a single layer is given in detail in Algorithms 1 . 2.1 STOCHASTIC BINARY ACTIVATION . Since the output of a linear operation using binary inputs is not restricted to be binary , it is required to apply a binarization operation to the pre-activation in order to obtain binary activations . Various works – e.g. , Hubara et al . ( 2016 ) and Rastegari et al . ( 2016 ) – use either deterministic or stochastic binarization functions , i.e. , bdet ( a ) = { +1 if a ≥ 0 −1 otherwise bstoch ( a ) = { +1 with probability p = sigmoid ( a ) −1 with probability 1− p . ( 5 ) In our case the pre-activations are random variables . Hence , applying a deterministic binarization function to a random pre-activations results in a stochastic binary activation . Specifically , let ai ∼ N ( µi , σ2i ) be a random pre-ctivation obtained using the normal approximation , as introduced in the previous section , then the activation ( after binarization ) is again given as a Binary random variable '' . Interestingly , the Binary probability can be computed in closed form by evaluating the probability density that lies above the binarization threshold : hi = bdet ( ai ) ∼ Binary ( qi ) , qi = 1− Φ ( 0|µi , σ2i ) , ( 6 ) where Φ ( ·|µ , σ2 ) denotes the CDF of N ( µ , σ2 ) . Applying the binarization function to a random pre-activation has two advantages . First , the derivatives ∂qi/∂µi and ∂qi/∂σi are not zero almost everywhere , in contrast to the derivatives of bdet and bstoch when applied to a deterministic input . Second , the distribution over hi reflects the true uncertainty about the sign of the activation , given the stochastic weights , whereas bstoch uses the magnitude of the pre-activation as a substitute . For example , a pre-activation with a high positive magnitude and high variance will be deterministically mapped to 1 by bstoch . In contrast , our method takes the variance into account and correctly assigns some probability mass to−1 . See Figure 1 for a graphical depiction of the stochastic binary activation .

Summary: This work presents a probabilistic training method for binary Neural Network with stochastic versions of Batch Normalization and max pooling. By sampling from the weight distribution an ensemble of Binary Neural Networks could further improve the performance. In the experimental section, the authors compare proposed PBNet with Binarized NN (Hubara et al., 2016) in two image datasets (MNIST and CIFAR10).