Variance Networks: When Expectation Does Not Meet Your Expectations

1 INTRODUCTION . Modern deep neural networks are usually trained in a stochastic setting . They use different stochastic layers ( Srivastava et al . ( 2014 ) ; Wan et al . ( 2013 ) ) and stochastic optimization techniques ( Welling & Teh ( 2011 ) ; Kingma & Ba ( 2014 ) ) . Stochastic methods are used to reduce overfitting ( Srivastava et al . ( 2014 ) ; Wang & Manning ( 2013 ) ; Wan et al . ( 2013 ) ) , estimate uncertainty ( Gal & Ghahramani ( 2016 ) ; Malinin & Gales ( 2018 ) ) and to obtain more efficient exploration for reinforcement learning ( Fortunato et al . ( 2017 ) ; Plappert et al . ( 2017 ) ) algorithms . Bayesian deep learning provides a principled approach to training stochastic models ( Kingma & Welling ( 2013 ) ; Rezende et al . ( 2014 ) ) . Several existing stochastic training procedures have been reinterpreted as special cases of particular Bayesian models , including , but not limited to different versions of dropout ( Gal & Ghahramani ( 2016 ) ) , drop-connect ( Kingma et al . ( 2015 ) ) , and even the stochastic gradient descent itself ( Smith & Le ( 2018 ) ) . One way to create a stochastic neural network from an existing deterministic architecture is to replace deterministic weights wij with random weights ŵij ∼ q ( ŵij |φij ) ( Hinton & Van Camp ( 1993 ) ; Blundell et al . ( 2015 ) ) . During training , a distribution over the weights is learned instead of a single point estimate . Ideally one would want to average the predictions over different samples of such distribution , which is known as test-time averaging , model averaging or ensembling . However , test-time averaging is impractical , so during inference the learned distribution is often discarded , and only the expected values of the weights are used instead . This heuristic is known as mean propagation or the weight scaling rule ∗Equal contribution ( Srivastava et al . ( 2014 ) ; Goodfellow et al . ( 2016 ) ) , and is widely and successfully used in practice ( Srivastava et al . ( 2014 ) ; Kingma et al . ( 2015 ) ; Molchanov et al . ( 2017 ) ) . In our work we study the an extreme case of stochastic neural network where all the weights in one or more layers have zero means and trainable variances , e.g . wij ∼ N ( 0 , σ2ij ) . Although no information get stored in the expected values of the weights , these models can learn surprisingly well and achieve competitive performance . Our key results can be summarized as follows : 1 . We introduce variance layers , a new kind of stochastic layers that store information only in the variances of its weights , keeping the means fixed at zero , and mapping the objects into zero-mean distributions over activations . The variance layer is a simple example when the weight scaling rule ( Srivastava et al . ( 2014 ) ) fails . 2 . We draw the connection between neural networks with variance layers ( variance networks ) and conventional Bayesian deep learning models . We show that several popular Bayesian models ( Kingma et al . ( 2015 ) ; Molchanov et al . ( 2017 ) ) converge to variance networks , and demonstrate a surprising effect – a less flexible posterior approximation may lead to much better values of the variational inference objective ( ELBO ) . 3 . Finally , we demonstrate that variance networks perform surprisingly well on a number of deep learning problems . They achieve competitive classification accuracy , are more robust to adversarial attacks and provide good exploration in reinforcement learning problems . 2 STOCHASTIC NEURAL NETWORKS . A deep neural network is a function that outputs the predictive distribution p ( t |x , W ) of targets t given an object x and weights W . Recently , stochastic deep neural networks — models that exploit some kind of random noise — have become widely popular ( Srivastava et al . ( 2014 ) ; Kingma et al . ( 2015 ) ) . We consider a special case of stochastic deep neural networks where the parameters W are drawn from a parametric distribution q ( W |φ ) . During training the parameters φ are adjusted to the training data ( X , T ) by minimizing the sum of the expected negative log-likelihood and an optional regularization term R ( φ ) . In practice this objective equation 1 is minimized using one-sample minibatch gradient estimation . − Eq ( W |φ ) log p ( T |X , W ) +R ( φ ) → min φ ( 1 ) This training procedure arises in many conventional techniques of training stochastic deep neural networks , such as binary dropout ( Srivastava et al . ( 2014 ) ) , variational dropout ( Kingma et al . ( 2015 ) ) and drop-connect ( Wan et al . ( 2013 ) ) . The exact predictive distribution Eq ( W |φ ) p ( t |x , W ) for such models is usually intractable . However , it can be approximated using K independent samples of the weights equation 2 . This technique is known as test-time averaging . Its complexity increases linearly in K. p ( t |x ) = Eq ( W |φ ) p ( t |x , W ) ' 1 K K∑ k=1 p ( t |x , Ŵk ) , Ŵk ∼ q ( W |φ ) ( 2 ) In order to obtain a more computationally efficient estimation , it is common practice to replace the weights with their expected values equation 3 . This approximation is known as the weight scaling rule ( Srivastava et al . ( 2014 ) ) . Eq ( W |φ ) p ( t |x , W ) ≈ p ( t |x , EqW ) . ( 3 ) As underlined by Goodfellow et al . ( 2016 ) , while being mathematically incorrect , this rule still performs very well on practice . The success of weight scaling rule implies that a lot of learned information is concentrated in the expected value of the weights . In this paper we consider symmetric weight distributions q ( W |φ ) = q ( −W |φ ) . Such distributions can not store any information about the training data in their means as EW = 0 . In the case of conventional layers with symmetric weight distribution , the predictive distribution p ( t |x , EW = 0 ) does not depend on the object x . Thus , the weight scaling rule results in a random guess quality predictions . We would refer to such layers as the variance layers , and will call neural networks that at least one variance layer the variance networks . 3 VARIANCE LAYER . In this section , we consider a single fully-connected layer1 with I input neurons and O output neurons , before a non-linearity . We denote an input vector by ak ∈ RI and an output vector by bk ∈ RO , a weight matrix by W ∈ RI×O . The output of the layer is computed as b > k = a > kW . A standard normal distributed random variable is denoted by ∼ N ( 0 , 1 ) . Most stochastic layers mostly rely on the expected values of the weights to make predicitons . We introduce a Gaussian variance layer that by design can not store any information in mean values of the weights , as opposed to conventional stochastic layers . In a Gaussian variance layer the weights follow a zero-mean Gaussian distribution wij = σij · ij ∼ N ( wij | 0 , σ2ij ) , so the information can be stored only in the variances of the weights . 3.1 DISTRIBUTION OF ACTIVATIONS . To get the intuition of how the variance layer can output any sensible values let ’ s take a look at the activations of this layer . A Gaussian distribution over the weights implies a Gaussian distribution over the activations ( Wang & Manning ( 2013 ) ; Kingma et al . ( 2015 ) ) . This fact is used in the local reparameterization trick ( Kingma et al . ( 2015 ) ) , and we also rely on it in our experiments . bkj = I∑ i=1 akiµij + j · √√√√ I∑ i=1 a2kiσ 2 ij . Conventional layer bkj = j · √√√√ I∑ i=1 a2kiσ 2 ij . Variance layer ( 4 ) In Gaussian variance layer an expectation of bmj is exactly zero , so the first term in eq . equation 4 disappears . During training , the layer can only adjust the variances ∑I i a 2 kiσ 2 ij of the output . It means that each object is encoded by a zero-centered fully-factorized multidimensional Gaussian rather than by a single point / a non-zero-mean Gaussian . The job of the following layers is then to classify such zero-mean Gaussians with different variances . It turns out that such encodings are surprisingly robust and can be easily discriminated by the following layers . 3.2 TOY PROBLEM . We illustrate the intuition on a toy classification problem . Object of four classes were generated from Gaussian distributions with µ ∈ { ( 3 , 3 ) , ( 3 , 10 ) , ( 10 , 3 ) , ( 10 , 10 ) } and identity covariance matrices . A classification network consisted of six fully-connected layers with ReLU non-linearities , where the fifth layer is a bottleneck variance layer with two output neurons . In Figure 1 we plot the activations of the variance layer that were sampled similar to equation equation 4 . The exact expressions are presented in Appendix E. Different colors correspond to different classes . We found that a variance bottleneck layer can learn two fundamentally different kinds of representations that leads to equal near perfect performance on this task . In the first case the same information is stored in two available neurons ( Figure 1a ) . We see this effect as a kind of in-network ensembling : averaging over two samples from the same distribution results in a more robust prediction . Note that in this case the information about four classes is robustly represented by essentially only one neuron . In the second case the information stored in these two neurons is different . Each neuron can be either activated ( i.e . have a large variance ) or deactivated ( i.e . have a low variance ) . Activation of the first neuron corresponds to either class 1 or class 2 , and activation of the second neuron corresponds to either class 1 or class 3 . This is also enough to robustly encode all four classes . Other combinations of these two cases are possible , but in principle , we see how the variances of the activations can be used to encode the same information as the means . As shown in Section 6 , although this representation is rather noisy , it robustly yields a relatively high accuracy even using only one sample , and test-time averaging allows to raise it to competitive levels . We observe the same behaviour with real tasks . See Appendix D for the visualization of the embeddings , learned by a variance LeNet-5 architecture on the MNIST dataset . One could argue that the non-linearity breaks the symmetry of the distribution . This could mean that the expected activations become non-zero and could be used for prediction . However , this is a fallacy : we argue that the correct intuition is that the model learns to distinguish activations of different variance . To prove this point , we train variance networks with antisymmetric non-linearities like ( e.g . a hyperbolic tangent ) without biases . That would make the mean activation of a variance layer exactly zero even after a non-linearity . See Appendix C for more details .

Summary: This paper introduced a new stochastic layer termed variance layer for Bayesian deep learning, where the posterior on weight is a zero-mean symmetric distribution (e.g., Gaussian, Bernoulli, Uniform). The paper showed that under 3 different prior distributions, the Gaussian Dropout layer can converge to variance layer. Experiments verified that it can achieve similar accuracies as conventional binary dropout in image classification and reinforcement learning tasks, is more robust to adversarial attacks, and can be used to sparsify deep models.