VARIANCE NETWORKS: WHEN EXPECTATION DOES NOT MEET YOUR EXPECTATIONS
Year: 2019

Abstract:
Ordinary stochastic neural networks mostly rely on the expected values of their weights to make predictions, whereas the induced noise is mostly used to capture the uncertainty, prevent overfitting and slightly boost the performance through test-time averaging. In this paper, we introduce variance layers, a different kind of stochastic layers. Each weight of a variance layer follows a zero-mean distribution and is only parameterized by its variance. It means that each object is represented by a zero-mean distribution in the space of the activations. We show that such layers can learn surprisingly well, can serve as an efficient exploration tool in reinforcement learning tasks and provide a decent defense against adversarial attacks. We also show that a number of conventional Bayesian neural networks naturally converge to such zero-mean posteriors. We observe that in these cases such zero-mean parameterization leads to a much better training objective than more flexible conventional parameterizations where the mean is being learned.

heading: 1 INTRODUCTION
text: Modern deep neural networks are usually trained in a stochastic setting. They use different stochastic layers (Srivastava et al. (2014); Wan et al. (2013)) and stochastic optimization techniques (Welling & Teh (2011); Kingma & Ba (2014)). Stochastic methods are used to reduce overfitting (Srivastava et al. (2014); Wang & Manning (2013); Wan et al. (2013)), estimate uncertainty (Gal & Ghahramani (2016); Malinin & Gales (2018)) and to obtain more efficient exploration for reinforcement learning (Fortunato et al. (2017); Plappert et al. (2017)) algorithms.
Bayesian deep learning provides a principled approach to training stochastic models (Kingma & Welling (2013); Rezende et al. (2014)). Several existing stochastic training procedures have been reinterpreted as special cases of particular Bayesian models, including, but not limited to different versions of dropout (Gal & Ghahramani (2016)), drop-connect (Kingma et al. (2015)), and even the stochastic gradient descent itself (Smith & Le (2018)). One way to create a stochastic neural network from an existing deterministic architecture is to replace deterministic weights wij with random weights ŵij ∼ q(ŵij |φij) (Hinton & Van Camp (1993); Blundell et al. (2015)). During training, a distribution over the weights is learned instead of a single point estimate. Ideally one would want to average the predictions over different samples of such distribution, which is known as test-time averaging, model averaging or ensembling. However, test-time averaging is impractical, so during inference the learned distribution is often discarded, and only the expected values of the weights are used instead. This heuristic is known as mean propagation or the weight scaling rule
∗Equal contribution
(Srivastava et al. (2014); Goodfellow et al. (2016)), and is widely and successfully used in practice (Srivastava et al. (2014); Kingma et al. (2015); Molchanov et al. (2017)).
In our work we study the an extreme case of stochastic neural network where all the weights in one or more layers have zero means and trainable variances, e.g. wij ∼ N (0, σ2ij). Although no information get stored in the expected values of the weights, these models can learn surprisingly well and achieve competitive performance. Our key results can be summarized as follows:
1. We introduce variance layers, a new kind of stochastic layers that store information only in the variances of its weights, keeping the means fixed at zero, and mapping the objects into zero-mean distributions over activations. The variance layer is a simple example when the weight scaling rule (Srivastava et al. (2014)) fails.
2. We draw the connection between neural networks with variance layers (variance networks) and conventional Bayesian deep learning models. We show that several popular Bayesian models (Kingma et al. (2015); Molchanov et al. (2017)) converge to variance networks, and demonstrate a surprising effect – a less flexible posterior approximation may lead to much better values of the variational inference objective (ELBO).
3. Finally, we demonstrate that variance networks perform surprisingly well on a number of deep learning problems. They achieve competitive classification accuracy, are more robust to adversarial attacks and provide good exploration in reinforcement learning problems.

heading: 2 STOCHASTIC NEURAL NETWORKS
text: A deep neural network is a function that outputs the predictive distribution p(t |x,W ) of targets t given an object x and weights W . Recently, stochastic deep neural networks — models that exploit some kind of random noise — have become widely popular (Srivastava et al. (2014); Kingma et al. (2015)). We consider a special case of stochastic deep neural networks where the parameters W are drawn from a parametric distribution q(W |φ). During training the parameters φ are adjusted to the training data (X,T ) by minimizing the sum of the expected negative log-likelihood and an optional regularization term R(φ). In practice this objective equation 1 is minimized using one-sample minibatch gradient estimation.
− Eq(W |φ) log p(T |X,W ) +R(φ)→ min φ
(1)
This training procedure arises in many conventional techniques of training stochastic deep neural networks, such as binary dropout (Srivastava et al. (2014)), variational dropout (Kingma et al. (2015)) and drop-connect (Wan et al. (2013)). The exact predictive distribution Eq(W |φ)p(t |x,W ) for such models is usually intractable. However, it can be approximated using K independent samples of the weights equation 2. This technique is known as test-time averaging. Its complexity increases linearly in K.
p(t |x) = Eq(W |φ)p(t |x,W ) ' 1
K K∑ k=1 p(t |x, Ŵk), Ŵk ∼ q(W |φ) (2)
In order to obtain a more computationally efficient estimation, it is common practice to replace the weights with their expected values equation 3. This approximation is known as the weight scaling rule (Srivastava et al. (2014)).
Eq(W |φ)p(t |x,W ) ≈ p(t |x,EqW ). (3) As underlined by Goodfellow et al. (2016), while being mathematically incorrect, this rule still performs very well on practice. The success of weight scaling rule implies that a lot of learned information is concentrated in the expected value of the weights.
In this paper we consider symmetric weight distributions q(W |φ) = q(−W |φ). Such distributions cannot store any information about the training data in their means as EW = 0. In the case of conventional layers with symmetric weight distribution, the predictive distribution p(t |x,EW = 0) does not depend on the object x. Thus, the weight scaling rule results in a random guess quality predictions. We would refer to such layers as the variance layers, and will call neural networks that at least one variance layer the variance networks.

heading: 3 VARIANCE LAYER
text: In this section, we consider a single fully-connected layer1 with I input neurons and O output neurons, before a non-linearity. We denote an input vector by ak ∈ RI and an output vector by bk ∈ RO, a weight matrix by W ∈ RI×O. The output of the layer is computed as b>k = a>kW . A standard normal distributed random variable is denoted by ∼ N (0, 1). Most stochastic layers mostly rely on the expected values of the weights to make predicitons. We introduce a Gaussian variance layer that by design cannot store any information in mean values of the weights, as opposed to conventional stochastic layers. In a Gaussian variance layer the weights follow a zero-mean Gaussian distribution wij = σij · ij ∼ N (wij | 0, σ2ij), so the information can be stored only in the variances of the weights.

heading: 3.1 DISTRIBUTION OF ACTIVATIONS
text: To get the intuition of how the variance layer can output any sensible values let’s take a look at the activations of this layer. A Gaussian distribution over the weights implies a Gaussian distribution over the activations (Wang & Manning (2013); Kingma et al. (2015)). This fact is used in the local reparameterization trick (Kingma et al. (2015)), and we also rely on it in our experiments.
bkj = I∑ i=1 akiµij + j · √√√√ I∑ i=1 a2kiσ 2 ij .
Conventional layer
bkj = j · √√√√ I∑ i=1 a2kiσ 2 ij .
Variance layer
(4)
In Gaussian variance layer an expectation of bmj is exactly zero, so the first term in eq. equation 4 disappears. During training, the layer can only adjust the variances ∑I i a 2 kiσ 2 ij of the output. It means that each object is encoded by a zero-centered fully-factorized multidimensional Gaussian rather than by a single point / a non-zero-mean Gaussian. The job of the following layers is then to classify such zero-mean Gaussians with different variances. It turns out that such encodings are surprisingly robust and can be easily discriminated by the following layers.

heading: 3.2 TOY PROBLEM
text: We illustrate the intuition on a toy classification problem. Object of four classes were generated from Gaussian distributions with µ ∈ {(3, 3), (3, 10), (10, 3), (10, 10)} and identity covariance matrices. A classification network consisted of six fully-connected layers with ReLU non-linearities, where the fifth layer is a bottleneck variance layer with two output neurons. In Figure 1 we plot the activations of the variance layer that were sampled similar to equation equation 4. The exact expressions are presented in Appendix E. Different colors correspond to different classes.
We found that a variance bottleneck layer can learn two fundamentally different kinds of representations that leads to equal near perfect performance on this task. In the first case the same information is stored in two available neurons (Figure 1a). We see this effect as a kind of in-network ensembling: averaging over two samples from the same distribution results in a more robust prediction. Note that in this case the information about four classes is robustly represented by essentially only one neuron. In the second case the information stored in these two neurons is different. Each neuron can be either activated (i.e. have a large variance) or deactivated (i.e. have a low variance). Activation of the first neuron corresponds to either class 1 or class 2, and activation of the second neuron corresponds to either class 1 or class 3. This is also enough to robustly encode all four classes. Other combinations of these two cases are possible, but in principle, we see how the variances of the activations can be used to encode the same information as the means. As shown in Section 6, although this representation is rather noisy, it robustly yields a relatively high accuracy even using only one sample, and test-time averaging allows to raise it to competitive levels. We observe the same behaviour with real tasks. See Appendix D for the visualization of the embeddings, learned by a variance LeNet-5 architecture on the MNIST dataset.
One could argue that the non-linearity breaks the symmetry of the distribution. This could mean that the expected activations become non-zero and could be used for prediction. However, this is a fallacy: we argue that the correct intuition is that the model learns to distinguish activations of different variance. To prove this point, we train variance networks with antisymmetric non-linearities like (e.g. a hyperbolic tangent) without biases. That would make the mean activation of a variance layer exactly zero even after a non-linearity. See Appendix C for more details.

heading: 3.3 OTHER ZERO-MEAN SYMMETRIC DISTRIBUTIONS
text: Other types of variance layers may exploit different kinds of multiplicative or additive symmetric zero-mean noise distributions . These types of noise include, but not limited to:
• Gaussian variance layer: q(wij) = σij · ij , where ij ∼ N (0, 1)
• Bernoulli variance layer: q(wij) = θij · (2 ij − 1), where ij ∼ Bernoulli( 12 )
• Uniform variance layer: q(wij) = θij · ij , where ij ∼ Uniform(−1, 1),
In all these models the learned information is stored only in the variances of the weights. Applied to these type of models, the weight scaling rule (eq. equation 3) will result in the random guess performance, as the mean of the weights is equal to zero. Note that we cannot perform an exact local reparameterization trick for Bernoulli or Uniform noise. We can however use moment-matching techniques similar to fast dropout (Wang & Manning (2013)). Under fast dropout approximation all three cases would be equivalent to a Gaussian variance layer.
We were able to train a LeNet-5 architecture (Caffe (2014)) on the MNIST dataset with the first dense layer being a Gaussian variance layer up to 99.3 accuracy, and up to up to 98.7 accuracy with a Bernoulli or a uniform variance layer. Such gap in the performance is due to the lack of the local reparameterization trick for Bernoulli or uniform random weights. The complete results for the Gaussian variance layer are presented in Section 6.
1In this section we consider fully-connected layers for simplicity. The same applies to convolutional layers and other linear transformations. In the experiments we consider both fully-connected and convolutional variance layers.

heading: 4 RELATION TO BAYESIAN DEEP LEARNING
text: In this section we review several Gaussian dropout posterior models with different prior distributions over the weights. We show that the Gaussian dropout layers may converge to variance layers in practice.

heading: 4.1 STOCHASTIC VARIATIONAL INFERENCE
text: Doubly stochastic variational inference (DSVI) (Titsias & Lázaro-Gredilla (2014)) with the (local) reparameterization trick (Kingma & Welling (2013); Kingma et al. (2015)) can be considered as a special case of training with noise, described by eq. equation 1. Given a likelihood function p(t |x,W ) and a prior distribution p(W ), we would like to approximate the posterior distribution p(W |Xtrain, Ttrain) ≈ q(W |φ) over the weights W . This is performed by maximization of the variational lower bound (ELBO) w.r.t. the parameters φ of the posterior approximation q(W |φ)
L(φ) = Eq(W |φ) log p(T |X,W )−KL(q(W |φ) ‖ p(W ))→ max φ . (5)
The variational lower bound consists of two parts. One is the expected log likelihood term Eq(W |φ) log p(T |X,W ) that reflects the predictive performance on the training set. The other is the KL divergence term KL(q(W |φ) ‖ p(W )) that acts as a regularizer and allows us to capture the prior knowledge p(W ).

heading: 4.2 MODELS
text: We consider the Gaussian dropout approximate posterior that is a fully-factorized Gaussian wij ∼ N (wij |µij , αµ2ij) (Kingma et al. (2015)) with the Gaussian “dropout rate” α shared among all weights of one layer. We explore the following prior distributions:
Symmetric log-uniform distribution p(wij) ∝ 1|wij | is the prior used in variational dropout (Kingma et al. (2015); Molchanov et al. (2017)). The KL-term for the Gaussian dropout posterior
turns out to be a function of α and can be expressed as follows:
KL(N (µij , αµ2ij) ‖LogU(wij)) = (6)
= const− 1 2 logαµ2 + 1 2 Eε∼N (0,1) logµ2
( 1 + √ αε )2 = (7)
= const− 1 2 logα+ Eε∼N (0,1) log ∣∣1 +√αε∣∣ (8) This KL-term can be estimated using one MC sample, or accurately approximated. In our experiments, we use the approximation, proposed in Sparse Variational Dropout (Molchanov et al. (2017)).
Student’s t-distribution with ν degrees of freedom is a proper analog of the log-uniform prior, as the log-uniform prior is a special case of the Student’s t-distribution with zero degrees of freedom. As ν goes to zero, the KL-term for the Student’s t-prior equation 10 approaches the KL-term for the log-uniform prior equation 7.
KL(N (µij , αµ2ij) ‖ St(wij | ν)) = (9)
= const− 1 2 logαµ2 + ν + 1 2 Eε∼N (0,1) log
( ν + µ2(1 + √ αε)2 ) (10)
As the use of the improper log-uniform prior in neural networks is questionable (Hron et al. (2017)), we argue that the Student’s t-distribution with diminishing values of ν results in the same properties of the model, but leads to a proper posterior. We use one sample to estimate the expectation equation 10 in our experiments, and use ν = 10−16.
Automatic Relevance Determination prior p(wij) = N (wij | 0, λ2ij) (Neal (1996); MacKay et al. (1994)) has been previously applied to linear models with DSVI (Titsias & Lázaro-Gredilla (2014)), and can be applied to Bayesian neural networks without changes. Following (Titsias & LázaroGredilla (2014)), we can show that in the case of the Gaussian dropout posterior, the optimal prior variance λ2ij would be equal to (α + 1)µ 2 ij , and the KL-term KL(q(W |φ) ‖ p(W )) would then be calculated as follows:
KL(N (µij , αµ2ij) ‖N (0, (α+ 1)µ2ij)) = log ( 1 + α−1 ) . (11)
Note that in the case of the log-uniform and the ARD priors, the KL-divergence between a zerocentered Gaussian and the prior is constant. For the ARD prior it is trivial, as the prior distribution p(w) is equal to the approximate posterior q(w) and the KL is zero. For the log-uniform distribution the proof is presented in Appendix F. Note that a zero-centered posterior is optimal in terms of these KL divergences. In the next section we will show that Gaussian dropout layers with these priors can indeed converge to variance layers.

heading: 4.3 CONVERGENCE TO VARIANCE LAYERS
text: As illustrated in Figure 2, in all three cases the KL term decreases in α and pushes α to infinity. We would expect the data term to limit the learned values of α, as otherwise the model would seem to be overregularized. Surprisingly, we find that in practice for some layers α’s may grow to essentially infinite values (e.g. α > 107). When this happens, the approximate posteriorN (µij , αµ2ij) becomes indistinguishable from its zero-mean approximation N (0, αµ2ij), as its standard deviation √ α|µij | becomes much larger than its mean µij . We prove that as α goes to infinity, the Maximum Mean Discrepancy (Gretton et al. (2012)) between the approximate posterior and its zero-mean approximation goes to zero. Theorem 1. Assume that αt −→ +∞ as t −→ +∞. Then the Gaussian dropout posterior qt(w) = ∏D i=1N (wi |µt,i, αtµ 2 t,i) becomes indistinguishable from its zero-centered approximation
q0t (w) = ∏D i=1N (wi | 0, αtµ 2 t,i) in terms of Maximum Mean Discrepancy:
MMD(q0t (w) ‖ qt(w)) ≤ √ 2D
π · 1√ αt (12)
lim t→+∞
MMD(q0t (w) ‖ qt(w)) = 0 (13)
The proof of this fact is provided in Appendix A. It is an important result, as MMD provides an upper bound equation 38 on the change in the predictions of the ensemble. It means that we can
replace the learned posteriorN (µij , αµ2ij) with its zero-centered approximationN (0, αµ2ij) without affecting the predictions of the model. In this sense we see that some layers in these models may converge to variance layers. Note that although α may grow to essentially infinite values, the mean and the variance of the corresponding Gaussian distribution remain finite. In practice, as α tends to infinity, the means µij tend to zero, and the variances σ2ij = αµ 2 ij converge to finite values.
During the beginning of training, the Gaussian dropout rates α are low, and the weights can be replaced with their expectations with no accuracy degradation. After the end of training, the dropout rates are essentially infinite, and all information is stored in the variances of the weights. In these two regimes the network behave very differently. If we track the train or test accuracy during training we can clearly see a kind of “phase transition” between these two regimes. See Figure 3 for details. We observe the same results for all mentioned prior distributions. The corresponding details are presented in Appendix B.

heading: 5 AVOIDING LOCAL OPTIMA
text: In this section we show how different parameterizations of the Gaussian dropout posterior influence the value of the variational lower bound and the properties of obtained solution. We consider the same objective that is used in sparse variational dropout model (Molchanov et al. (2017)), and consider the following parameterizations for the approximate posterior q(wij):
q(wij) zero-mean N (0, σ2ij) layer-wise N (µij , αµ2ij) neuron-wise N (µij , αjµ2ij) weight-wise N (µij , αijµ2ij) additive N (µij , σ2ij) (14)
Note that the additive and weight-wise parameterizations are equivalent and that the layer- and the neuron-wise parameterizations are their less flexible special cases. We would expect that a more flexible approximation would result in a better value of variational lower bound. Surprisingly, in practice we observe exactly the opposite: the simpler the approximation is, the better ELBO we obtain.
The optimal value of the KL term is achieved when all α’s are set to infinity, or, equivalently, the mean is set to zero, and the variance is nonzero. In the weight-wise and additive parameterization α’s for some weights get stuck in low values, whereas simpler parameterizations have all α’s converged to effectively infinite values. The KL term for such flexible parameterizations is orders of magnitude worse, resulting in a much lower ELBO. See Table 1 for further details. It means that a more flexible parameterization makes the optimization problem much more difficult. It potentially introduces a large amount of poor local optima, e.g. sparse solutions, studied in sparse variational dropout (Molchanov et al. (2017)). Although such solutions have lower ELBO, they can still be very useful in practice.

heading: 6 EXPERIMENTS
text: We perform experimental evaluation of variance networks on classification and reinforcement learning problems. Although all learned information is stored only in the variances, the models perform surprisingly well on a number of benchmark problems. Also, we found that variance networks
are more resistant to adversarial attacks than conventional ensembling techniques. All stochastic models were optimized with only one noise/weight sample per step. Experiments were implemented using PyTorch (Paszke et al. (2017)). The code is available at https://github.com/ da-molchanov/variance-networks.
6.1 CLASSIFICATION
We consider three image classification tasks, the MNIST (LeCun et al. (1998)), CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton (2009)) datasets. We use the LeNet-5-Caffe architecture (Caffe (2014)) as a base model for the experiments on the MNIST dataset, and a VGGlike architecture (Zagoruyko (2015)) on CIFAR10/100. As can be seen in Table 2, variance networks provide the same level of test accuracy as conventional binary dropout. In the variance LeNet-5 all 4 layers are variational dropout layers with layer-wise parameterization equation 14. Only the first fully-connected layer converged to a variance layer. In the variance VGG the first fully-connected layer and the last three convolutional layers are variational dropout lay-
ers with layer-wise parameterization equation 14. All these layers converged to variance layers. For the first dense layer in LeNet-5 the value of logα reached 6.9, and for the VGG-like architecture logα > 15 for convolutional layers and logα > 12 for the dense layer. As shown in Figure 4, all samples from the variance network posterior robustly yields a relatively high classification accuracy. In Appendix D we show that the intuition provided for the toy problem still holds for a convolutional network on the MNIST dataset.
Similar to conventional pruning techniques (Han et al. (2015); Wen et al. (2016)), we can prune variance layer in LeNet5 by the value of weights variances. Weights with low variances have small contributions into the variance of activation and can be ignored. In Figure 5 we show sparsity and accuracy of obtained model for different threshold values. Accuracy of the model is evaluated by test time averaging over 20 random samples. Up to 98% of the layer parameters can be zeroed out with no accuracy degradation.
6.2 REINFORCEMENT LEARNING
Recent progress in reinforcement learning shows that parameter noise may provide efficient exploration for a number of reinforcement learning algorithms (Fortunato et al. (2017); Plappert et al. (2017)). These papers utilize different types of Gaussian noise on the parameters of the model. However, increasing the level of noise while keeping expressive ability may lead to better exploration. In this section, we provide a proof-of-concept result for exploration with variance network parameter noise on two simple gym (Brockman et al. (2016)) environments with discrete action space: the CartPole-v0 (Barto et al. (1983)) and the Acrobot-v1 (Sutton (1996)). The approach we used is a policy gradient proposed by (Williams (1992); Sutton et al. (2000)).
In all experiments the policy was approximated with a three layer fully-connected neural network containing 256 neurons on each hidden layer. Parameter noise and variance network policies had the second hidden layer to be parameter noise (Fortunato et al. (2017); Plappert et al. (2017)) and variance (Section 3) layer respectively. For both methods we made a gradient update for each episode with individual samples of noise per episode. Stochastic gradient learning is performed using Adam (Kingma & Ba (2014)). Results were averaged over nine runs with different random seeds. Figure 6 shows the train-
ing curves. Using the variance layer parameter noise the algorithm progresses slowly but tends to reach a better final result.
6.3 ADVERSARIAL EXAMPLES
Deep neural networks suffer from adversarial attacks (Goodfellow et al. (2014)) — the predictions are not robust to even slight deviations of the input images. In this experiment we study the robustness of variance networks to targeted adversarial attacks.
The experiment was performed on CIFAR10 (Krizhevsky & Hinton (2009)) dataset on a VGG-like architecture (Zagoruyko (2015)). We build target adversarial attacks using the iterative fast sign algorithm (Goodfellow et al. (2014)) with a fixed step length ε = 0.5, and report the successful attack rate (Figure 7). We compare our approach to the following baselines: a dropout network with test time averaging (Srivastava et al. (2014)), and deep ensembles (Lakshminarayanan et al. (2017)) and a deterministic network. We average over 10 samples in ensemble inference techniques. Deep
ensembles were constructed from five separate networks. All methods were trained without adversarial training (Goodfellow et al. (2014)). Our experiments show that variance network has better resistance to adversarial attacks. We also present the results with deep ensembles of variance networks (denoted variance ensembles) and show that these two techniques can be efficiently combined to improve the robustness of the network even further.

heading: 7 DISCUSSION
text: In this paper we introduce variance networks, surprisingly stable stochastic neural networks that learn only the variances of the weights, while keeping the means fixed at zero in one or several layers.
We show that such networks can still be trained well and match the performance of conventional models. Variance networks are more stable against adversarial attacks than conventional ensembling techniques, and can lead to better exploration in reinforcement learning tasks.
The success of variance networks raises several counter-intuitive implications about the training of deep neural networks:
• DNNs not only can withstand an extreme amount of noise during training, but can actually store information using only the variances of this noise. The fact that all samples from such zero-centered posterior yield approximately the same accuracy also provides additional evidence that the landscape of the loss function is much more complicated than was considered earlier (Garipov et al. (2018)).
• A popular trick, replacing some random variables in the network with their expected values, can lead to an arbitrarily large degradation of accuracy — up to a random guess quality prediction.
• Previous works used the signal-to-noise ratio of the weights or the layer output to prune excessive units (Blundell et al. (2015); Molchanov et al. (2017); Neklyudov et al. (2017)). However, we show that in a similar model weights or even a whole layer with an exactly zero SNR (due to the zero mean output) can be crucial for prediction and can’t be pruned by SNR only.
• We show that a more flexible parameterization of the approximate posterior does not necessarily yield a better value of the variational lower bound, and consequently does not necessarily approximate the posterior distribution better.
We believe that variance networks may provide new insights on how neural networks learn from data as well as give new tools for building better deep models.

heading: ACKNOWLEDGMENTS
text: We would like to thank Max Welling and Ekaterina Lobacheva for valuable discussions and feedback on the earliest version of this paper. Kirill Neklyudov and Dmitry Molchanov were supported by Samsung Research, Samsung Electronics.

heading: A PROOF OF THEOREM 1
text: Theorem 1. Assume that lim t→+∞ αt = +∞. Then the Gaussian dropout posterior qt(w) =∏D i=1N (wi |µt,i, αtµ2t,i) becomes indistinguishable from its zero-centered approximation q0t (w) =∏D i=1N (wi | 0, αtµ2t,i) in terms of Maximum Mean Discrepancy:
MMD(q0t (w) ‖ qt(w)) ≤ √ 2D
π · 1√ αt (15)
lim t→+∞
MMD(q0t (w) ‖ qt(w)) = 0 (16)
Proof. By the definition of the Maximum Mean Discrepancy, we have
MMD(q0t (w) ‖ qt(w)) = sup f∈C ‖f‖∞≤1 Eq0t (w)f(w)− Eqt(w)f(w), (17)
where the supremum is taken over the set of continuous functions, bounded by 1. Let’s reparameterize and join the expectations:
sup f∈C ‖f‖∞≤1 Eq0t (w)f(w)− Eqt(w)f(w) = sup f∈C ‖f‖∞≤1
Eε∼N (0,ID) [f( √ αtµt ε)− f(µt + √ αtµt ε)]
(18) Since linear transformations of the argument do not change neither the norm of the function, nor its continuity, we can hide the component-wise multiplication of ε by √ αtµt inside the function f(ε). This would not change the supremum.
sup f∈C ‖f‖∞≤1
Eε∼N (0,ID) [f( √ αtµt ε)− f(µt + √ αtµt ε)] = (19)
= sup f∈C ‖f‖∞≤1
Eε∼N (0,ID) [ f(ε)− f ( 1√ αt + ε )] (20)
There exists a rotation matrix R such that R( 1√αt , . . . , 1√ αt )> = ( √ D√ αt , 0, . . . , 0)>. As ε comes from an isotropic Gaussian ε ∼ N (0, ID), its rotation Rε would follow the same distribution Rε ∼ N (0, ID). Once again, we can incorporate this rotation into the function f without affecting the supremum.
sup f∈C ‖f‖∞≤1
Eε∼N (0,ID) [ f(ε)− f ( 1√ αt + ε )] = (21)
= sup f∈C ‖f‖∞≤1
Eε∼N (0,ID) [ f(R>Rε)− f ( R>R ( 1√ αt + ε ))] = (22)
= sup f∈C ‖f‖∞≤1
Eε∼N (0,ID) [ f(Rε)− f ( R ( 1√ αt + ε ))] = (23)
= sup f∈C ‖f‖∞≤1 Eε∼N (0,ID)
f(Rε)− f (√D√
αt , 0, . . . , 0
)> +Rε  = (24) = sup
f∈C ‖f‖∞≤1
E ε̂=Rε ε∼N (0,ID)
f(ε̂)− f (√D√
αt , 0, . . . , 0
)> + ε̂  = (25) = sup
f∈C ‖f‖∞≤1
Eε∼N (0,ID)
[ f(ε)− f (√ D√ αt + ε1, ε2, . . . , εD )] (26)
Let’s consider the integration over ε1 separately (φ(ε1) denotes the density of the standard Gaussian distribution):
sup f∈C ‖f‖∞≤1 Eε∼N (0,ID)
[ f(ε)− f (√ D√ αt + ε1, ε2, . . . , εD )] = (27)
= sup f∈C ‖f‖∞≤1
Eε\1∼N (0,ID−1) ∫ +∞ −∞ [ f(ε)− f (√ D√ αt + ε1, ε2, . . . , εD )] φ(ε1)dε1 = (28)
Next, we view f(ε1, . . . ) as a function of ε1 and denote its antiderivative as F 1(ε) = ∫ f(ε)dε1. Note that as f is bounded by 1, hence F 1 is Lipschitz in ε1 with a Lipschitz constant L = 1. It would allow us to bound its deviation
∣∣∣F 1(ε)− F 1 ( √D√αt + ε1, ε2, . . . , εD)∣∣∣ ≤ √D√αt . Let’s use integration by parts:
sup f∈C ‖f‖∞≤1
Eε\1∼N (0,ID−1) ∫ +∞ −∞ [ f(ε)− f (√ D√ αt + ε1, ε2, . . . , εD )] φ(ε1)dε1 = (29)
= sup f∈C ‖f‖∞≤1 Eε\1∼N (0,ID−1)
[( F 1(ε)− F 1 (√ D√ αt + ε1, ε2, . . . , εD )) φ(ε1) ∣∣∣∣∣ +∞
ε1=−∞
− (30)
− ∫ +∞ −∞ ( F 1(ε)− F 1 (√ D√ αt + ε1, ε2, . . . , εD )) dφ(ε1) ] (31)
The first term is equal to zero, as ( F 1(ε)− F 1 ( √ D√ αt + ε1, ε2, . . . , εD )) is bounded and φ(−∞) = φ(+∞) = 0. Using dφ(ε1) = −φ(ε1)ε1dε1, we obtain MMD(q0t (w) ‖ qt(w)) = (32)
= sup f∈C ‖f‖∞≤1
Eε\1∼N (0,ID−1) ∫ +∞ −∞ ( F 1(ε)− F 1 (√ D√ αt + ε1, ε2, . . . , εD )) φ(ε1)ε1dε1 (33)
Finally, we can use the Lipschitz property of F 1(ε) to bound this value:
MMD(q0t (w) ‖ qt(w)) ≤ (34)
≤ sup f∈C ‖f‖∞≤1
Eε\1∼N (0,ID−1) ∫ +∞ −∞ ∣∣∣∣∣F 1(ε)− F 1 (√ D√ αt + ε1, ε2, . . . , εD )∣∣∣∣∣φ(ε1)|ε1|dε1 ≤ (35) ≤ sup
f∈C ‖f‖∞≤1
Eε\1∼N (0,ID−1) ∫ +∞ −∞ √ D√ αt φ(ε1)|ε1|dε1 = √ 2D π · 1√ αt (36)
Thus, we obtain the following bound on the MMD: MMD(q0t (w) ‖ qt(w)) ≤ √ 2D
π · 1√ αt (37)
This bound goes to zero as αt goes to infinity.
As the output of a softmax network lies in the interval [0, 1], we obtain the following bound on the deviation of the prediction of the ensemble after applying the zero-mean approximation:∣∣∣Eq0t (w)p(t |x,w)− Eqt(w)p(t |x,w)∣∣∣ ≤ √ D 2π · 1√ αt ∀t,∀x (38)

heading: B PHASE TRANSITION PLOTS FOR DIFFERENT PRIORS

heading: C ANTI-SYMMETRIC NON-LINEARITIES
text: We have considered the following setting. We used a LeNet-5 network on the MNIST dataset with only tanh non-linearities and with no biases.
Note that it works well even if the second-to-last layer is a variance layer. It means that the zeromean variance-only encodings are robustly discriminated using a linear model.

heading: D MNIST VARIANCE ACTIVATIONS

heading: E LOCAL REPARAMETERIZATION TRICK FOR VARIANCE NETWORKS
text: Here we provide the expressions for the forward pass through a fully-connected and a convolutional variance layer with different parameterizations.
Fully-connected layer, q(wij) = N (µij , αijµ2ij):
bj = Din∑ i=1 µijai + εj √√√√Din∑ i=1 αijµ2ija 2 i (39)
Fully-connected layers, q(wij) = N (0, σ2ij):
bj = εj √√√√Din∑ i=1 a2iσ 2 ij (40)
For fully-connected layers εj ∼ N (0, 1) and all variables mentioned above are scalars. Convolutional layer, q(wijhw) = N (µijhw, αijhwµ2ijhw):
bj = Ai ? µi + εj √ A2i ? (αi µ2i ) (41)
Convolutional layers, q(wijhw) = N (0, σ2ijhw): bj = εj √ A2i ? σ 2 i (42)
In the last two equations denotes the component-wise multiplication, ? denotes the convolution operation, and the square and square root operations are component-wise. εjhw ∼ N (0, 1). All variables bj , µi, Ai, σi are 3D tensors. For all layers ε is sampled independently for each object in a mini-batch. The optimization is performed w.r.t. µ, logα or w.r.t. log σ, depending on the parameterization.

heading: F KL DIVERGENCE FOR ZERO-CENTERED PARAMETERIZATION
text: We show below that the KL divergence DKL(N (0, σ2) ‖LogU) is constant w.r.t. σ. DKL(N (0, σ2) ‖LogU) ∝ (43)
∝ −1 2 log 2πeσ2 − Ew∼N (0,σ2) log 1 |x| = (44)
= −1 2 log 2πeσ2 + Eε∼N (0,1) log |σε| = (45) = −1 2 log 2πe+ Eε∼N (0,1) log |ε| (46)

Review summaries:
This paper studies variance neural networks, which approximate the posterior of Bayesian neural networks with zero-mean Gaussian distributions. The inference results are surprisingly well though there is no information in the mean of the posterior. It further shows that the several variational dropout methods are closed related to the proposed method. The experiment indicates that the ELBO can actually better optimized with this restricted form of variational distribution. 

This paper introduced a new stochastic layer termed variance layer for Bayesian deep learning, where the posterior on weight is a zero-mean symmetric distribution (e.g., Gaussian, Bernoulli, Uniform). The paper showed that under 3 different prior distributions, the Gaussian Dropout layer can converge to variance layer. Experiments verified that it can achieve similar accuracies as conventional binary dropout in image classification and reinforcement learning tasks, is more robust to adversarial attacks, and can be used to sparsify deep models.