Title: VARIANCE NETWORKS: WHEN EXPECTATION DOES NOT MEET YOUR EXPECTATIONS

Abstract: Ordinary stochastic neural networks mostly rely on the expected values of their weights to make predictions, whereas the induced noise is mostly used to capture the uncertainty, prevent overfitting and slightly boost the performance through test-time averaging. In this paper, we introduce variance layers, a different kind of stochastic layers. Each weight of a variance layer follows a zero-mean distribution and is only parameterized by its variance. It means that each object is represented by a zero-mean distribution in the space of the activations. We show that such layers can learn surprisingly well, can serve as an efficient exploration tool in reinforcement learning tasks and provide a decent defense against adversarial attacks. We also show that a number of conventional Bayesian neural networks naturally converge to such zero-mean posteriors. We observe that in these cases such zero-mean parameterization leads to a much better training objective than more flexible conventional parameterizations where the mean is being learned.

1. INTRODUCTION:

Modern deep neural networks are usually trained in a stochastic setting. They use different stochastic layers (Srivastava et al. (2014) ; Wan et al. (2013) ) and stochastic optimization techniques (Welling & Teh (2011) ; Kingma & Ba (2014) ). Stochastic methods are used to reduce overfitting (Srivastava et al. (2014) ; Wang & Manning (2013) ; Wan et al. (2013) ), estimate uncertainty (Gal & Ghahramani (2016) ; Malinin & Gales (2018) ) and to obtain more efficient exploration for reinforcement learning (Fortunato et al. (2017) ; Plappert et al. (2017) ) algorithms.

Bayesian deep learning provides a principled approach to training stochastic models (Kingma & Welling (2013) ; Rezende et al. (2014) ). Several existing stochastic training procedures have been reinterpreted as special cases of particular Bayesian models, including, but not limited to different versions of dropout (Gal & Ghahramani (2016) ), drop-connect (Kingma et al. (2015) ), and even the stochastic gradient descent itself (Smith & Le (2018) ). One way to create a stochastic neural network from an existing deterministic architecture is to replace deterministic weights w ij with random weights ŵij ∼ q( ŵij | φ ij ) (Hinton & Van Camp (1993) ; Blundell et al. (2015) ). During training, a distribution over the weights is learned instead of a single point estimate. Ideally one would want to average the predictions over different samples of such distribution, which is known as test-time averaging, model averaging or ensembling. However, test-time averaging is impractical, so during inference the learned distribution is often discarded, and only the expected values of the weights are used instead. This heuristic is known as mean propagation or the weight scaling rule (Srivastava et al. (2014) ; Goodfellow et al. (2016) ), and is widely and successfully used in practice (Srivastava et al. (2014) ; Kingma et al. (2015) ; Molchanov et al. (2017) ).

In our work we study the an extreme case of stochastic neural network where all the weights in one or more layers have zero means and trainable variances, e.g. w ij ∼ N (0, σ 2 ij ). Although no information get stored in the expected values of the weights, these models can learn surprisingly well and achieve competitive performance. Our key results can be summarized as follows:

1. We introduce variance layers, a new kind of stochastic layers that store information only in the variances of its weights, keeping the means fixed at zero, and mapping the objects into zero-mean distributions over activations. The variance layer is a simple example when the weight scaling rule (Srivastava et al. (2014) ) fails.

2. We draw the connection between neural networks with variance layers (variance networks) and conventional Bayesian deep learning models. We show that several popular Bayesian models (Kingma et al. (2015) ; Molchanov et al. (2017) ) converge to variance networks, and demonstrate a surprising effect -a less flexible posterior approximation may lead to much better values of the variational inference objective (ELBO).

3. Finally, we demonstrate that variance networks perform surprisingly well on a number of deep learning problems. They achieve competitive classification accuracy, are more robust to adversarial attacks and provide good exploration in reinforcement learning problems.

2. STOCHASTIC NEURAL NETWORKS:

A deep neural network is a function that outputs the predictive distribution p(t | x, W ) of targets t given an object x and weights W . Recently, stochastic deep neural networks -models that exploit some kind of random noise -have become widely popular (Srivastava et al. (2014) ; Kingma et al. (2015) ). We consider a special case of stochastic deep neural networks where the parameters W are drawn from a parametric distribution q(W | φ). During training the parameters φ are adjusted to the training data (X, T ) by minimizing the sum of the expected negative log-likelihood and an optional regularization term R(φ). In practice this objective equation 1 is minimized using one-sample minibatch gradient estimation.

-

E q(W | φ) log p(T | X, W ) + R(φ) → min φ (1)

This training procedure arises in many conventional techniques of training stochastic deep neural networks, such as binary dropout (Srivastava et al. (2014) ), variational dropout (Kingma et al. (2015) ) and drop-connect (Wan et al. (2013) ). The exact predictive distribution

E q(W | φ) p(t | x, W )

for such models is usually intractable. However, it can be approximated using K independent samples of the weights equation 2. This technique is known as test-time averaging. Its complexity increases linearly in K.

EQUATION

In order to obtain a more computationally efficient estimation, it is common practice to replace the weights with their expected values equation 3. This approximation is known as the weight scaling rule (Srivastava et al. (2014) ).

EQUATION

As underlined by Goodfellow et al. (2016) , while being mathematically incorrect, this rule still performs very well on practice. The success of weight scaling rule implies that a lot of learned information is concentrated in the expected value of the weights.

In this paper we consider symmetric weight distributions q(W | φ) = q(-W | φ). Such distributions cannot store any information about the training data in their means as EW = 0. In the case of conventional layers with symmetric weight distribution, the predictive distribution p(t | x, EW = 0) does not depend on the object x. Thus, the weight scaling rule results in a random guess quality predictions. We would refer to such layers as the variance layers, and will call neural networks that at least one variance layer the variance networks. 

3. VARIANCE LAYER:

In this section, we consider a single fully-connected layer 1 with I input neurons and O output neurons, before a non-linearity. We denote an input vector by a k ∈ R I and an output vector by b k ∈ R O , a weight matrix by W ∈ R I×O . The output of the layer is computed as b k = a k W . A standard normal distributed random variable is denoted by ∼ N (0, 1).

Most stochastic layers mostly rely on the expected values of the weights to make predicitons. We introduce a Gaussian variance layer that by design cannot store any information in mean values of the weights, as opposed to conventional stochastic layers. In a Gaussian variance layer the weights follow a zero-mean Gaussian distribution

w ij = σ ij • ij ∼ N (w ij | 0, σ 2 ij )

, so the information can be stored only in the variances of the weights.

3.1. DISTRIBUTION OF ACTIVATIONS:

To get the intuition of how the variance layer can output any sensible values let's take a look at the activations of this layer. A Gaussian distribution over the weights implies a Gaussian distribution over the activations (Wang & Manning (2013) ; Kingma et al. (2015) ). This fact is used in the local reparameterization trick (Kingma et al. (2015) ), and we also rely on it in our experiments.

b kj = I i=1 a ki µ ij + j • I i=1 a 2 ki σ 2 ij .

Conventional layer

b kj = j • I i=1 a 2 ki σ 2 ij .

Variance layer (4)

In Gaussian variance layer an expectation of b mj is exactly zero, so the first term in eq. equation 4 disappears. During training, the layer can only adjust the variances I i a 2 ki σ 2 ij of the output. It means that each object is encoded by a zero-centered fully-factorized multidimensional Gaussian rather than by a single point / a non-zero-mean Gaussian. The job of the following layers is then to classify such zero-mean Gaussians with different variances. It turns out that such encodings are surprisingly robust and can be easily discriminated by the following layers.

3.2. TOY PROBLEM:

We illustrate the intuition on a toy classification problem. Object of four classes were generated from Gaussian distributions with µ ∈ {(3, 3), (3, 10), (10, 3), (10, 10)} and identity covariance matrices. A classification network consisted of six fully-connected layers with ReLU non-linearities, where the fifth layer is a bottleneck variance layer with two output neurons. In Figure 1 we plot the activations of the variance layer that were sampled similar to equation equation 4. The exact expressions are presented in Appendix E. Different colors correspond to different classes.

We found that a variance bottleneck layer can learn two fundamentally different kinds of representations that leads to equal near perfect performance on this task. In the first case the same information is stored in two available neurons (Figure 1a ). We see this effect as a kind of in-network ensembling: averaging over two samples from the same distribution results in a more robust prediction. Note that in this case the information about four classes is robustly represented by essentially only one neuron.

In the second case the information stored in these two neurons is different. Each neuron can be either activated (i.e. have a large variance) or deactivated (i.e. have a low variance). Activation of the first neuron corresponds to either class 1 or class 2, and activation of the second neuron corresponds to either class 1 or class 3. This is also enough to robustly encode all four classes. Other combinations of these two cases are possible, but in principle, we see how the variances of the activations can be used to encode the same information as the means. As shown in Section 6, although this representation is rather noisy, it robustly yields a relatively high accuracy even using only one sample, and test-time averaging allows to raise it to competitive levels. We observe the same behaviour with real tasks. See Appendix D for the visualization of the embeddings, learned by a variance LeNet-5 architecture on the MNIST dataset.

One could argue that the non-linearity breaks the symmetry of the distribution. This could mean that the expected activations become non-zero and could be used for prediction. However, this is a fallacy: we argue that the correct intuition is that the model learns to distinguish activations of different variance. To prove this point, we train variance networks with antisymmetric non-linearities like (e.g. a hyperbolic tangent) without biases. That would make the mean activation of a variance layer exactly zero even after a non-linearity. See Appendix C for more details.

3.3. OTHER ZERO-MEAN SYMMETRIC DISTRIBUTIONS:

Other types of variance layers may exploit different kinds of multiplicative or additive symmetric zero-mean noise distributions . These types of noise include, but not limited to:

• Gaussian variance layer: q(w ij ) = σ ij • ij , where ij ∼ N (0, 1) • Bernoulli variance layer: q(w ij ) = θ ij • (2 ij -1), where ij ∼ Bernoulli( 1 2 ) • Uniform variance layer: q(w ij ) = θ ij • ij , where ij ∼ U nif orm(-1, 1),

In all these models the learned information is stored only in the variances of the weights. Applied to these type of models, the weight scaling rule (eq. equation 3) will result in the random guess performance, as the mean of the weights is equal to zero. Note that we cannot perform an exact local reparameterization trick for Bernoulli or Uniform noise. We can however use moment-matching techniques similar to fast dropout (Wang & Manning (2013) ). Under fast dropout approximation all three cases would be equivalent to a Gaussian variance layer.

We were able to train a LeNet-5 architecture (Caffe (2014) ) on the MNIST dataset with the first dense layer being a Gaussian variance layer up to 99.3 accuracy, and up to up to 98.7 accuracy with a Bernoulli or a uniform variance layer. Such gap in the performance is due to the lack of the local reparameterization trick for Bernoulli or uniform random weights. The complete results for the Gaussian variance layer are presented in Section 6. 

4. RELATION TO BAYESIAN DEEP LEARNING:

In this section we review several Gaussian dropout posterior models with different prior distributions over the weights. We show that the Gaussian dropout layers may converge to variance layers in practice.

4 

EQUATION

The variational lower bound consists of two parts. One is the expected log likelihood term E q(W | φ) log p(T | X, W ) that reflects the predictive performance on the training set. The other is the KL divergence term KL(q(W | φ) p(W )) that acts as a regularizer and allows us to capture the prior knowledge p(W ).

4.2. MODELS:

We consider the Gaussian dropout approximate posterior that is a fully-factorized Gaussian

EQUATION

)

) with the Gaussian "dropout rate" α shared among all weights of one layer. We explore the following prior distributions:

Symmetric log-uniform distribution p(w ij ) ∝ 1 |wij |

is the prior used in variational dropout (Kingma et al. (2015) ; Molchanov et al. (2017) ). The KL-term for the Gaussian dropout posterior turns out to be a function of α and can be expressed as follows:

EQUATION

This KL-term can be estimated using one MC sample, or accurately approximated. In our experiments, we use the approximation, proposed in Sparse Variational Dropout (Molchanov et al. (2017) ).

Student's t-distribution with ν degrees of freedom is a proper analog of the log-uniform prior, as the log-uniform prior is a special case of the Student's t-distribution with zero degrees of freedom.

As ν goes to zero, the KL-term for the Student's t-prior equation 10 approaches the KL-term for the log-uniform prior equation 7.

KL(N (µ ij , αµ 2 ij ) St(w ij | ν)) = (9) = const - 1 2 log αµ 2 + ν + 1 2 E ε∼N (0,1) log ν + µ 2 (1 + √ αε) 2 (10)

As the use of the improper log-uniform prior in neural networks is questionable (Hron et al. (2017) ), we argue that the Student's t-distribution with diminishing values of ν results in the same properties of the model, but leads to a proper posterior. We use one sample to estimate the expectation equation 10 in our experiments, and use ν = 10 -16 . (1996) ; MacKay et al. ( 1994)) has been previously applied to linear models with DSVI (Titsias & Lázaro-Gredilla (2014) ), and can be applied to Bayesian neural networks without changes. Following (Titsias & Lázaro-Gredilla ( 2014)), we can show that in the case of the Gaussian dropout posterior, the optimal prior variance λ 2 ij would be equal to (α + 1)µ 2 ij , and the KL-term KL(q(W | φ) p(W )) would then be calculated as follows:

Automatic Relevance Determination prior p(w ij ) = N (w ij | 0, λ 2 ij ) (Neal

EQUATION

)

Note that in the case of the log-uniform and the ARD priors, the KL-divergence between a zerocentered Gaussian and the prior is constant. For the ARD prior it is trivial, as the prior distribution p(w) is equal to the approximate posterior q(w) and the KL is zero. For the log-uniform distribution the proof is presented in Appendix F. Note that a zero-centered posterior is optimal in terms of these KL divergences. In the next section we will show that Gaussian dropout layers with these priors can indeed converge to variance layers.

4.3. CONVERGENCE TO VARIANCE LAYERS:

As illustrated in Figure 2 , in all three cases the KL term decreases in α and pushes α to infinity. We would expect the data term to limit the learned values of α, as otherwise the model would seem to be overregularized. Surprisingly, we find that in practice for some layers α's may grow to essentially infinite values (e.g. α > 10 7 ). When this happens, the approximate posterior N (µ ij , αµ 2 ij ) becomes indistinguishable from its zero-mean approximation N (0, αµ 2 ij ), as its standard deviation √ α|µ ij | becomes much larger than its mean µ ij . We prove that as α goes to infinity, the Maximum Mean Discrepancy (Gretton et al. (2012) ) between the approximate posterior and its zero-mean approximation goes to zero. Theorem 1. Assume that αt -→ +∞ as t -→ +∞. Then the Gaussian dropout posterior

qt(w) = D i=1 N (wi | µt,i, αtµ 2 t,i

) becomes indistinguishable from its zero-centered approximation

q 0 t (w) = D i=1 N (wi | 0, αtµ 2 t,i

) in terms of Maximum Mean Discrepancy:

EQUATION

The proof of this fact is provided in Appendix A. It is an important result, as MMD provides an upper bound equation 38 on the change in the predictions of the ensemble. It means that we can replace the learned posterior N (µ ij , αµ 2 ij ) with its zero-centered approximation N (0, αµ 2 ij ) without affecting the predictions of the model. In this sense we see that some layers in these models may converge to variance layers. Note that although α may grow to essentially infinite values, the mean and the variance of the corresponding Gaussian distribution remain finite. In practice, as α tends to infinity, the means µ ij tend to zero, and the variances σ 2 ij = αµ 2 ij converge to finite values. During the beginning of training, the Gaussian dropout rates α are low, and the weights can be replaced with their expectations with no accuracy degradation. After the end of training, the dropout rates are essentially infinite, and all information is stored in the variances of the weights. In these two regimes the network behave very differently. If we track the train or test accuracy during training we can clearly see a kind of "phase transition" between these two regimes. See Figure 3 for details. We observe the same results for all mentioned prior distributions. The corresponding details are presented in Appendix B.

5. AVOIDING LOCAL OPTIMA:

In this section we show how different parameterizations of the Gaussian dropout posterior influence the value of the variational lower bound and the properties of obtained solution. We consider the same objective that is used in sparse variational dropout model (Molchanov et al. (2017) ), and consider the following parameterizations for the approximate posterior q(w ij ):

EQUATION

Note that the additive and weight-wise parameterizations are equivalent and that the layer-and the neuron-wise parameterizations are their less flexible special cases. We would expect that a more flexible approximation would result in a better value of variational lower bound. Surprisingly, in practice we observe exactly the opposite: the simpler the approximation is, the better ELBO we obtain.

The optimal value of the KL term is achieved when all α's are set to infinity, or, equivalently, the mean is set to zero, and the variance is nonzero. In the weight-wise and additive parameterization α's for some weights get stuck in low values, whereas simpler parameterizations have all α's converged to effectively infinite values. The KL term for such flexible parameterizations is orders of magnitude worse, resulting in a much lower ELBO. See Table 1 for further details. It means that a more flexible parameterization makes the optimization problem much more difficult. It potentially introduces a large amount of poor local optima, e.g. sparse solutions, studied in sparse variational dropout (Molchanov et al. (2017) ). Although such solutions have lower ELBO, they can still be very useful in practice.

6. EXPERIMENTS:

We perform experimental evaluation of variance networks on classification and reinforcement learning problems. Although all learned information is stored only in the variances, the models perform surprisingly well on a number of benchmark problems. Also, we found that variance networks Figure 5 : Here we show that a variance layer can be pruned up to 98% with almost no accuracy degradation. We use magnitude-based pruning for σ's (replace all σ ij that are below the threshold with zeros), and report test-time-averaging accuracy.

are more resistant to adversarial attacks than conventional ensembling techniques. All stochastic models were optimized with only one noise/weight sample per step. Experiments were implemented using PyTorch (Paszke et al. (2017) ). The code is available at https://github.com/ da-molchanov/variance-networks. We consider three image classification tasks, the MNIST (LeCun et al. ( 1998)), CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton (2009) ) datasets. We use the LeNet-5-Caffe architecture (Caffe (2014) ) as a base model for the experiments on the MNIST dataset, and a VGGlike architecture (Zagoruyko (2015) ) on CIFAR-10/100. As can be seen in Table 2 , variance networks provide the same level of test accuracy as conventional binary dropout. In the variance LeNet-5 all 4 layers are variational dropout layers with layer-wise parameterization equation 14. Only the first fully-connected layer converged to a variance layer. In the variance VGG the first fully-connected layer and the last three convolutional layers are variational dropout layers with layer-wise parameterization equation 14. All these layers converged to variance layers. For the first dense layer in LeNet-5 the value of log α reached 6.9, and for the VGG-like architecture log α > 15 for convolutional layers and log α > 12 for the dense layer. As shown in Figure 4 , all samples from the variance network posterior robustly yields a relatively high classification accuracy.

6.1. CLASSIFICATION:

In Appendix D we show that the intuition provided for the toy problem still holds for a convolutional network on the MNIST dataset.

Similar to conventional pruning techniques (Han et al. (2015) ; Wen et al. ( 2016)), we can prune variance layer in LeNet5 by the value of weights variances. Weights with low variances have small contributions into the variance of activation and can be ignored. In Figure 5 we show sparsity and accuracy of obtained model for different threshold values. Accuracy of the model is evaluated by test time averaging over 20 random samples. Up to 98% of the layer parameters can be zeroed out with no accuracy degradation. Recent progress in reinforcement learning shows that parameter noise may provide efficient exploration for a number of reinforcement learning algorithms (Fortunato et al. (2017) ; Plappert et al. (2017) ). These papers utilize different types of Gaussian noise on the parameters of the model. However, increasing the level of noise while keeping expressive ability may lead to better exploration. In this section, we provide a proof-of-concept result for exploration with variance network parameter noise on two simple gym (Brockman et al. (2016) ) environments with discrete action space: the CartPole-v0 (Barto et al. (1983) ) and the Acrobot-v1 (Sutton (1996) ). The approach we used is a policy gradient proposed by (Williams (1992) ; Sutton et al. (2000) ).

In all experiments the policy was approximated with a three layer fully-connected neural network containing 256 neurons on each hidden layer. Parameter noise and variance network policies had the second hidden layer to be parameter noise (Fortunato et al. (2017) ; Plappert et al. (2017) ) and variance (Section 3) layer respectively. For both methods we made a gradient update for each episode with individual samples of noise per episode. Stochastic gradient learning is performed using Adam (Kingma & Ba ( 2014)). Results were averaged over nine runs with different random seeds. Figure 6 shows the training curves. Using the variance layer parameter noise the algorithm progresses slowly but tends to reach a better final result. Deep neural networks suffer from adversarial attacks (Goodfellow et al. (2014) ) -the predictions are not robust to even slight deviations of the input images.

6.3. ADVERSARIAL EXAMPLES:

In this experiment we study the robustness of variance networks to targeted adversarial attacks.

The experiment was performed on CIFAR-10 ( Krizhevsky & Hinton (2009) ) dataset on a VGG-like architecture (Zagoruyko (2015) ). We build target adversarial attacks using the iterative fast sign algorithm (Goodfellow et al. (2014) ) with a fixed step length ε = 0.5, and report the successful attack rate (Figure 7 ). We compare our approach to the following baselines: a dropout network with test time averaging (Srivastava et al. (2014) ), and deep ensembles (Lakshminarayanan et al. ( 2017)) and a deterministic network. We average over 10 samples in ensemble inference techniques. Deep ensembles were constructed from five separate networks. All methods were trained without adversarial training (Goodfellow et al. (2014) ). Our experiments show that variance network has better resistance to adversarial attacks. We also present the results with deep ensembles of variance networks (denoted variance ensembles) and show that these two techniques can be efficiently combined to improve the robustness of the network even further.

7. DISCUSSION:

In this paper we introduce variance networks, surprisingly stable stochastic neural networks that learn only the variances of the weights, while keeping the means fixed at zero in one or several layers.

We show that such networks can still be trained well and match the performance of conventional models. Variance networks are more stable against adversarial attacks than conventional ensembling techniques, and can lead to better exploration in reinforcement learning tasks.

The success of variance networks raises several counter-intuitive implications about the training of deep neural networks:

• DNNs not only can withstand an extreme amount of noise during training, but can actually store information using only the variances of this noise. The fact that all samples from such zero-centered posterior yield approximately the same accuracy also provides additional evidence that the landscape of the loss function is much more complicated than was considered earlier (Garipov et al. (2018) ). • A popular trick, replacing some random variables in the network with their expected values, can lead to an arbitrarily large degradation of accuracy -up to a random guess quality prediction. • Previous works used the signal-to-noise ratio of the weights or the layer output to prune excessive units (Blundell et al. (2015) ; Molchanov et al. (2017) ; Neklyudov et al. (2017) ). However, we show that in a similar model weights or even a whole layer with an exactly zero SNR (due to the zero mean output) can be crucial for prediction and can't be pruned by SNR only. • We show that a more flexible parameterization of the approximate posterior does not necessarily yield a better value of the variational lower bound, and consequently does not necessarily approximate the posterior distribution better.

We believe that variance networks may provide new insights on how neural networks learn from data as well as give new tools for building better deep models.

A PROOF OF THEOREM 1

Theorem 1.

Assume that lim t→+∞ α t = +∞. Then the Gaussian dropout posterior q t (w) = D i=1 N (w i | µ t,i , α t µ 2 t,i ) becomes indistinguishable from its zero-centered approximation q 0 t (w) = D i=1 N (w i | 0, α t µ 2 t,i ) in terms of Maximum Mean Discrepancy:

EQUATION

Proof. By the definition of the Maximum Mean Discrepancy, we have

EQUATION

where the supremum is taken over the set of continuous functions, bounded by 1. Let's reparameterize and join the expectations:

sup f ∈C f ∞≤1 E q 0 t (w) f (w) -E qt(w) f (w) = sup f ∈C f ∞≤1 E ε∼N (0,I D ) [f ( √ α t µ t ε) -f (µ t + √ α t µ t ε)]

(18) Since linear transformations of the argument do not change neither the norm of the function, nor its continuity, we can hide the component-wise multiplication of ε by √ α t µ t inside the function f (ε).

This would not change the supremum.

EQUATION

There exists a rotation matrix R such that R( 1 √ αt , . . . , 1 √ αt ) = ( √ D √ αt , 0, . . . , 0) . As ε comes from an isotropic Gaussian ε ∼ N (0, I D ), its rotation Rε would follow the same distribution Rε ∼ N (0, I D ). Once again, we can incorporate this rotation into the function f without affecting the supremum.

EQUATION

Let's consider the integration over ε 1 separately (φ(ε 1 ) denotes the density of the standard Gaussian distribution):

EQUATION

Next, we view f (ε 1 , . . . ) as a function of ε 1 and denote its antiderivative as

F 1 (ε) = f (ε)dε 1 . Note that as f is bounded by 1, hence F 1 is Lipschitz in ε 1 with a Lipschitz constant L = 1. It would allow us to bound its deviation F 1 (ε) -F 1 √ D √ αt + ε 1 , ε 2 , . . . , ε D ≤ √ D √ αt .

Let's use integration by parts:

EQUATION

The first term is equal to zero, as

F 1 (ε) -F 1 √ D √ αt + ε 1 , ε 2 , . . . , ε D is bounded and φ(-∞) = φ(+∞) = 0. Using dφ(ε 1 ) = -φ(ε 1 )ε 1 dε 1 , we obtain MMD(q 0 t (w) q t (w)) = (32) = sup f ∈C f ∞≤1 E ε \1 ∼N (0,I D-1 ) +∞ -∞ F 1 (ε) -F 1 √ D √ α t + ε 1 , ε 2 , . . . , ε D φ(ε 1 )ε 1 dε 1 (33)

Finally, we can use the Lipschitz property of F 1 (ε) to bound this value:

MMD(q 0 t (w) q t (w)) ≤ (34) ≤ sup f ∈C f ∞ ≤1 E ε \1 ∼N (0,I D-1 ) +∞ -∞ F 1 (ε) -F 1 √ D √ α t + ε 1 , ε 2 , . . . , ε D φ(ε 1 )|ε 1 |dε 1 ≤ (35) ≤ sup f ∈C f ∞ ≤1 E ε \1 ∼N (0,I D-1 ) +∞ -∞ √ D √ α t φ(ε 1 )|ε 1 |dε 1 = 2D π • 1 √ α t (36)

Thus, we obtain the following bound on the MMD:

MMD(q 0 t (w) q t (w))

EQUATION

This bound goes to zero as α t goes to infinity.

As the output of a softmax network lies in the interval [0, 1], we obtain the following bound on the deviation of the prediction of the ensemble after applying the zero-mean approximation: Figure 8 : These are the learning curves for VGG-like architectures, trained on CIFAR-10 with layerwise parameterization and with different prior distributions. These plots show that all three priors are equivalent in practice: all three models converge to variance networks. The convergence for the Student's prior is slower, because in this case the KL-term is estimated using one-sample MC estimate. This makes the stochastic gradient w.r.t. log α very noisy when α is large.

EQUATION

C ANTI-SYMMETRIC NON-LINEARITIES:

We have considered the following setting. We used a LeNet-5 network on the MNIST dataset with only tanh non-linearities and with no biases. Note that it works well even if the second-to-last layer is a variance layer. It means that the zeromean variance-only encodings are robustly discriminated using a linear model. Figure 9: Average distributions of activations of a variance layer for objects from different classes for four random neurons. Each line corresponds to an average distribution, and the filled areas correspond to the standard deviations of these p.d.f.s. Each neuron essentially has several "energy levels", one for each class / a group of classes. On one "energy level" the samples have roughly the same average magnitude, and samples with different magnitudes can easily be told apart with successive layers of the neural network.

D MNIST VARIANCE ACTIVATIONS:



:

In this section we consider fully-connected layers for simplicity. The same applies to convolutional layers and other linear transformations. In the experiments we consider both fully-connected and convolutional variance layers.

Figures and Tables:

Figure 1: The visualization of objects activation samples from a variance layer with two variance neurons. The network was learned on a toy four-class classification problem. The two plots correspond to two different random initializations. We demonstrate that a variance layer can learn two fundamentally different kinds of representations (a) two neurons repeat each other, the information about each class is encoded in variance of each neuron (b) two neurons encode an orthogonal information, both neurons are needed to identify the class of the object.

Figure2: The KL divergence between the Gaussian dropout posterior and different priors. The KL-divergence between the Gaussian dropout posterior and the Student's t-prior is no longer a function of just α; however, for small enough ν it is indistinguishable from the KL-divergence with the log-uniform prior.

Figure 4: Histograms of test accuracy of LeNet-5 networks on MNIST dataset. The blue histogram shows an accuracy of individual weight samples. The red histogram demonstrates that test time averaging over 200 samples significantly improves accuracy.

Figure 6: Evaluation mean scores for CartPole-v0 and Acrobot-v1 environments obtained by policy gradient after each episode of training for deterministic, parameter noise (Fortunato et al. (2017); Plappert et al. (2017)) and variance network policies (Section 3).

Figure 7: Results on iterative fast sign adversarial attacks for VGG-like architecture on CIFAR-10 dataset. For each iteration we report the successful attack rate. Deep ensemble of variance networks has a lower successful attacks rate.

Variational lower bound (ELBO), its decomposition into the data term and the KL term, and test set accuracy for different parameterizations. The test-time averaging accuracy is roughly the same for all procedures, but a clear phase transition is only achieved in layer-wise and neuron-wise parameterizations. The same result is reproduced on a VGG-like architecture on CIFAR-10; the achievedELBO values are -116.2,  -233.4 and -1453.7  for the layer-wise, neuron-wise and weight-wise parameterizations respectively.

Table content: <table><tr><td>Metric</td><td/><td>zero-mean</td><td colspan="3">Parameterization layer neuron</td><td>weight</td><td>additive</td></tr><tr><td>Evidence Lower Bound Data term Regularizer term</td><td>L(φ) Eq log p(T | X, W ) KL(q p)</td><td>-4.0 -4.0 0.0</td><td colspan="2">-17.4 -31.4 -15.8 -17.0 1.7 14.4</td><td colspan="3">-602.6 -227.9 -33.8 -31.2 568.8 196.7</td></tr><tr><td>Mean propagation acc. (%)</td><td>ŷ = arg maxt p(t | x, EqW )</td><td>11.3</td><td>11.3</td><td>11.3</td><td/><td>96.6</td><td>99.2</td></tr><tr><td colspan="2">Test-time averaging acc. (%) ŷ = arg maxt Eqp(t | x, W )</td><td>99.4</td><td>99.2</td><td>99.2</td><td/><td>99.4</td><td>99.2</td></tr></table>

Test set classification accuracy for differ-

Table content: <table><tr><td colspan="6">ent methods and datasets. "Variance" stands for vari-</td></tr><tr><td colspan="6">ational dropout model in the layer-wise parameteriza-</td></tr><tr><td colspan="6">tion equation 14. "1 samp." corresponds to the accu-</td></tr><tr><td colspan="6">racy of one sample of the weights, "Det." corresponds</td></tr><tr><td colspan="6">to mean propagation, and "20 samp." corresponds to</td></tr><tr><td colspan="6">the MC estimate of the predictive distribution using 20</td></tr><tr><td colspan="2">samples of the weights.</td><td/><td/><td/><td/></tr><tr><td colspan="2">Architecture Dataset</td><td>Network</td><td colspan="3">Accuracy (%) 1 samp. Det. 20 samp.</td></tr><tr><td>LeNet5</td><td>MNIST</td><td>Dropout Variance</td><td>99.1 98.2</td><td>99.4 11.3</td><td>99.4 99.3</td></tr><tr><td>VGG-like</td><td>CIFAR10</td><td>Dropout Variance</td><td>91.0 91.3</td><td>93.1 10.0</td><td>93.4 93.4</td></tr><tr><td>VGG-like</td><td>CIFAR100</td><td>Dropout Variance</td><td>77.5 76.9</td><td>79.8 5.0</td><td>81.7 82.2</td></tr></table>

One-sample (stochastic)  and test-time-averaging (ensemble) accuracy of LeNet-5 tanh networks. "det" stands for conventional deterministic layers and "var" stands for variance layers (layers with zero-mean parameterization).

Table content: <table><tr><td colspan="6">conv1 conv2 dense1 dense2 Stochastic Ensemble</td></tr><tr><td>det</td><td>det</td><td>det</td><td>det</td><td>99.4</td><td>99.4</td></tr><tr><td>det</td><td>var</td><td>det</td><td>det</td><td>98.8</td><td>99.4</td></tr><tr><td>det</td><td>det</td><td>var</td><td>det</td><td>98.8</td><td>99.2</td></tr><tr><td>det</td><td>var</td><td>var</td><td>det</td><td>93.2</td><td>97.5</td></tr></table>

Peer review (iclr2019):
 This paper introduced a new stochastic layer termed variance layer for Bayesian deep learning, where the posterior on weight is a zero-mean symmetric distribution (e.g., Gaussian, Bernoulli, Uniform). The paper showed that under 3 different prior distributions, the Gaussian Dropout layer can converge to variance layer. Experiments verified that it can achieve similar accuracies as conventional binary dropout in image classification and reinforcement learning tasks, is more robust to adversarial attacks, and can be used to sparsify deep models. Pros: (1) Proposed a new type of stochastic layer (variance layer) (2) Competitive performance on a variety of tasks: image classification, robustness to adversarial attacks, reinforcement learning, model compression (3) Theoretically grounded algorithm Cons: (1) My main concern is verification. Most of the comparisons are between variance layer (zero-mean) and conventional binary dropout, while the main argument of the paper is that it’s better to set Gaussian posterior’s mean to zero. So in all the experiments the paper should compare zero-mean variance layer against variational dropout (neuron-wise Eq. 14) and sparse variational dropout (additive Eq. 14), where the mean isn’t zero. (2) The paper applies variance layers to some specific layers. Are there any guidelines to select which layers should be variance layers? Some minor issues: (1) Page 4, equations of Gaussian/Bernoulli/Uniform variance layer, they should be w_ij=…, instead of q(w_ij)= … (2) What’s the prior distribution used in the experiment of Table 1?