ADAPTIVE NETWORK SPARSIFICATION VIA DEPENDENT VARIATIONAL BETA-BERNOULLI DROPOUT

1 INTRODUCTION . One of the main obstacles in applying deep learning to large-scale problems and low-power computing systems is the large number of network parameters , as it can lead to excessive memory and computational overheads . To tackle this problem , researchers have explored network sparsification methods to remove unnecessary connections in a network , which is implementable either by weight pruning ( Han et al. , 2016 ) or sparsity-inducing regularizations Wen et al . ( 2016 ) . Recently , variational Bayesian approaches have shown to be useful for network sparsification , outperforming non-Bayesian counterparts . They take a completely different approach from the conventional methods that either uses thresholding or sparsity-inducing norms on parameters , and uses well-known dropout regularization instead . Specifically , these approaches use variational dropout Kingma et al . ( 2015 ) which adds in multiplicative stochastic noise to each neuron , as a means of obtaining sparse neural networks . Removal of unnecessary neurons could be done by either setting the dropout rate individually for each neuron with unbounded dropout rate Molchanov et al . ( 2017 ) or by pruning based on the signal-to-noise ratio Neklyudov et al . ( 2017 ) . While these variational dropout approaches do yield compact networks , they are suboptimal in that the dropout rate for each neuron is learned completely independently of the given input data and labels . With input-independent dropout regularization , each neuron has no choice but to encode generic information for all possible inputs , since it does not know what input and tasks it will be given at evaluation time , as each neuron will be retained with fixed rate regardless of the input . Obtaining high degree of sparsity in such as setting will be difficult as dropping any of the neurons will result in information loss . For maximal utilization of the network capacity and thus to obtain a more compact model , however , each neuron should be either irreplaceably generic and used by all tasks , or highly specialized for a task such that there exists minimal redundancy among the learned representations . This goal can be achieved by adaptively setting the dropout probability for each input , such that some of the neurons are retained with high probability only for certain types of inputs and tasks . To this end , we propose a novel input-dependent variational dropout regularization for data and task-dependent network sparsification . We first propose beta-Bernoulli dropout that learns to set dropout rate for each individual neuron , by generating the dropout mask from beta-Bernoulli prior , and show how to train it using variational inference . This dropout regularization is a proper way of obtaining a Bayesian neural network and also sparsifies the network , since beta-Bernoulli distribution is a sparsity-inducing prior . Then , we propose dependent beta-Bernoulli dropout , which is an input-dependent version of our variational dropout regularization . Such adaptive regularization has been utilized for general network regularization by a non-Bayesian and non-sparsity-inducing model Ba & Frey ( 2013 ) ; yet , the increased memory and computational overheads that come from learning additional weights for dropout mask generation made it less appealing for generic network regularization . In our case of network sparsification , however , the overheads at training time is more than rewarded by the reduced memory and computational requirements at evaluation time , thanks to the high degree of sparsification obtained in the final output model . We validate our dependent beta-Bernoulli variational dropout regularizer on multiple public datasets for network sparsification performance and prediction error , on which it obtains more compact network with substantially reduced prediction errors , when compared with both the base network and existing network sparsification methods . Further analysis of the learned dropout probability for each unit reveals that our input-adaptive variational dropout approach generates a clearly distinguishable dropout mask for each task , thus enables each task to utilize different sets of neurons for their specialization . Our contribution in this paper is threefold : • We propose beta-Bernoulli dropout , a novel dropout regularizer which learns to generate Bernoulli dropout mask for each neuron with sparsity-inducing prior , that obtains high degree of sparsity without accuracy loss . • We further propose dependent beta-Bernoulli dropout , which yields significantly more compact network than input-independent beta-Bernoulli dropout , and further perform runtime pruning for even less computational cost . • Our beta-Bernoulli dropout regularizations provide novel ways to implement a sparse Bayesian Neural Network , and we provide a variational inference framework for learning it . 2 RELATED WORK . Deep neural networks are known to be prone to overfitting , due to its large number of parameters . Dropout Srivastava et al . ( 2014 ) is an effective regularization that helps prevent overfitting by reducing coadaptations of the units in the networks . During dropout training , the hidden units in the networks are randomly dropped with fixed probability p , which is equivalent to multiplying the Bernoulli noises z ∼ Ber ( 1− p ) to the units . It was later found that multiplying Gaussian noises with the same mean and variance , z ∼ N ( 1 , p1−p ) , works just as well or even better Srivastava et al . ( 2014 ) . Dropout regularizations generally treat the dropout rate p as a hyperparameter to be tuned , but there have been several studies that aim to automatically determine proper dropout rate . Kingma et al . ( 2015 ) propose to determine the variance of the Gaussian dropout by stochastic gradient variational Bayes . Generalized dropout Srinivas & Babu ( 2016 ) places a beta prior on the dropout rate and learn the posterior of the dropout rate through variational Bayes . They showed that by adjusting the hyperparameters of the beta prior , we can obtain several regularization algorithms with different characteristics . Our beta-Bernoulli dropout is similar to one of its special cases , but while they obtain the dropout estimates via point-estimates and compute the gradients of the binary random variables with biased heuristics , we approximate the posterior distribution of the dropout rate with variational distributions and compute asymptotically unbiased gradients for the binary random variables . Ba et al . Ba & Frey ( 2013 ) proposed adaptive dropout ( StandOut ) , where the dropout rates for each individual neurons are determined as function of inputs . This idea is similar in spirit to our dependent beta-Bernoulli dropout , but they use heuristics to model this function , while we use proper variational Bayesian approach to obtain the dropout rates . One drawback of their model is the increased memory and computational cost from additional parameters introduced for dropout mask generation , which is not negligible when the network is large . Our model also requires additional parameters , but with our model the increased cost at training time is rewarded at evaluation time , as it yields a significantly sparse network than the baseline model as an effect of the sparsity-inducing prior . Recently , there has been growing interest in structure learning or sparsification of deep neural networks . Han et al . Han et al . ( 2016 ) proposed a strategy to iteratively prune weak network weights for efficient computations , and Wen et al . Wen et al . ( 2016 ) proposed a group sparsity learning algorithm to drop neurons , filters or even residual blocks in deep neural networks . In Bayesian learning , various sparsity inducing priors have been demonstrated to efficiently prune network weights with little drop in accuracies Molchanov et al . ( 2017 ) ; Louizos et al . ( 2017 ) ; Neklyudov et al . ( 2017 ) ; Louizos et al . ( 2018 ) . In the nonparametric Bayesian perspective , Feng et al . Feng & Darrell ( 2015 ) proposed IBP based algorithm that learns proper number of channels in convolutional neural networks using the asymptotic small-variance limit approximation of the IBP . While our dropout regularizer is motivated by IBP as with this work , our work is differentiated from it by the input-adaptive adjustments of dropout rates that allow each neuron to specialize into features specific for some subsets of tasks . 3 BACKGROUNDS . 3.1 BAYESIAN NEURAL NETWORKS AND STOCHASTIC GRADIENT VARIATIONAL BAYES . Suppose that we are given a neural network f ( x ; W ) parametrized by W , a training set D = { ( xn , yn ) } Nn=1 , and a likelihood p ( y|f ( x ; W ) ) chosen according to the problem of interest ( e.g. , the categorical distribution for a classification task ) . In Bayesian neural networks , the parameter W is treated as a random variable drawn from a pre-specified prior distribution p ( W ) , and the goal of learning is to compute the posterior distribution p ( W|D ) : p ( W|D ) ∝ p ( W ) N∏ n=1 p ( yn|f ( xn ; W ) ) . ( 1 ) When a novel input x∗ is given , the prediction y∗ is obtained as a distribution , by mixing W from p ( W|D ) as follows : p ( y∗|x∗ , D ) = ∫ p ( y∗|f ( x∗ ; W ) ) p ( W|D ) dW . ( 2 ) Unfortunately , p ( W|D ) is in general computationally intractable due to computing p ( D ) , and thus we resort to approximate inference schemes . Specifically , we use variational Bayes ( VB ) , where we posit a variational distribution q ( W ; φ ) of known parametric form and minimize the KL-divergence between it and the true posterior p ( W|D ) , DKL [ q ( W ; φ ) ‖p ( W|D ) ] . It turns out that minimizing DKL [ q ( W ; φ ) ‖p ( W|D ) ] is equivalent to maximizing the evidence lower-bound ( ELBO ) , L ( φ ) = N∑ n=1 Eq [ log p ( yn|f ( xn ; W ) ) ] −DKL [ q ( W ; φ ) ‖p ( W ) ] , ( 3 ) where the first term measures the expected log-likelihood of the dataset w.r.t . q ( W ; φ ) , and the second term regularizes q ( W ; φ ) so that it does not deviate too much from the prior . The parameter φ is learned by gradient descent , but these involves two challenges . First , the expected likelihood is intractable in many cases , and so is its gradient . To resolve this , we assume that q ( W ; φ ) is reparametrizable , so that we can obtain i.i.d . samples from q ( W ; φ ) by computing differentiable transformation of i.i.d . noise ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) as ε ( s ) ∼ r ( ε ) , W ( s ) = T ( ε ( s ) ; φ ) . Then we can obtain a low-variance unbiased estimator of the gradient , namely ∇φEq [ log p ( yn|f ( xn ; W ) ) ] ≈ 1 S S∑ s=1 ∇φ log p ( yn|f ( xn ; T ( ε ( s ) ; φ ) ) ) . ( 4 ) The second challenge is that the number of training instances N may be too large , which makes it impossible to compute the summation of all expected log-likelihood terms . Regarding on this challenge , we employ the stochastic gradient descent technique where we approximate with the summation over a uniformly sampled mini-batch B , N∑ n=1 ∇φEq [ log p ( yn|f ( xn ; W ) ) ] ≈ N |B| ∑ n∈B ∇φEq [ log p ( yn|f ( xn ; W ) ) ] . ( 5 ) Combining the reparametrization and the mini-batch sampling , we obtain an unbiased estimator of∇φL ( φ ) to update φ . This procedure , often referred to as stochastic gradient variational Bayes ( SGVB ) Kingma & Welling ( 2014 ) , is guaranteed to converge to local optima under proper learningrate scheduling .

Summary: This work proposes Variational Beta-Bernoulli Dropout, a Bayesian way to sparsify neural networks by adopting Spike and Slab priors over the parameters of the network. Motivated by the Indian Buffet Process the authors further adopt Beta hyperpriors for the parameters of the Bernoulli distribution and also propose a way to set up the model such that it allows for input specific priors over the Bernoulli distributions. They then provide the necessary details for their variational approximations to the posterior distributions of both such models and experimentally validate their performance on the tasks of MNIST and CIFAR 10/100 classification.