Adaptive network sparsification with dependent variational beta-Bernoulli dropout
Authors: Juho Lee
Saehoon Kim
Jaehong Yoon
Hae Beom Lee
Eunho Yang
Sung Ju Hwang
Venue:  (2020)
Keywords: network sparsification
variational inference
pruning

Abstract: While variational dropout approaches have been shown to be effective for network sparsification, they are still suboptimal in the sense that they set the dropout rate for each neuron without consideration of the input data. With such input independent dropout, each neuron is evolved to be generic across inputs, which makes it difficult to sparsify networks without accuracy loss. To overcome this limitation, we propose adaptive variational dropout whose probabilities are drawn from sparsity inducing beta-Bernoulli prior. It allows each neuron to be evolved either to be generic or specific for certain inputs, or dropped altogether. Such input-adaptive sparsity- inducing dropout allows the resulting network to tolerate larger degree of sparsity without losing its expressive power by removing redundancies among features. We validate our dependent variational beta-Bernoulli dropout on multiple public datasets, on which it obtains significantly more compact networks than baseline methods, with consistent accuracy improvements over the base networks.

Under review as a conference paper at ICLR 2020
ADAPTIVE NETWORK SPARSIFICATION WITH DEPEN- DENT VARIATIONAL RETA-BERNOUILL DROPOUT
Anonymous authors Paper under double-blind review
ABSTRACT
While variational dropout approaches have been shown be effective for network sparsification, they are still suboptimal in the sense that they set the dropout rate for each neuron without consideration of the input data. With such input-independent dropout, each neuron is evolved to be generic across inputs, which makes it difficult to sparsify networks without accuracy loss. To overcome this limitation, we propose adaptive variational dropout whose probabilities are drawn from sparsity-induccin beta- Bernoulli prior. It allows each neuron to be evolved either be generic or specific for certain inputs, or dropped altogether. Such input-adaptive sparsity- inducing dropout allows the resulting network to tolerate larger degree of sparsity without losing its expressive power by removing redundancies among features. We validate our dependent variational beta- eeta-Bernoulli dropout on multiple public datasets, which it obtains significantly more compact networks than baseline methods, with consistent accuracy improvements over the base networks.
INTRODUCTION
One of the main obstacles in applying deep learning large-scale problems and low-power com- puting systems is the large number of network parameters, as it can lead to excessive memory and computational overheads. To tackle this problem, researchers have explored network sparsification methods to remove unnecessary connections in network, which is implementable either by weight pruning (Han et al. 2016) or sparsity sparsity-inducing inducing regularizations (Wen et al. 2016).
Recently, variational Bayesian approaches have shown to be useful for network sparsification, outperforming non-Bayesian counterparts. They take a completely different approach from the conventional methods that either uses thresholding or sparsity-inducing norms on parameters, and uses well-known dropout regularization instead. Specifically, these approaches use variational dropout (Kingma al. 2015) which adds in multiplicative stochastic noise to each neuron, means of obtaining sparse neural networks. Removal of unnecessary neurons could be done by either setting the dropout rate individually for each neuron with unbounded dropout rate (Molchanov et al.) 2017) or by pruning based on the signal-to-noise ratio (Neklyudov et al. (017).
While these variational dropout approaches do yield compact networks, they are suboptimal in that the dropout rate for each neuron is learned completely independently of the given input data and labels. With anpt-independent dropout regularization, each neuron has no choice but to encode generic information for all possible inputs, since it does not know what input and tasks it will be given evaluation time, as each neuron will be retained with fixed rate regardless of the input. Obtaining high degree of sparsity in such as setting will be difficult as dropping any of the neurons will result in information loss. For maximal utilization of the network capacity and thus to obtain a more compact model, however, each neuron should be either irreplaceably generic and used by all tasks, or highly specialized for a task such that there exists minimal redundancy among the learned representations. This goal can be achieved by adaptively setting the dropout probability for each input, such that some of the neurons are retained with high probability only for certain types of inputs and tasks.
To this end, we propose novel input-dependent variational dropout regularization for network sparsification. We first propose individual neuron, by generating the dropout mask from beta- Bernoulli prior, and show how train using variational inference. This dropout regularization a proper way of obtaining a Bayesian neural network and also sparsifies the network, since BertoBll distribution is sparsity-inducing
dropout that learns to set dropout rate for each

---

Under review as a conference paper at ICLR 2020
prior. Then, we propose dependent geta-Bernoulll dropout, which is an input-dependent version of our variational dropout regularization.
Such adaptive regularization has been utilized for general network regularization by non-Bayesian and spwasstt-iiddueeeee model (Ba & Frey 2013); yet, the increased memory and computational overheads that come from learning additional weights for dropout mask generation made it less appealing for generic network regularization. In our case of network sparsification, however, the over- heads training time more than rewarded by the reduced memory and computational requirements at evaluation time, thanks to the high degree of sparsification obtained in the final output model.
We validate our dependent beta-Bernoull variational dropout regularizer on multiple public datasets for network sparsification performance and prediction errоr, on which it obtains more compact network with substantially reduced prediction errors, when compared with both the base network and existing network sparsification methods. Further analysis of the learned dropout probability for each unit reveals that our input-adaptive variational dropout approach generates a clearly distinguishable dropout mask for each task, thus enables each task to utilize different sets of neurons for their specialization.
Our contribution in this paper is threefold:
We propose geta-Bernould dropout, a novel dropout regularizer which learns to generate Bernoulli dropout mask for each neuron with sparsity-inducing prior, that obtains high degree of sparsity without accuracy loss.
We further propose dependent beta- veta-Bernuulli dropout, which yields significantly more compact network than -ipuu-indeeendent beta- Beta-Berouulli dropout, and further perform run- time pruning for even less computational cost.
Our Deta-Bernoulli dropout regularizations provide novel ways implement sparse Bayesian Neural Network, and we provide variational inference framework for learning it.
2 RELATED WORK
Deep neural networks are prone to overfitting, due to its large number of parameters. Dropout (Srivas tava al. 2014) is an effective regularization that helps prevent overfitting by reducing coadaptations of the units in the networks. During dropout training, the hidden units in the networks are ran- domly dropped with fixed probability p. which is equivalent to multiplying the Bernoulli noises Ber(1 p) to the units. It was later found that multiplying Gaussian noises with the same mean and variance, 2 ~ (11, ) works just as well or even better (Srivastava et al. 2014
Dropout generally treat the dropout rate p as a hyperparameter to be tuned, but there have been several studies that aim to automatically determine proper dropout rate. Kingma et al. (2015) propose to determine the variance of the Gaussian dropout by stochastic gradient variational Bayes. Generalized dropout (Srinivas & Babu 2016) places beta prior the dropout rate and learn the posterior of the dropout rate through variational Bayes. They showed that by adjusting the hyperparameters of the beta prior,
obtain several regularization algorithms with different characteristics. Our beta- Bernoulli dropout is similar to one of its special cases, but while they obtain the dropout estimates via point-estimates and compute the gradients of the binary random variables with biased heuristics, we approximate the posterior distribution of the dropout rate using continuous relaxation.
Ba& Frey 2013) proposed adaptive dropout (StandOut), where the dropout rates for each individual neurons are determined as function of inputs. This idea is similar in spirit to our dependent beta- Bernoulli dropout, but they use heuristics to model this function, while we use proper variational Bayesian approach to obtain the dropout rates. One drawback of their model is the increased memory and computational cost from additional parameters introduced for dropout mask generation, which not negligible when the network is large. Our model also requires additional parameters, but with our model the increased cost at training time is rewarded at evaluation time, as it yields a significantly sparse network than the baseline model as an effect of the sparsity-inducis prior.
Recently, there has been growing interest in structure learning or sparsification of deep neural networks. HHn et al.] (2016) proposed strategy to iteratively prune weak network weights for efficient computations, and Wen et al. (2016) proposed a group sparsity learning algorithm to drop neurons, filters or even residual blocks in deep neural networks. In Bayesian learning, various sparsity
2

---

Under review as a conference paper at ICLR 2020
inducing priors have been demonstrated to efficiently prune network weights with little drop in
accuracies (Molchanov et al. 2017; Louizos et al., 2017]] Neklyudov et al. 2017 Louizos et al. 2018 Dai et al. 2018). In the nonparametric Bayesian perspective, Feng al. Feng & Darrell (2015) proposed IBP based algorithm that learns proper numher of channels in convolutional neural networks using the asymptotic small-variance limit approximation of the IBP. While
dropout regularizer
is motivated by IBP as with this work, our work is differentiated from it by the input-adaptive adjustments of dropout rates that allow each neuron to specialize into features specific for some subsets of tasks.
3 BACKGROUNDS
3.1 BAYESIAN NEURAL NETWORKS AND STOCHASTIC GRADIENT VARIATIONAL BAYES
Suppose that we are given a neural network f(x; W) parametrized by W, training set D {(x.. h= and likelihood ffy|f(x: W)) chosen according to the problem of interest. In Bayesian neural networks, the parameter W is treated as random variable drawn from a pre- specified prior distribution p( (W), and the goal is compute the posterior distribution p( (WID)
N
P(WDD) x (WW)
II
(fxx) W)).
(1)
7=1
When novel input x, is given, the prediction y. is obtained as a distribution,
ply. xx. D) = ply, (fxx; W))p(W(D)dW)
(2)
Unfortunately, p( (D) is in general computationally intractable due computing p( p(D). and thus we resort to approximate inference schemes. Specifically, we use variational Bayes (VB), where we posit variational distribution a( (W; d)) of known parametric form and minimize the KL-divergence between it and the true posterior p( PW) It turns out that minimizing DKL[g(W) W: @)plp>(pPpW))) W (D] is equivalent to maximizing the evidence lower-bound (ELBO),
((Φ)
Eyllog (x:: W)) DKL(9(W; ()
(3)
where the first term measures the expected log-likelihood of the dataset w.r.t. o( W: Φ). and the second term regularizes q( (W::)) so that does not deviate too much from the prior. The pa- rameter & is learned by gradient descent, but these involves two challenges. First, the expected many cases, and so its gradient. To resolve this, we assume that q( ((W:q) Φ) is reparametrizable, so that we can obtain i.i.d. samples from q( ((W:d) Φ) by computing Welling, 2014 Rezende et al. 2014) as g(() T(E), (+)) (<((*); Φ). Then we can obtain a low-variance unbiased estimator of the gra- dient of the ELBO. The second challenge is that the number of training instances N may be too large, which makes impossible to compute the summation of all expected log- likelihooo terms. Regarding on this challenge, we employ the stochastic gradient descent technique where we approximate with the summation over uniformly sampled mini-batch B.
likelihood is intractable
differentiable transformation of i.i.d. noise (Kingma
Combining the reparametrization and the mini-batch sampling, we obtain an unbiased estimator of ((L()) to update Φb This procedure, often referred to as stochastic gradient variational Bayes (SGVB) (Kingma & Welling 2014), is guaranteed to converge to local optima under proper learning- rate scheduling.
3.2 LATENT FEATURE MODELS AND INDIAN BUFFET PROCESSES
In latent feature model, data are assumed be generated as combinations of latent features:
d = f(Wz.)) = 2(2
K
(4)
1=1
where =n,A means that d, possesses the k--th feature WA. and is arbitrary function.
3

---

Under review as a conference paper at ICLR 2020
The Indian Buffet Process (IBP) (Griffiths & Ghahramani 2005) is a generative process of binary matrices with infinite number of columns. Given N data points, IBP generates a binary matrix Z {0, (0,1).M 1| whose n-th encodes the feature indicator z The IBP is suitable to prior process in latent feature models, since it generates possibly infinite number of columns and adaptively adjust the number of features on given dataset. Hence, with IBP prior need not specify the number of features in advance.
One interesting observation that while it is a marginal of the -eta-Bernuulli processes Thibaux & Jordan 2007), the IBP may also be understood as limit of the finite-dimensional process. More specifically, the IBP with parameter > can be obtained
Th ~ beta(a//Ha K, 1), Zn.k Ber(m:), K xo.
(5)
This Beta-Bernouuli process naturally induces sparsity in the latent feature allocation matrix Z. As K- oo, the expected number of nonzero entries in Z converges No: (Griffiths & Ghahramani 2005).
where a is a hyperparameter to control the overall sparsity level of Z.
In this paper, we relate the latent feature models ically, the binary random variables An, correspond to the dropout indicator, and the features w correspond to the inputs units in neural networks. From this connection, can think of a hierar- chical Bayesian model with IBP or mnite-imensional Beta-Bernoulli priors. We expect that due to the property of the IBP, the resulting neural network would also be sparse
neural networks with dropout mask. Specif-
3.3 DEPENDENT INDIAN BUFFET PROCESSES
One important assumption in the IBP is that features are exchangeable the distribution is invariant to the permutation of feature assignments. This restricts flexibility when we want to model the dependency of feature assignments to the input covariates x, such as times or spatial locations. To this end. WWiliiason al. (2100) proposed dependent Indian Buffet processes (dIBP), which triggered line of follow-up work (Zhou et al., 2011 Ren et al. 2011). These models can be summarized as
following generative process:
p(T) 2 π,, x Ber(g(T,, T x)),
where g(:;) -) is an arbitrary function that maps π and x to a probability. In our latent feature interpretation for neural network layers above, the input covariates corresponds to the input activations in the previous layer. In other words, we build a data-dependent dropout model where the dropout rates depend on the inputs.
4 MAIN CONTRIBUTION
4.1 VARIATIONAL RETA-BERNOULLE DROPOUT
Inspired by the latent-feature model interpretation of layers in neural networks, we propose a neural network layer overlaid with binary random masks sampled from the finite-dimensional beta- Bernoulli prior. Specifically. let W be parameter of a neural network layer, and l,, z,, (0, 1)x be binary mask vector to applied for the x,, The dimension W needs not be equal to K. Instead, we may enforce arbitrary group sparsity by sharing the binary masks among multiple elements of W. For instance, let W e R xx be a parameter tensor in a convolutional neural network with K channels.
Bayesian
To enforce a shannel-wise sparsity, we introduce Z,, E {0, 1] to produce masked parameter W,, for the n-th observation is given as [2n,&WK,e.. WK.em (ks, m) = (1, 1,)...... (K, L, M)}. With slight abuse of notation, we denote this binary mask multiplication as W,, process of our Bayesian neural network is then described as
W. The generative
K
W ~ NO AI), π II
(aaa(maa//..1. 2n π
(er(zn,e) Te),
W. = nn @ W.
(7)
Rm
4

---

Under review as a conference paper at ICLR 2020
To approximate the posterior (, Z, π (D),
introduce variational distributions as
K II k=1
N
K
a W 2, π X = kw
((TT))
II (Z 70
(8)
k=1
For W, we conduct computationally efficient point-estimate to get the single value W. For π following Nalisnick & Smyth (017). we use the Kumaraswamy distribution (Kumaraswamy 1980)
(TT ag, b() axbb= "apr--11111 (1 Tx )
(9)
since it closely resembles the beta distribution and easily reparametrizable
Th (u; ajs, bx) = (1 3 " u ~~ unif([o, I)).
(10)
We further assume that 2n (T)) = p(2n. 2m. (T)) = (er(TT). 24 is reparametrized as in Maddison al. 2017); Jang al. (017); Gal al. (2017),
/ )
TTk
2k:
sgm
log
+ log
(11)
where 7 is a temperature of continuous relaxation, (n0 1)). and sgmm(z I+e The KL-divergence between the prior and the variational distribution is then obtained as follows (Nalisnick)
& (onyth/2017)
ak af K (- (
ag a
bk
(Kil pZ, rT]
(bx)
log
(12)
ak
by
bk
where Euler-Mascheroni constant and (-( is the digamma function. Note that the infinite series in the Kl--divergence vanishes because of the choice p(TT)) = beta(TT;; K, 1).
We apply the SGVB framework described in Section 3. to optimize the variational parameters {ax, by H After the training, the prediction for novel input x. is given as
xx, x. D. W) ~ E) fxx.; ; z. X W))l
(13)
and we found that the following näive approximation works well in practice,
ply./x., D, W) ply. |(y. E,Iz. & W)),
(14)
where
b1T + ag (bb)))
[xg = (T(SL)) [G(ME)(TTR]
(15)
T(1 + ax bk)
4.2 VARIATIONAL. DEPENDENT BETA- DROPOUT
We define a Bayesian neural network model with input-dependent beta-Bernoulli prior as follows:
K
K
W ~ AD),
beta // K, 1), Zn k=1
II
II (i
(16)
Here, x, is the input to the dropout layer In principle, we may introduce another fully connected layer as (1) (In.1)...., PP (æn,K)) sgm (xx, + c). with additional parameters V k and с RR but this undesirable for the network sparsification. Instead, we propose simple yet effective way to generate input-dependent probability with minimal parameters. Motivated by the batch normalization (loffe & Szegedy 2015). we construct each T& (2n,k) independently as follows:
Ppk (N(xk) = clamp
7k
Fn.k
+ Bk,
(17)
Ok
1 Herre we wrote as we only have one binary mask layer, but in practice every layer a neural network cquipped with its own binary mask. We abbreviate those multiple binary masks with Z for simplicity. Forr convolutional layers, we apply the global average pooling to tensors to get vectorized inputs.
5

---

Under review as a conference paper at ICLR 2020
where b4 and o% are the estimates of k-th components of mean and standard deviation of inputs, and " and 85 are scaling and shifting parameters and e > is some small tolerance value, and max(e,z)). The intuition behind this construction is follows. The inputs after the standardization would approximately be distributed as (0, 1). If we pass them through min(1 c, max(e, 2)), most of insignificant dimensions would have probability near zero. However, some inputs may be important regardless of the significance of activation, for which we expect the shifting 3i to compensate. With f we control the overall sparsity, but we want them be small unless required to get sparse outcomes. We enforce this by placing prior 3 ~ (0, pI).
clamp(2z, e) min(1
The goal of variational inference is hence to learn the posterior distribution p( (W Z, 3. and we upproximate this with variational distribution of the form
N II
q W. Z. 7| w (W) II (TT))((()))
II (2n, (Tk ==1 k=1
(18)
A=1
where (TTx) are the same as in beta-Bernoulli dropout, q(ak) = (Vxx; [Bx::m=>>> me, m.), and ( (
(TT)
p( TTy x,) [ The KL-divergence is computed as DKL (g nT X m) + DkLlu())lp((3), where both terms can be computed analytically.
The prediction for the novel input is similarity done in the beta-Bernoulli dropout, with the naïve approximation for the expectation:
p(y.|x., D, W) xo p(y.|f(xx;;))) E,(2.] @ W)),
(19)
where
Ey(Z=,,]] -EgTTR] clamp
Ykk
Fn.k
Mkk
+ Tik:
(20)
o
Two stage pruning scheme Since TT& 2 TT (1), for all Tn,k. we expect the resulting network to be sparser than the network pruned only with the beta- Beranull dropout (only with T&). To achieve this, propose two-stage pruning scheme, where first prune the network with BettoBernoulli dropout, and prune the network again with T (Fn,k) while holding the variables æ fixed. By fixing π the resulting network is guaranteed to be sparser than the network before the second pruning.
5 EXPERIMENTS
We now compare our beta-Bernould dropout (BB) and input-dependent beta- dropout (DBB) to other structure learning/pruning algorithms on several neural networks using benchmark datasets.
Experiment Settings We follow common experimental setting used by existing work to
evaluate pruning performance, and use LeNet 500-300, LeNet 5-Caffe and VGG-like (Zagoruyko
2015) MNIST (LeCun al. 1998), CIFAR- 10, and CIFAR- datasets (Krizhevsky & Hinton
2009). For baselines, we use the following recent Bayesian pruning methods: 1) SBP: Structured
Bayesian Pruning (Neklyudov et al. 2017) 2) VIB: Variational Information Bottleneck (Dai et al.
2018). 3) Lo: Lo regularization (Louizos et al. 2018). 4) GD: Generalized Dropout (Srinivas & Babu 2016). 5) CD: Concrete Dropout (Gal al. 2017 We faithfully tune all hyperparameters of baselines on validation set to find reasonable solution that is well balanced between accuracy and sparsification, while fixing batch size (100) and the number of maximum epochs (200) match our experiment setting.
Implementation Details We pretrain all networks using the standard training procedure before fine-tuning for network sparsification Neklyudov et al. (2017); Dai et al. (2018) 1) Louizos et al. (2018). While pruning, we set the learning rate for the weights W to be 0.1 times smaller than those for the variational parameters in Neklyudov et al. 2017). We used Adam (Kingma & Ba 2015) for all
In principle, we may introduce an inference network q(z) y) and minimizes the KL KL-divergence between (Z y. and p(z) n. x). but this results in discrepancy between training and testing for sampling z, and also make optimization cumhersome. Hence, we chose to simply set them equal. Please refer to so/Sonn et al. 2015) for discussion about this.

---

Under review as a conference paper at ICLR 2020
Table 1: Results for LeNet-500-300 and LeNetS-Caffe MNIST.
LeNer 500-300
CaNest-Cffee
Emur (S)
Neuroms
<FLOPs
Memory (8)
Eree (%)
Neurons/Filur
xFLOPs
Memory (%)
Original
1.63
784,300-300
1.0
10010
0.71
20-30-300-500
1.0
100.0
CD
1.542:04
784-500-500
1.0
100.0
0.67
20-50-300-500
1.0
100.0
GD
1.450002
518-187-135
0.67:0002
12-37-496-300
2211
41.29
SRP
1.58::066
0.70:0002
5-15-134-44
11.41
6.47
1.o
06440000
148-116-127
16.16
6.21
0.66:20003
11-22-79-157
4.08
10.61
VIH
1.000:000
138-98-26
33.63
2,98
0.66:200004
7:25-297-38
5,0
8.75
BB
1.39:000
137590337
0:06
7-17-132:30
7,3
1.26:10.04
288 114-65
13.38
7.48
0.59:10.04
9-24-174-58
4.55
9.5
DBB
1.59+-0.00
93-35-25
47.07
2,22
0.68-+00001
7-17-64-38
7:27
7,30
1.42::0066
104:31-29
34.98
0.59:10.02
9:25-61-32
4.55
8.28
240
115
1411
1827
3pentip PLA0S
Figure 1: Top: Classification errors sparsificaton performance of various pruning methods for LeNet-500- 300 and LeNet5-Caffe on the MNIST datasct. Bottom: Spror-p tradcoff plots for LeNet 500-300 (left) and Lefeff-Caffe (right).
methods. For DBB, mentioned in Section 4.2
first prune networks with BB, and then prune again with DBB whiling holding the variational parameters for qTT) fixed.
We report all hyperparameters of BB and DBB for reproducing our results. We set a K 10-4 for all layers of BB and DBB. In principle, we may fix K to be large number and tune a. However, in the network sparsification tasks, K is given as the neurons/filters be pruned. Hence, set the ratio a/K K to be 10 for all layers in all networks. For the inference, we prune neurons/filters whose expected binary mask E, [a.] are smaller than a threshold, and excluded them from the computation, in order to show the performance of the pruned networks. We fixed the threshold to 10-3 for all settings 5] For the input-dependent dropout, since the number of pruned neurons/filters differ according the inputs, we report them as the running average over the test data. We fix the temperature parameter of concrete distribution т 10 and the prior variance of 3, v5 for all experiments. We ran all experiments three times to report mean and standard deviations for the error.
choose
To control the tradeoff between classification error and pruned network size, we run each algorithm with various tradeoff parameters. For VIB and T.g, we controlled the tradeoff parameters originally introduced in the papers. For variational inference based algorithms including SBP and BB, scaled the KL terms in ELBO with tradeoff parameter 7 > 1. Note that when 7 > 1, the modified ELBO is a still lower bound on the marginal likelihood. For DBB we fixed parameter settings but retrain the model with different runs of BBs, that are trained with different tradeoff parameters yy For more detailed settings of tradeoff control, please refer the appendix.
5.1 MNIST EXPERIMENTS
We use LeNet 500-300 and LeNet 5-Caffe networks on MNIST for comparison. Following the conventions, we apply dropout to the inputs to the fully connected layers and right after the convolution for the convolutional layers. We report the trade-off between the error and speedup in terms FLOPs Fig. I and show one representative result per each algorithm in Table Too compare the speedups and memory savings with respect to a particular error level. For DBB, we included additional overhead coming from deput-depnndent computation into FLOPs and memory computation. For LeNet 5 5-Caffe, following Dai et al. (2018), Neklyudov et al. (2017), we used larger tradeoff parameters for the first two convolutional layers please refer to the appendix for the detail.
We tried different threshold values ranging from 10 -2 14 but the difference insignificant.
7

---

Under review as a conference paper at ICLR 2020
CIFAR-1O
CIFAR 100
Eroor (8)
xFLOPs
Memory (%)
Erurr (%)
AFLON
Memory (8)
Original
7.13
1.0
190.0
33.10
1.0
100.0
CD
6.91:10.08
1.0
100.0
30.31:10.13
1.0
100.0
GD
6.87-00.2
20.39
3129-10.34
4.24
31.4740.222
Lg
7.051:0.29
3:46
10.54
32.09:10.39
2.77
15.64
VIB
7:01 0.01
3.31
19.26
3009
233
16.38
7.18:20:12 12
29.2740.49
12.59
0.
3,00
11.13
28:97:1::344
2446
15.49
DRR
7.0422000
4.60
8.86
70 08-00-22222
2.91
13.30
6.56:20.04
3:23
11.64
28.46:::.255
2.60
14,46
SE27
(1241
Speidap
Figure 2: Top: Classification errors and sparsification performances of various pruning methods on CIFAR-10 and CIFAR-100 datasets. Bottom: Error-Speedup tradeoff plots (left, middle) and an empirical analysis of learned filters from DBB (right).
With both neworks, BB and DBB either achieve significantly smaller error than the baseline methods, or significant speedup and memory saving similar errоr rates. DBB, with its input-adaptive pruning, obtains larger speedup and memory saving compared to BB, which is better shown in the error- speedup tradeoff plot. For LeNet 500-300, BB acheives the mean error 1.40% with 21.79 xFLOPs, while DBB acheives similar the mean error with 38. 75% xFLOPs. Table 2 in the appendix clearly show this advantage.
On LeNet-500-300, DBB prunes large amount of neurons in the input layer, because the inputs to this network are simply vectorized pixel values, so it can prune the inputs according to the digit classes, which is shown in Fig. I Also, we observe that the dropout masks generated by DBB tend to be generic at lower network layers to extract common features, but become class-specific at higher layers specialize features for class discriminativity (see Fig. 3 in the appendix). We observed similar behavior of DBB in the experiments with VGG on both CIFARIO and CIFAR100 datasets.
5.2 CIFAR-10 AND CIFAR-100 EXPERIMENTS
We further compare the pruning algorithms on VGG-like network on CIFAR-10 and CIFAR- 100 datasets. Fig. summarizes the performance of each algorithm on particular setting, where BB and DBB achieve both impressive speedups and memory savings with significantly improved accracy. When compared with the baseline sparsification methods, they either achieve better error at similar sparsification rates, or achieve better speedup and memory saving at simlar error rates. DBB achieves larger speedup than BB similar error rates, and all other baselines. For instance, in CIFAR-100 BB achieves the error of 28.86% with 2.60 xFLOPs while DBB records the eπтоr 28.84% with 2.98 xFLOPs. As clearly depicted in the error-speedup tradeoff plot 2] DBB provides the best tradeoff among the algorithms tested. Please refer Table in the appendix for more detailed experimental results.
Further analysis of the filters retained by DBB in Fig. shows that DBB either retains most filters (layer 3) or perform generic pruning (layer 8) at lower layers, while performing diversified pruning at higher layers (layer 15). Further, layer 15, instances from the same class (bus) retained similar filters, while instances from different classes (bus vs. apple) retained different filters.
6 CONCLUSION
We have proposed novel aita-Bermoull dropout for network regularization and sparsification. where we learn dropout probabilities for each neuron either in an input-indeg inpee-indeenndent or input-dependent manner. Our beta-Bernoue dropout leamns the distribution of sparse Bernoulli dropout mask for
8

---

Under review as a conference paper at ICLR 2020
each neuron in variational inference framework, in contrast existing work that learned the distribution of Gaussian multiplicative noise or weights, and obtains significantly more compact network compared to those competing approaches. Further, dependent that adattadday decides which neuron to drop further improves on the input-independent beta- Bernoulli dropout, both in terms of size of the final network obtained and run-time computations. Future work may include network structure learning (c.g. a tree structure) using generalized version of the method where dropout mask is applied block of weights rather than to each hidden unit.
dropout
REFERENCES
Ba and B. Frey. Adaptive dropout for training deep neural networks. In Advances in Neural Information Processing Systems 26, 2013.
Bin Dai, Chen Zhou, and David Wipf. Compressing neural networks using the variational information bottleneck. arX.v: 10399, 2018.
Feng and Darrell. Learning the structure of deep convolutional networks. IEEE International Conference Computer Vision, 2015.
Y. Gal, Hron, and Kendall. Concrete dropout. Advances in Neural Information Processing Systems, 2017.
L. Griffiths and Z. Ghahramani. Infinite latent feature models and the Indian buffet process. In NIPS, 2005.
Han, H. Mao, and W. J. Dally. Deep compression: trained quantization and Huffman coding. In Proceedings of the International Conference on Learning Representations, 2016.
compressing deep neural networks with pruning,
S. loffe and C. Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, 2015.
E. Jang, S. Gu, and B. Poole. Categorical reparametrization with Gumbel-softmax. Proceedings of the International Conference on Learning Representations, 2017.
D. P. Kingma and J. L. Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference Learning Representations, 2015.
D. P. Kingma and M. Welling. Auto-encoding variational Bayes. In Proceedings of the International Conference Learning Representations, 2014.
D. Kingma, T. Salimans, and M. Welling. Variational dropout and the local reparametrization trick.
In Advances in Neural Information Processing Systems 28, 2015.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Computer Science Department, University of Toronto, 2009.
Pounambalam Kumaraswamy, generalized probability density function for double-bounder random processes. Journal of Hydrology, 1980.
Y. LeCun, L. Bottou, Y. Bengio, and Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, 86( 86(11):278-22244 1998.
C. Louizos, K. Ullrich, and M. Welling. Bayesian compression for deep learning. Advances in Neural Information Processing Systems, 2017.
C. Louizos, M. Welling, and D. P. Kingma. Learning sparse neural networks through Lo regularization. International Conference on Learning Representations, 2018.
C. J. Maddison, A. Mnih, and Y. W. Teh. The concrete distribution: a continuous relaxation of discrete random variables. Proceedings of the International Conference on Learning Representations, 2017.

---

Under review as a conference paper at ICLR 2020
D. Molchanov, A. Ashukha, and D. Vetrov. Variational dropout sparsifies deep neural networks, In Proceedings of the 34th International Conference on Machine Learning, 2017.
E. Nalisnick and P. Smyth. Stick-breaking variational autoencoders. In Proceedings of the Interna- tional Conference on Learning Representations, 2017.
K. Neklyudov, D. Molchanov, A. Ashukha, and D. Vetrov. Structured Bayesian pruning via log- normal multiplicative noise. Advances in Neural Information Processing Systems, 2017.
L. Ren, Y. Wang, D. B. Dunson, and L. Carin. The kernel beta process. In Advances in Neural Information Processing Systems 24, 2011.
D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference Machine Learning, 2014.
K. Sohn, H. Lee, and X. Yan. Learning structured ouput representation using deep conditional generative models. Advances Neural Information Processing Systems 28, 2015.
Srinivas and R. Babu. Generalized dropout. arXiv preprint arXiv:1611.06791, 2016.
N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1): 1929-1958, 2014.
R. Thibaux and M. I. Jordan. Hierarchical beta processess and the Indian buffet processes. In Proceedings of the 1thh International Conference on Artificial Intelligence and Statistics, 2007.
W. Wen, Wu, Wang, Y. Chen, and H. Li. Learning structured sparsity deep neural networks. In Advances in Neural Information Processing Systems 29, 2016.
S. Williamson, P. Orbanz, and Z. Ghahramani. Dependent indian buffet processes. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, 2010.
S. Zagoruyko. 92.45 on CIFAR-10 in Torch. 2015.
M. Zhou, H. Yang, G. Sapiro, and D. B. Dunson. Dependent hierarchical beta process for image interpolation and denoising. Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, 2011.
10

---

Under review as a conference paper at ICLR 2020
conv20
conv50
dense800
dense500
class
class
class
class
class
Figure 3: Correlation coefficients of class averages of (x(x) for the four layers in LeNet5-Caffe, where the darker shows the more correlation between the class average values of (xx). The correlation between classes diminishes as layer goes up, implying that the input filters are being specialized to the classes.
A APPENDIX
B MORE DETAILS ON THE EXPERIMENTS
We first describe the tradeoff parameter settings we used in the experiments.
For the variational inference based methods (BB, SBP), we scaled the KL-term by y 2 1. We tested with y {1.0, 2.0, 4.0, 6.0, 8.0}.
For GD, we tested with hyperparameter E (10-3, 10-1 10-5 10-
For DBB, we didn't i do any tradeoff control. Instead, we started from different results of BB produced with various tradeoff parameters.
For VIB, tested with tradeoff parameter {10-4, 1000-,105,,5 10 10-6, 5. 10-7 for LeNet-500-300, 7 E {10-6, 6 50- 10-7,5- 10- 10- for LeNet5-Caffe and VGG-like.
0.01/,000000000,,, /60000, 0.005 /60000] For Lo, we tested with [0.1/60000, 0.05/60000, 0.02/60000, 0.01
for LeNet-500-300 and LeNet5-Caffe À (10-7,5 1,-8,.2- 10 10 5 10 9) for VGG-like.
For LeNet5-Caft we used larger tradeoff parameters for the first two conv layers, because the penalty for them are relatively underestimated due to the small number of filters (20, 50) compared to those fully-connected layers (800-500).
For BB, SBP, and GD, multiplied the kl scaling factor y by 20 and 8 for the first two convolutional layers.
For VIB, following the paper, we multiplied the sizes feature maps, 24 x 24 and to the tradeoff parameters of the first and second convolutional layers.
For T-g reg, we used the setting specified in the paper (Zog-sep).
C ADDITIONAL RESULTS
We present the errors, speedup in FLOPs, and memory savings for all trade-off settings of every algorithm. The results of LeNet-500-300 and LeNet-5-Caffe are displayed in Table 2, and the results for VGG-like on CIFAR-10 10 and CIFAR- 100 are displayed in Table 3
11

---

Under review as a conference paper at ICLR 2020
Table Comparison of pruning methods on LeNet-500-300 and LeNet5-Caffe with MNIST. Error
and Memory are in %.
LeNet 500-300
LeNet5 -Cffee
Error
Speedup
Memory
Error
Speedup
Memory
CD
1.04
1.0
100.0
0.67++0.04
1.0
100.0
1.48
1.00
99.62
0.70
1.93
45.14
1.47=0.02
4.23
23.64
0.73++0.03
2.01
45.84
GD
1.53++0.04
4.44
22.55
0.68+00.02
1.96
45.56
1.45+0.02
4.31
23.22
0.70
2.03
45.36
1.93#0.09
22.99
4.34
0.83+0.07
6.77
12.3
2.06+0.113
27.68
3,61
0.84
8.06
11.83
SSL
2.30+0.09
39.13
2.55
0.97+00006
12.74
10.97
2.63+0.12
54.83
1.82
1.03++0.05
20.26
10.20
3.52+0.12
75.69
1.32
1.44
23.17
10.09
1,45+0.02
8.4
11.89
0.68
2.92
17.95
1.51
18.39
5.43
0.71
4.78
14.75
SVD
1.50:L0.04
14.73
6.78
0.72+00.11
5.17
14.04
1.51
18.39
5.43
0.74+0000
4.92
14.05
2.75+0.14
21.44
4.66
1.47#0.11
3,62
16.38
1,59
18.92
5.30
0.70
11.41
6.47
1.66:0.04
22.96
4.37
0.77+0001
11.95
6.20
SBP
1.58
30.43
3.30
0.81+0.04
13.73
5.95
1.6440.04
38.68
2.60
0.88+0.06
15.30
5.68
1.68
45.78
2.19
0.96+0.04
20.13
5.18
1.64:0.03
16.16
6,21
0.70+10.06
5.23
9.77
1.46=0.02
12.82
7.82
0.66+00.03
4.08
10.61
Lo
1.33+
9.74
10.29
0.62+-0.04
3.42
11.57
0.28
8.27
12.10
0.55+0005
3.15
12.64
1,27
7.57
13.22
0.52+0.01
2.86
14.08
1.60
33.63
2.98
0.67+00.03
8.46
8.17
1.71
21.54
4.65
0.69
7.11
8.44
VIB
1.665
20.10
4.99
0.66
5.80
8.75
1.80+0.06
18.44
5.43
0.69
5.25
9.15
1.68
17.10
5.86
0.7310004
4.67
9.68
1.26400.04
13.38
7.48
0.59
4.55
9.50
1,28
16.23
6.18
0.59
5.58
8.94
BB
1,40
21.79
4.60
0.66
7.30
7.85
1,49
28.28
3.55
0.78
10.45
6.60
1.59+
33.81
2.97
0.86+0.05
13.77
5.87
1.41
34.98
3.00
0.59
4.55
8.28
1.40:0.06
38.75
2.71
0.6310.02
5.58
7.83
DBB
1.52+
43.27
2.42
0.68+0.01
7.27
7.10
1.59++0.03
47.07
2.22
0.78+-0.06
10.32
6.32
1.69
52.09
2.00
0.8610005
13.53
5.78
12

---

Under review as a conference paper at ICLR 2020
Table 3: Comparison of pruning methods on VGG-like with CIFARIO and CIFAR100. Error and
Memory are %.
VGG-CIFARIO
VGG-CIFARI00
Error
Speedup
Memory
Error
Speedup
Memory
CD
6.94++0.08
1.0
100.0
30.031 +0013
1.0
100.0
6.91
2.25
20.58
31.17:0.442
2.03
29.01
6.69
2.23
20.53
30.64+t0.27
2.06
28.71
GD
6.841:008
2.28
20.40
30.84:10.04
2.02
29.08
6.80:10.10
2.24
20.40
30.86:10.68
1.99
29.55
7.12::0166
1.41
20.30
30,45::0.19
1.25
36.14
7.32++0.07
1.43
19.04
30.64++0.06
1.25
34.44
SSL
8.27+0.11
1.67
14.78
12.75++0.16
1.28
28.24
9.56
2.09
12.01
36.2710.22
1.48
23.21
E.00
1.49
21.71
31.22+00.06
1.25
36.80
7.38
1.40
22.47
31.31
1.25
36.90
SVD
7.008
1.41
21.18
31.26:10.06
1.25
36.77
0.71
1.41
18.48
34.26:t0.19
1.25
35.76
4..04
3.69
9.76
30.29
2.54
16.30
7.3
3,90
8.71
31,87
2.49
14.31
SBP
7.5
4.24
8.04
31.52:10.99
2.56
13.74
7.00
4.45
7.72
31.85
2.61
13.55
0.
4.78
7.27
31.47:0.22
2.63
13.41
7.05
3.46
10.54
31.73+:0.46
2.82
15.10
0.4
3,42
10.65
31.59+0.66
2.79
15.36
Lo
0.23
3.38
10.81
31.94
2.79
15.45
7.25++005
3.33
10.89
31.87++0.00
2.78
15.58
7.11+0.21
3.33
10.93
32.09+0.339
2.77
15.64
+.01
3.31
10.26
30.17+0.11
2.37
15.81
6.95+10.11
3.22
10.49
30.10:10.21
2.36
16.06
VIB
6.83
3.10
10.93
30.43:10.18
2.31
16.38
6.92-10.14
3.11
10.87
30.09.10.13
2.32
16.40
6.78:10.08
3.01
11.13
30.0940.00
2.32
16.38
6.45-H0.19
3.00
11.13
28.97+00.24
2.46
15.49
6.57
3.30
10.12
28.5340.04
2.53
14.63
BB
6.88
4.00
8,45
28.86
2.60
13,80
7.18
4.35
7.88
29.2710.43
2.79
12.59
7.40
5.29
6.83
29.36+0.223
2.84
12.19
6.56:10.05
3.23
11.64
29.22-10.40
2.63
15.26
6.51+#0.20
3.52
10.70
29.07
2.68
14.89
DBB
7.02+:022
4.24
9.47
28.46:10.25
2.73
14.26
7.04
4.60
8.86
29.08+0.27
2.91
13.30
7.52+00.0
5.57
7.91
28. 0..3
2.96
13.29
13


Reviews:
[{"review_id": "rygd9WuvqH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes Variational Beta-Bernoulli Dropout to sparsify the parameters of the network. The authors also proposed an input-dependent dropout using input-dependent Beta-Bernoulli priors. The paper presents necessary theoretical details as well as the experimental comparison with the other dropout methods on MNIST and CIFAR 10/100 datasets.\n\nThe paper is well-written overall and easy to follow. The paper provides a thorough background on previous work; however the motivation for having an input-dependent dropout method is relatively weak. \n\nI believe the paper presents main idea and necessary details thoroughly, though I have some confusions:\n\n- In sections 4.1 and 4.2 (more specifically formula 7, 8 and 18), is there a global z (i.e. only one z), or there is a separate z for each layer? I implicitly inferred it's per layer but it's not clear from the paper.\n\n- The paper explains well how the two proposed dropouts can be learned during training, but it's not clear about the inference, specially in variational dependent Beta-Bernoulli dropout. In section 5, experimental details, it is said: \"In the testing phase, we prune the neurons/filters whose expected dropout mask probability E_q[pi_k] are smaller than a fixed threshold 10^-3.\" This is not clear to me what it means, and also the related footnote which says different thresholds were tried but the difference is insignificant! More elaboration is helpful.\n\n- In the experimental results, the authors reported median and standard-deviation. Is there any reason the authors didn't report mean and standard-deviation instead? Also, it is not clear if xFLOP and Memory report the best, mean or median of 3 runs.\n\n- In Table 1, what does column Neurons represent? Is it the model parameters at inference time? and if yes, is it the smallest number of parameters in 3 runs? In Table 2, how large the network is in terms of model parameters?\n\n- In the experimental results, some models are missing that are mentioned in the paper as previous work which seem to be good to use for comparison, such as Sparse Variational Dropout (Molchanov 2017) and Structured Sparsity Learning (Wen et al 2016).\n\n- I encourage the authors to expand more on analyzing the experimental results! In Table 1 and Table 2, there are 2 lines for BB and DBB, which is helpful to explain what each line represents. Looking at the results of all tables, and comparing different dropouts w.r.t :1) Error, 2) xFLOPs and 3) Memory, I cannot draw a clear conclusion from the results. For example, in Table 2, CIFAR-10, comparing first line of BB to SBP and second line to VIB, the difference is not significant. More analysis about the results can be helpful!", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A"}, "extra_text": {}}, {"review_id": "SkgvcIJm5H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a model with beta-bernoulli dropout of neurons for network pruning. The model assumes the probability to dropout a neuron is a function pi(phi, x), which depends both the beta variable phi and the input x. The model is trained with stochastic gradient variational Bayes with continous relaxation. The model and algorithm sound, and the intuition of determining the dropout probability based on the importance of each dimension makes sense. \n\nOne weakness of this work is the lack of large-scale experiments, for example, pruning a MobileNet on ImageNet. This work also seems incremental due to its resemblance with CD.\n\nFigure 1: bottom-right figure (input-regions pruned by DBB) is missing?\n\n====\nUpdate:\nThe authors do address my concern #2. After reading other reviews and reading the revised paper I do think this approach of this paper is novel and can potentially lead to a gain. However I still don't think the experiments are convincing enough. It needs to be tested on a larger variety of models (ResNet or MobileNet) / datasets (ImageNet, etc.) / tasks (vision, nlp, etc.) to prove its significance. Therefore I won't change my score.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJeGyk0TFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new way of training variational dropout which is adaptive to input samples due to the proposed sparsity-inducing beta-Bernoulli prior. The authors provide a good motivation for their model, introduce beta-Bernoulli and dependant beta-Bernoulli prior and propose the method in the variational inference framework. \n\nConcerns:\n1) The main concern relates to the significance of benefits from the input dependency property. From the Tables 1, 2 we can see that BB is comparable with DBB and the latter is not uniformly better than the former in terms of error, xFLOPs and memory. This similarity in the performance is more significant for LeNet5-Caffe network, for CIFAR-10 and CIFAR-100 datasets. Is the overhead of DBB worth the benefits it gives? \n2) The second question is about the memory consumption of DBB. The authors use two stage pruning scheme for DBB: at first, they prune DBB using the beta-Bernoulli dropout, then prune the network for each input individually. It means that DBB should keep all weights after the first stage, in other words the memory consumption of DBB is the same as BB. Some clarifications about this concern are necessary. \n\nOverall, the paper proposes interesting and well-motivated method for training sparse networks. Although, there are concerns about the DBB extension I would recommend considering this paper for acceptance. \n\n---------------------------------------\nUpdate after author rebuttal\n\nThank you for your thoughtful response. You addressed my second concern about memory consumption of DBB. Indeed, the run-time memory mostly consists of activation maps, therefore DBB can benefit from input-dependent sparsity. Considering the first concern I still think that the advantage of DBB is not clear and I agree with AnonReviewer3 who said that except LeNet-500-300 other results are mixed. \n\nHowever, overall I do think that the proposed method is novel, well theoretically grounded and is proved that it works comparably if not better than the state-of-the-art approaches. Therefore, I remain my score as weak accept. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkg6KFj0Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new method for learning to make neural networks sparse by adopting a beta-Bernoulli prior with variational dropout. One motivation for the method is to make the dropout rate dependent on the input, by introducing the input to a layer as a factor in the computation of the Bernoulli parameter for the layer which controls whether some parts of it will get turned off. \n\nThe motivation for the goal of making neural networks sparse is clear, since it can potentially lead to significant memory and computation savings (although given that hardware architectures typically used for executing neural networks generally expect dense computations, it may be difficult to realize these savings in practice). Furthermore, it's satisfying to see that the underlying method has a strong probabilistic justification.\n\nHowever, while a significant amount of the paper was devoted to the input-dependent version, it was unclear from the empirical results whether there is much actual advantage to the additional complexity.\nEmpirical validation of the method with experiments on larger datasets such as ImageNet would lend further credence to the viability of the approach.\nI was also somewhat disappointed to see that in order to actually accomplish the pruning, the method involves a naive approximation (equations 14 and 19). I think an interesting experiment would be to see how the method performs when computing the expectation on equation 13 through empirical samples; given the savings in FLOPs and memory, we could run the network with several samples even without exceeding the original computation budget.\nSome other design choices (such as the factorization of q, and equation 17) seem somewhat arbitrary, so I would also prefer to see further justification or a sketch/empirical evidence of why alternative methods may not work as well.\n\nFor the above reasons, I am rating the paper as weak accept.\n\nSmall note: please use \\citep and \\citet (instead of \\cite, when using natbib) properly throughout the paper so that citations are formatted correctly.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new adaptive variational dropout approach to balance accuracy, sparsity and computation. 

The method proposed here is sound, the motivation for smaller (perhaps sparser) networks is easy to follow. The paper provides experiments in several data-sets and compares against several other regularization/pruning approaches, and measures accuracy, speedup, and memory. The reviewers agreed on all these points, but overall they found the results unconvincing. They requested (1) more baselines (which the authors added), (2) larger tasks/datasets, and (3) more variety in network architectures.  The overall impression was it was hard to see a clear benefit of the proposed approach, based on the provided tables of results.

The paper could sharpen its impact with several adjustments. The results are much more clear looking at the error vs speedup graphs. Presenting "representative results" in the tables was confusing, especially considering the proposed approach rarely dominated across all measures. It was unclear how the variants of the algorithms presented in the tables were selected---explaining this would help a lot. In addition, more text is needed to help the reader understand how improvements in speed, accuracy, and memory matter. For example in LeNet 500-300 is a speedup of ~12 @ 1.26 error for BB worth-it/important compared a speedup of ~8 for similar error for L_0? How should the reader think about differences in speedup, memory and accuracy---perhaps explanations linking to the impact of these metrics to their context in real applications. I found myself wondering this about pretty much every result, especially when better speedup and memory could be achieved at the cost of some accuracy---how much does the reduction in accuracy actually matter? Is speed and size the dominant thing? I don't know.

Overall the analysis and descriptions of the results are very terse, leaving much to the reader to figure out. For example (fig 2 bottom right). If a result is worth including in the paper it's worth explaining it to the reader. Summary statements like "BB and DBB either achieve significantly smaller error than the baseline methods, or significant speedup and memory saving at similar error rates." Is not helpful where there are so many dimensions of performance to figure out. The paper spends a lot of time explaining what was done in a matter of fact way, but little time helping the reader interpret the results.

There are other issues that hurt the paper, including reporting the results of only 3 runs, sometimes reporting median without explanation, undefined metrics like speedup ,%memory (explain how they are calculated), restricting the batchsize for all methods to a particular value without explanation, and overall somewhat informal and imprecise discussion of the empirical methodology.

The authors did a nice job responding to the reviewers (illustrating good understanding of the area and the strengths of their method), and this could be a strong paper indeed if the changes suggested above were implemented. Including SSL and SVG in the appendix was great, but they really should have been included in the speedup vs error plots throughout the paper. This is a nice direction and was very close. Keep going!
Decision: Reject