Adaptive network sparsification with dependent variational beta-Bernoulli dropout

Abstract: While variational dropout approaches have been shown to be effective for network sparsification, they are still suboptimal in the sense that they set the dropout rate for each neuron without consideration of the input data. With such input independent dropout, each neuron is evolved to be generic across inputs, which makes it difficult to sparsify networks without accuracy loss. To overcome this limitation, we propose adaptive variational dropout whose probabilities are drawn from sparsity inducing beta-Bernoulli prior. It allows each neuron to be evolved either to be generic or specific for certain inputs, or dropped altogether. Such input-adaptive sparsity- inducing dropout allows the resulting network to tolerate larger degree of sparsity without losing its expressive power by removing redundancies among features. We validate our dependent variational beta-Bernoulli dropout on multiple public datasets, on which it obtains significantly more compact networks than baseline methods, with consistent accuracy improvements over the base networks.

Reviews and author responses:

The paper proposes a new way of training variational dropout which is adaptive to input samples due to the proposed sparsity-inducing beta-Bernoulli prior. The authors provide a good motivation for their model, introduce beta-Bernoulli and dependant beta-Bernoulli prior and propose the method in the variational inference framework. 

Concerns:
1) The main concern relates to the significance of benefits from the input dependency property. From the Tables 1, 2 we can see that BB is comparable with DBB and the latter is not uniformly better than the former in terms of error, xFLOPs and memory. This similarity in the performance is more significant for LeNet5-Caffe network, for CIFAR-10 and CIFAR-100 datasets. Is the overhead of DBB worth the benefits it gives? 
2) The second question is about the memory consumption of DBB. The authors use two stage pruning scheme for DBB: at first, they prune DBB using the beta-Bernoulli dropout, then prune the network for each input individually. It means that DBB should keep all weights after the first stage, in other words the memory consumption of DBB is the same as BB. Some clarifications about this concern are necessary. 

Overall, the paper proposes interesting and well-motivated method for training sparse networks. Although, there are concerns about the DBB extension I would recommend considering this paper for acceptance. 

---------------------------------------
Update after author rebuttal

Thank you for your thoughtful response. You addressed my second concern about memory consumption of DBB. Indeed, the run-time memory mostly consists of activation maps, therefore DBB can benefit from input-dependent sparsity. Considering the first concern I still think that the advantage of DBB is not clear and I agree with AnonReviewer3 who said that except LeNet-500-300 other results are mixed. 

However, overall I do think that the proposed method is novel, well theoretically grounded and is proved that it works comparably if not better than the state-of-the-art approaches. Therefore, I remain my score as weak accept. 


This paper presents a model with beta-bernoulli dropout of neurons for network pruning. The model assumes the probability to dropout a neuron is a function pi(phi, x), which depends both the beta variable phi and the input x. The model is trained with stochastic gradient variational Bayes with continous relaxation. The model and algorithm sound, and the intuition of determining the dropout probability based on the importance of each dimension makes sense. 

One weakness of this work is the lack of large-scale experiments, for example, pruning a MobileNet on ImageNet. This work also seems incremental due to its resemblance with CD.

Figure 1: bottom-right figure (input-regions pruned by DBB) is missing?

====
Update:
The authors do address my concern #2. After reading other reviews and reading the revised paper I do think this approach of this paper is novel and can potentially lead to a gain. However I still don't think the experiments are convincing enough. It needs to be tested on a larger variety of models (ResNet or MobileNet) / datasets (ImageNet, etc.) / tasks (vision, nlp, etc.) to prove its significance. Therefore I won't change my score.

Thank you for your reply.

> 1) Unclear advantage of dependent Beta-Bernoulli dropout
I took another look at Figure 1, Figure 2, and the appendix's Table 2 and Table 3. Among the 4 experiments, it seems that only one (LeNet-500-300) shows the advantage, while the others are much more mixed.

Given the multiple dimensions that are relevant (error, FLOPs, memory), perhaps it would be useful to more explicitly show how one of the dimensions vary as the others remain fixed. In other words, for multiple pruning schemes that reach the same error rate, how does the FLOPs or memory compare? Or for the same FLOPs, how does the error compare?

> 2) Naive approximation
Thank you for the explanation that sampling is not useful. If space allows, it may be helpful to include a brief explanation about this in the paper (unless I missed it already in the paper).

1) Concern regarding the input-dependent beta-Bernoulli dropout. 
- DBB indeed obtains significant performance gain over BB in both accuracy and efficiency, except for LeNet5-Caffe. Please refer to our first response to R3 for more details. We have included the discussion on this part in the revision.

2) Concern regarding the memory consumption of DBB.

- This seems like a misunderstanding. DBB does not use the same amount of memory as BB, although it makes use of the same amount of parameters, since the largest part of run-time memory goes to the memory to store the activation maps. The runtime memory usage by DBB is significantly lower than BB, since during the inference we compute the feature map from the bottom layer to the upper layer, eliminating any neurons/filters with the mask which do not need to be stored. We include all costs, including additional overhead for storing the parameter to compute the mask and the masks themselves, when reporting the memory use.

1) Unclear advantage of dependent Beta-Bernoulli dropout

Please take a closer look at Figure 1 and Table 2 in the appendix. In the error-speedup tradeoff graphs in Figure 1 and Figure 2, DBB is much closer to the bottom right corner, which means that the method achieves lower error with larger speedup. This trend is clearly shown in the LeNet-500-300. Thus DBB yields a significantly more compact network than BB. CIFAR-10 and CIFAR-100 experiments, except on LeNet5-Caffe. We conjecture that input-dependent pruning is less effective on highly underparameterized networks such as LeNet5-Caffe. Yet, even in this case, DBB can always prune more since it starts from the solution of BB.

This relative advantage of DBB over BB is shown more clearly in Table 2 in the appendix. BB achieves error of 1.40 with 21.79 xFLOPs while DBB achieves the same error with 38.75 xFlOPs. This is a 1.77 times speedup. In Table 3 of the appendix, which shows the results on VGG-CIFAR-100, BB yields the error of 28.86 with 2.60 xFLOPS while DBB records the error of 28.84 with 2.96 xFLOPs. Please note that all FLOPs and memory footprint computations reported in our paper accounts for all the additional overheads due to input-dependent dropout. Thus we believe that the results provided in the paper clearly demonstrate the effectiveness of DBB over BB. We added in these discussions in the revision.

2) Naive approximation
- We tested MC sampling and averaging inference as you suggested, but found no gain in the accuracy. This is due to the following fact; the variance of the binary mask is given as Var(z) = E_q[pi](1 - E_q[pi]) by the law of total variance. Since the pruning method either tries to make E_q[pi] close to one (retaining the neuron) or zero (pruning the neuron), Var(z) in general tends to zero. Hence, the overall variance of the pruned network is close to zero, which makes the sampling not very different from the naive approximation. The previous works (VIB paper, SBP paper, and so on) also reported this phenomenon, presumably due to a similar mechanism we described above. It would be an interesting research direction to prune the network while maintaining the prediction uncertainty. 

3) Justification for design choices
The factorization of q is mainly due to the computational reason. We may introduce normalizing flow like variational distributions to increase flexibility, but that would harm the speedup and require more parameters. We agree that the design choice in Eq. (17) may look arbitrary, but we believe that it is nonetheless a reasonable choice for constructing efficient input-dependent dropout gates. An alternative to this may be using layer normalization, which may work as well. 

4) Small note.
Thanks for pointing out this. We have corrected this in the revision.

1) No large scale experiments: 

We agree that our paper could be made stronger with large-scale experiments, although we believe that the current set of results is sufficient in showing the superiority of our method over existing sparsification approaches. Due to the limited time provided for rebuttal, we could not perform ImageNet-scale experiments, but we will perform this experiment and include it in the final version if the paper is accepted.

2) Resemblance to Concrete Dropout (CD)

While both works are similar in the learning of dropout rates for individual neurons/filters, our Beta-Bernoulli Dropout is fundamentally different from CD as it is a sparsification method that yields sparse deep neural networks, while CD does not. We show in the experiments (See Table 1) that DBB outperforms CD in terms of accuracy while using only a fraction of the parameters. Further, input-dependent sparsification with dependent Beta-Bernoulli is a novel concept that has not been discussed in existing works on neural network sparsification. 

3) Typo in the caption of Figure 1
Thanks for pointing out this. We have corrected it in the revision.

We thank you for the constructive feedback. During the rebuttal period, we have done our best to address your confusions. 

1) In sections 4.1 and 4.2 (formula 7, 8 and 18) is there a global z or is there a separate z for each layer? I implicitly prefer it is per layer but it is not clear from the paper.

- Since we mention in Page 4 that W is a parameter of a neural network layer, we are referring to a layerwise z throughout the subsection. We clarified this in the revision as you suggested.

2) How is the inference done? 

- First of all, we apologize for the typo. E_q[z_k] smaller than a fixed threshold 10^-3 is a more accurate description. Once the training is done, we can pre-compute the average probability to active the neurons/filters, which is E_q[z_k] (Eq.(15) for beta-Bernoulli and Eq.(20) for dependent beta-Bernoulli). 

Then we can explicitly prune neurons/filters having probability lower than a threshold value (1e-3) to ignore them in the inference step. We set this threshold to 1e-3 to eliminate the need for hyperparameter tuning, but setting them to some other values, such as 1e-2 or 1e-4 does not significantly impact the accuracy. While we feel that all the necessary information is already there, in the revision, we described the inference step more in detail to help readers better understand the procedure. 

3) Median and standard deviation? 

- The reported errors are mean values. We apologize for the confusion and have corrected it in the revision. We have reported memory use and xFLOPS since we did not have space to include them. Previous works have also reported the median values following [Luizos et al. 18].

4) In Table 1, what does the column neurons represent? 

- The neurons in Table 1 report reports the number of active neurons after the pruning, which actually participate in the inference. We *do not* report the best performance but report the median over three runs.

5) Missing baselines: SVD (sparse variational dropout) and SSL (structured sparsity learning). 
 
We did have a comparison to SVD and SSL in an earlier version of the paper but removed them to focus on the comparison against more recent baselines. The two methods largely underperform the recent baselines and ours. For example, for LeNet 500-300 experiments, SVD achieved an error rate of 1.43 with 16.2x FLOPs and SSL recorded error 2.25 with 15.3x FLOPs. On the other hand, according to Table 2 in the appendix, BB recorded error 1.28 with 16.23x FLOPs. We included the results on the two baselines back into the revised paper. Please see Tables 2 and 3 in the appendix.
 
6) Confusing results in Table 2: 


We report two versions with different sparsity level for our BB and DBB since there is a trade-off relationship between sparsity and accuracy. Both Table 1 and 2, in the top rows, reports the performance of BB and DBB when the error is similar to the other methods. They show that given a similar error level, BB and DBB achieve higher sparsity. The bottom rows report the best error with corresponding xFLOPs and memories. To see the overall tradeoff trend, please refer to the tradeoff line figures or Tables 2 and 3 in the appendix. Both of them show clear advantage of our method in terms of accuracy over existing models at similar sparsity levels, and in terms of memory use at similar accuracy levels.  

We thank all the reviewers for their thoughtful and constructive comments. We uploaded the revision reflecting the reviews, and the changes made are
- We gave more detail in the inference procedure (how the neurons/filters are pruned),
- We discussed the advantage of DBB over BB,
- We included the results for Structured Sparsity Learning (SSL) and Sparse Variational Dropout (SVD) in Table 2 and 3 in the appendix,
- Fixed typos and misleading statements.
We address the reviewers’ concerns in the individual comments.


The paper proposes a new method for learning to make neural networks sparse by adopting a beta-Bernoulli prior with variational dropout. One motivation for the method is to make the dropout rate dependent on the input, by introducing the input to a layer as a factor in the computation of the Bernoulli parameter for the layer which controls whether some parts of it will get turned off. 

The motivation for the goal of making neural networks sparse is clear, since it can potentially lead to significant memory and computation savings (although given that hardware architectures typically used for executing neural networks generally expect dense computations, it may be difficult to realize these savings in practice). Furthermore, it's satisfying to see that the underlying method has a strong probabilistic justification.

However, while a significant amount of the paper was devoted to the input-dependent version, it was unclear from the empirical results whether there is much actual advantage to the additional complexity.
Empirical validation of the method with experiments on larger datasets such as ImageNet would lend further credence to the viability of the approach.
I was also somewhat disappointed to see that in order to actually accomplish the pruning, the method involves a naive approximation (equations 14 and 19). I think an interesting experiment would be to see how the method performs when computing the expectation on equation 13 through empirical samples; given the savings in FLOPs and memory, we could run the network with several samples even without exceeding the original computation budget.
Some other design choices (such as the factorization of q, and equation 17) seem somewhat arbitrary, so I would also prefer to see further justification or a sketch/empirical evidence of why alternative methods may not work as well.

For the above reasons, I am rating the paper as weak accept.

Small note: please use \citep and \citet (instead of \cite, when using natbib) properly throughout the paper so that citations are formatted correctly.

The paper proposes Variational Beta-Bernoulli Dropout to sparsify the parameters of the network. The authors also proposed an input-dependent dropout using input-dependent Beta-Bernoulli priors. The paper presents necessary theoretical details as well as the experimental comparison with the other dropout methods on MNIST and CIFAR 10/100 datasets.

The paper is well-written overall and easy to follow. The paper provides a thorough background on previous work; however the motivation for having an input-dependent dropout method is relatively weak. 

I believe the paper presents main idea and necessary details thoroughly, though I have some confusions:

- In sections 4.1 and 4.2 (more specifically formula 7, 8 and 18), is there a global z (i.e. only one z), or there is a separate z for each layer? I implicitly inferred it's per layer but it's not clear from the paper.

- The paper explains well how the two proposed dropouts can be learned during training, but it's not clear about the inference, specially in variational dependent Beta-Bernoulli dropout. In section 5, experimental details, it is said: "In the testing phase, we prune the neurons/filters whose expected dropout mask probability E_q[pi_k] are smaller than a fixed threshold 10^-3." This is not clear to me what it means, and also the related footnote which says different thresholds were tried but the difference is insignificant! More elaboration is helpful.

- In the experimental results, the authors reported median and standard-deviation. Is there any reason the authors didn't report mean and standard-deviation instead? Also, it is not clear if xFLOP and Memory report the best, mean or median of 3 runs.

- In Table 1, what does column Neurons represent? Is it the model parameters at inference time? and if yes, is it the smallest number of parameters in 3 runs? In Table 2, how large the network is in terms of model parameters?

- In the experimental results, some models are missing that are mentioned in the paper as previous work which seem to be good to use for comparison, such as Sparse Variational Dropout (Molchanov 2017) and Structured Sparsity Learning (Wen et al 2016).

- I encourage the authors to expand more on analyzing the experimental results! In Table 1 and Table 2, there are 2 lines for BB and DBB, which is helpful to explain what each line represents. Looking at the results of all tables, and comparing different dropouts w.r.t :1) Error, 2) xFLOPs and 3) Memory, I cannot draw a clear conclusion from the results. For example, in Table 2, CIFAR-10, comparing first line of BB to SBP and second line to VIB, the difference is not significant. More analysis about the results can be helpful!

Meta-review: This paper introduces a new adaptive variational dropout approach to balance accuracy, sparsity and computation. 

The method proposed here is sound, the motivation for smaller (perhaps sparser) networks is easy to follow. The paper provides experiments in several data-sets and compares against several other regularization/pruning approaches, and measures accuracy, speedup, and memory. The reviewers agreed on all these points, but overall they found the results unconvincing. They requested (1) more baselines (which the authors added), (2) larger tasks/datasets, and (3) more variety in network architectures.  The overall impression was it was hard to see a clear benefit of the proposed approach, based on the provided tables of results.

The paper could sharpen its impact with several adjustments. The results are much more clear looking at the error vs speedup graphs. Presenting "representative results" in the tables was confusing, especially considering the proposed approach rarely dominated across all measures. It was unclear how the variants of the algorithms presented in the tables were selected---explaining this would help a lot. In addition, more text is needed to help the reader understand how improvements in speed, accuracy, and memory matter. For example in LeNet 500-300 is a speedup of ~12 @ 1.26 error for BB worth-it/important compared a speedup of ~8 for similar error for L_0? How should the reader think about differences in speedup, memory and accuracy---perhaps explanations linking to the impact of these metrics to their context in real applications. I found myself wondering this about pretty much every result, especially when better speedup and memory could be achieved at the cost of some accuracy---how much does the reduction in accuracy actually matter? Is speed and size the dominant thing? I don't know.

Overall the analysis and descriptions of the results are very terse, leaving much to the reader to figure out. For example (fig 2 bottom right). If a result is worth including in the paper it's worth explaining it to the reader. Summary statements like "BB and DBB either achieve significantly smaller error than the baseline methods, or significant speedup and memory saving at similar error rates." Is not helpful where there are so many dimensions of performance to figure out. The paper spends a lot of time explaining what was done in a matter of fact way, but little time helping the reader interpret the results.

There are other issues that hurt the paper, including reporting the results of only 3 runs, sometimes reporting median without explanation, undefined metrics like speedup ,%memory (explain how they are calculated), restricting the batchsize for all methods to a particular value without explanation, and overall somewhat informal and imprecise discussion of the empirical methodology.

The authors did a nice job responding to the reviewers (illustrating good understanding of the area and the strengths of their method), and this could be a strong paper indeed if the changes suggested above were implemented. Including SSL and SVG in the appendix was great, but they really should have been included in the speedup vs error plots throughout the paper. This is a nice direction and was very close. Keep going!

Decision: reject