Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes Variational Beta-Bernoulli Dropout to sparsify the parameters of the network. The authors also proposed an input-dependent dropout using input-dependent Beta-Bernoulli priors. The paper presents necessary theoretical details as well as the experimental comparison with the other dropout methods on MNIST and CIFAR 10/100 datasets.

The paper is well-written overall and easy to follow. The paper provides a thorough background on previous work; however the motivation for having an input-dependent dropout method is relatively weak. 

I believe the paper presents main idea and necessary details thoroughly, though I have some confusions:

- In sections 4.1 and 4.2 (more specifically formula 7, 8 and 18), is there a global z (i.e. only one z), or there is a separate z for each layer? I implicitly inferred it's per layer but it's not clear from the paper.

- The paper explains well how the two proposed dropouts can be learned during training, but it's not clear about the inference, specially in variational dependent Beta-Bernoulli dropout. In section 5, experimental details, it is said: "In the testing phase, we prune the neurons/filters whose expected dropout mask probability E_q[pi_k] are smaller than a fixed threshold 10^-3." This is not clear to me what it means, and also the related footnote which says different thresholds were tried but the difference is insignificant! More elaboration is helpful.

- In the experimental results, the authors reported median and standard-deviation. Is there any reason the authors didn't report mean and standard-deviation instead? Also, it is not clear if xFLOP and Memory report the best, mean or median of 3 runs.

- In Table 1, what does column Neurons represent? Is it the model parameters at inference time? and if yes, is it the smallest number of parameters in 3 runs? In Table 2, how large the network is in terms of model parameters?

- In the experimental results, some models are missing that are mentioned in the paper as previous work which seem to be good to use for comparison, such as Sparse Variational Dropout (Molchanov 2017) and Structured Sparsity Learning (Wen et al 2016).

- I encourage the authors to expand more on analyzing the experimental results! In Table 1 and Table 2, there are 2 lines for BB and DBB, which is helpful to explain what each line represents. Looking at the results of all tables, and comparing different dropouts w.r.t :1) Error, 2) xFLOPs and 3) Memory, I cannot draw a clear conclusion from the results. For example, in Table 2, CIFAR-10, comparing first line of BB to SBP and second line to VIB, the difference is not significant. More analysis about the results can be helpful!
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper presents a model with beta-bernoulli dropout of neurons for network pruning. The model assumes the probability to dropout a neuron is a function pi(phi, x), which depends both the beta variable phi and the input x. The model is trained with stochastic gradient variational Bayes with continous relaxation. The model and algorithm sound, and the intuition of determining the dropout probability based on the importance of each dimension makes sense. 

One weakness of this work is the lack of large-scale experiments, for example, pruning a MobileNet on ImageNet. This work also seems incremental due to its resemblance with CD.

Figure 1: bottom-right figure (input-regions pruned by DBB) is missing?

====
Update:
The authors do address my concern #2. After reading other reviews and reading the revised paper I do think this approach of this paper is novel and can potentially lead to a gain. However I still don't think the experiments are convincing enough. It needs to be tested on a larger variety of models (ResNet or MobileNet) / datasets (ImageNet, etc.) / tasks (vision, nlp, etc.) to prove its significance. Therefore I won't change my score.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper proposes a new method for learning to make neural networks sparse by adopting a beta-Bernoulli prior with variational dropout. One motivation for the method is to make the dropout rate dependent on the input, by introducing the input to a layer as a factor in the computation of the Bernoulli parameter for the layer which controls whether some parts of it will get turned off. 

The motivation for the goal of making neural networks sparse is clear, since it can potentially lead to significant memory and computation savings (although given that hardware architectures typically used for executing neural networks generally expect dense computations, it may be difficult to realize these savings in practice). Furthermore, it's satisfying to see that the underlying method has a strong probabilistic justification.

However, while a significant amount of the paper was devoted to the input-dependent version, it was unclear from the empirical results whether there is much actual advantage to the additional complexity.
Empirical validation of the method with experiments on larger datasets such as ImageNet would lend further credence to the viability of the approach.
I was also somewhat disappointed to see that in order to actually accomplish the pruning, the method involves a naive approximation (equations 14 and 19). I think an interesting experiment would be to see how the method performs when computing the expectation on equation 13 through empirical samples; given the savings in FLOPs and memory, we could run the network with several samples even without exceeding the original computation budget.
Some other design choices (such as the factorization of q, and equation 17) seem somewhat arbitrary, so I would also prefer to see further justification or a sketch/empirical evidence of why alternative methods may not work as well.

For the above reasons, I am rating the paper as weak accept.

Small note: please use \citep and \citet (instead of \cite, when using natbib) properly throughout the paper so that citations are formatted correctly.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper proposes a new way of training variational dropout which is adaptive to input samples due to the proposed sparsity-inducing beta-Bernoulli prior. The authors provide a good motivation for their model, introduce beta-Bernoulli and dependant beta-Bernoulli prior and propose the method in the variational inference framework. 

Concerns:
1) The main concern relates to the significance of benefits from the input dependency property. From the Tables 1, 2 we can see that BB is comparable with DBB and the latter is not uniformly better than the former in terms of error, xFLOPs and memory. This similarity in the performance is more significant for LeNet5-Caffe network, for CIFAR-10 and CIFAR-100 datasets. Is the overhead of DBB worth the benefits it gives? 
2) The second question is about the memory consumption of DBB. The authors use two stage pruning scheme for DBB: at first, they prune DBB using the beta-Bernoulli dropout, then prune the network for each input individually. It means that DBB should keep all weights after the first stage, in other words the memory consumption of DBB is the same as BB. Some clarifications about this concern are necessary. 

Overall, the paper proposes interesting and well-motivated method for training sparse networks. Although, there are concerns about the DBB extension I would recommend considering this paper for acceptance. 

---------------------------------------
Update after author rebuttal

Thank you for your thoughtful response. You addressed my second concern about memory consumption of DBB. Indeed, the run-time memory mostly consists of activation maps, therefore DBB can benefit from input-dependent sparsity. Considering the first concern I still think that the advantage of DBB is not clear and I agree with AnonReviewer3 who said that except LeNet-500-300 other results are mixed. 

However, overall I do think that the proposed method is novel, well theoretically grounded and is proved that it works comparably if not better than the state-of-the-art approaches. Therefore, I remain my score as weak accept. 

review_title: Official Blind Review #2

Meta-review: This paper introduces a new adaptive variational dropout approach to balance accuracy, sparsity and computation. 

The method proposed here is sound, the motivation for smaller (perhaps sparser) networks is easy to follow. The paper provides experiments in several data-sets and compares against several other regularization/pruning approaches, and measures accuracy, speedup, and memory. The reviewers agreed on all these points, but overall they found the results unconvincing. They requested (1) more baselines (which the authors added), (2) larger tasks/datasets, and (3) more variety in network architectures.  The overall impression was it was hard to see a clear benefit of the proposed approach, based on the provided tables of results.

The paper could sharpen its impact with several adjustments. The results are much more clear looking at the error vs speedup graphs. Presenting "representative results" in the tables was confusing, especially considering the proposed approach rarely dominated across all measures. It was unclear how the variants of the algorithms presented in the tables were selected---explaining this would help a lot. In addition, more text is needed to help the reader understand how improvements in speed, accuracy, and memory matter. For example in LeNet 500-300 is a speedup of ~12 @ 1.26 error for BB worth-it/important compared a speedup of ~8 for similar error for L_0? How should the reader think about differences in speedup, memory and accuracy---perhaps explanations linking to the impact of these metrics to their context in real applications. I found myself wondering this about pretty much every result, especially when better speedup and memory could be achieved at the cost of some accuracy---how much does the reduction in accuracy actually matter? Is speed and size the dominant thing? I don't know.

Overall the analysis and descriptions of the results are very terse, leaving much to the reader to figure out. For example (fig 2 bottom right). If a result is worth including in the paper it's worth explaining it to the reader. Summary statements like "BB and DBB either achieve significantly smaller error than the baseline methods, or significant speedup and memory saving at similar error rates." Is not helpful where there are so many dimensions of performance to figure out. The paper spends a lot of time explaining what was done in a matter of fact way, but little time helping the reader interpret the results.

There are other issues that hurt the paper, including reporting the results of only 3 runs, sometimes reporting median without explanation, undefined metrics like speedup ,%memory (explain how they are calculated), restricting the batchsize for all methods to a particular value without explanation, and overall somewhat informal and imprecise discussion of the empirical methodology.

The authors did a nice job responding to the reviewers (illustrating good understanding of the area and the strengths of their method), and this could be a strong paper indeed if the changes suggested above were implemented. Including SSL and SVG in the appendix was great, but they really should have been included in the speedup vs error plots throughout the paper. This is a nice direction and was very close. Keep going!
Decision: Reject