Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary:
The paper introduces a general framework for distilling expectations of the Bayesian posterior distribution of a deep neural network, aiming to extend the original Bayesian Dark Knowledge approach [1]. More concretely, the generalized framework takes as input a teacher network, a general posterior expectation of interest, a student network, and thus performs an online compression of the selected posterior expectation using iteratively generated Monte Carlo samples from the parameter posterior of the teacher model. The proposed framework is applied to the case of classification models and empirical results demonstrate that distilling into a student model with an architecture that matches the teacher, as is done in Bayesian Dark Knowledge, can lead to sub-optimal performance. It is also shown that student architecture search methods can identify student models with significantly improved speed-storage-accuracy trade-offs.

Strengths:
Overall, the paper is well written and the relationship to previous works is well described. I personally like the Bayesian Dark Knowledge approach, which combines SGLD and knowledge distillation or dark knowledge, and very happy to see its generalization. Unlike the previous work, it is clearly shown that restricting the student architecture to match the teacher can sometimes lead to a significant performance drop, which provides a basis for guiding future developments.

Weaknesses:
- I think it is a valuable contribution, but my major concern is that the authors only conduct experiments for the classification task, whereas the original Bayesian Dark Knowledge approach also deals with the regression task and shows some interesting results (see Sect. 3.2 and 3.3 in Ref. [1]). I would recommend the authors to extend the experimental evaluation and provide some insight on how to extend the proposed framework to cover the regression task.
- On page 5, the choice of loss function does not seem to be discussed. I would like the authors to clarify why cross entropy loss is replaced with l(h, h’)=|h-h’| in the classification case.
- The size of some figures appears too small, for example Fig. 1 and Fig. 2, which may hinder readability.

At the moment, I recommend a weak reject as the main weakness is the experimental evaluation, but I could be open to increasing my score if my concerns are addressed.

References:
[1] Anoop Korattikara Balan, Vivek Rathod, Kevin P Murphy, and Max Welling. Bayesian dark knowledge. In Advances in Neural Information Processing Systems, pp. 3438–3446, 2015.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The authors consider the problem of distilling expectations with respect to Bayesian neural network (BNN) posteriors. These expectations rely on Monte Carlo integration and owing to the large number of BNN parameters can be computationally expensive and memory intensive to compute, motivating the need for distillation.  

I recommend a weak accept for the paper. The authors generalize previous work on distilling posterior predictives by allowing for the computation of posterior expectations beyond posterior predictive distributions, proposing alternate low variance MC estimators, and using an amortization network whose architecture need not be identical to the original BNNs architecture. 

While the extensions individually are incremental and not particularly exciting, taken together, I believe, they do address a gap in the existing literature. The experiments successfully demonstrate a) when naive distillation fails and b) the proposed extensions help alleviate some of the observed issues. The paper would likely be an useful resource for practitioners in the area. 

Minor:
+ Us vs Uo estimators: It would be interesting to more clearly see what the additional storage (and computation) of Uo is buying us. How much worse are the posterior predictive entropies if Uo is switched with Us? And do the posterior predictive estimates improve if Uo is used inlace of Us? 

+ In the paragraph following equation 4, the posterior marginal variance expression implicitly assumes that p(y|x, \theta) is a Categorical distribution. This should be clarified. The expression doesn’t generally hold, for example if p(y | x, \theta) is a Gaussian.

+ Figures 1 and 2 are too small and difficult to parse. I would recommend moving some of these to the supplement. 

+ It would be good to explicitly point out how much larger is the best (one with the smallest teacher student gap) l1/l2 regularized model compared to the base student model. I realize this is hiding in Figure 2 somewhere, but is not obvious.  
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Contributions:

The paper considers the distillation of a Bayesian neural network as presented in [Balan et al. 2015]

The main contribution of the paper is the extension of [Balan et al. 2015] to apply to general posterior expectations instead of being restricted to predictions. 

A second contribution of the paper is the finding that restricting the architecture of the student network to coincide with the teacher can lead to suboptimal performance and this can be mitigated by expanding the student's architecture using architecture search.

Originality/Significance:

I want to discuss the result regarding the generalization of the posterior expectation (section 3.1). To my knowledge this is novel, however, I am failing to see the importance of this result. The paper mentions two cases as motivations: to calculate the entropy and the variance of the marginal. The  problem with these two examples is that it is unclear why these are important and they are not used in the experiments anywhere. Their use should be motivated and the performance of the distillation should be properly evaluated in the experiments.

The result regarding architecture search is interesting, but it should be expanded and explained more to be a main contribution.

Clarity:

The paper generally understandable, and well written, but it could be better organized. It should expand on the motivation and the architecture search, since these are key components of the paper. The figures are not legible. Even fully zoomed in, they are difficult to read.

Overall assessment:

The paper has some interesting ideas, but it lacks motivation and significant results.

_______________________________________________________________________

Response to the rebuttal:

Thank you for the detailed reply.

> A primary motivation for generalising posterior expectations is to help quantify model uncertainty. Indeed, the expectation of the predictive entropy is an important quantity that is distinct from the entropy of the posterior predictive distribution. For instance, the difference between these two quantities is exactly the BALD score used in active learning [1]. ...

BALD would be problematic to use with this framework due to computational costs. The main benefit of distillation is the reduced computational cost at inference time. But training itself is still expensive. In BALD, the bulk of the computational cost is fitting the model after each new observation. Distillation does not provide a speedup here.

If the method indeed works well in an active learning setting, it would be interesting to see experiments showcasing this result.

> - The result regarding architecture search is interesting, but it should be expanded and explained more to be a main contribution.

I was hoping for more discussion/guidance on finding the right architecture or perhaps an algorithm that efficiently optimises the architecture. But I understand that this is more of a future work so I am not holding this against the paper.

I realise that my initial assessment was rather short so I decided to increase my rating and lower my confidence score. I think the paper is borderline, but I am slightly leaning towards rejection due to the insufficient motivation.
review_title: Official Blind Review #3

Meta-review: The authors consider distilling posterior expectations for Bayesian neural networks. While reviewers found the material interesting, and the responses thoughtful, there were questions about the practical utility of the work. Evaluations of classification favour NLL (and typically do not show accuracy), and regression (which was considered in the original Bayesian Dark Knowledge paper) is not considered. In general, it is difficult to assess and interpret how the approach is working, and in what application regime it would be a gold standard, e.g., with respect to downstream tasks. The authors are encouraged to continue with this work, taking reviewer comments into account in a final version.
Decision: Reject