Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper studies the problem of avoiding Monte Carlo (MC) estimate for the predictive distribution during the test for Bayesian methods. MC estimate will incur multiple passes where the number of passes depends on the number of samples and therefore the cost can be huge. The authors propose One-Pass Uncertainty (OPU) methods to approximate the predictive distribution through distillation. Experiments on Bayesian neural networks are conducted to demonstrate the proposed method.

Quality:
The proposed method appears to be technically sound. The view of approximating the predictive distribution over simplex is interesting and may inspire future studies under this formulation. Although the restriction of the student distribution to be tractable seems to limit the design of the student model significantly. And this restrictive distribution family may cause large amortization error, as suggested by Lemma 1 in the paper.

The experiments are well-conducted, and the proposed method is well-evaluated.

Significance:
This paper studies an important problem in Bayesian machine learning and the proposed method can be combined with many Bayesian methods to reduce the computational cost during the test. 

Originality:
As far as I know, the method is novel. The related work is adequately cited. 

Clarity:
This paper is well-written and easy to follow. 


review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Thank the authors for your detailed rebuttal. I agree with the authors that the proposed method acts as a useful tool for "real-time evaluation of induced predictive uncertainty", and the experiments also validate that the method indeed achieves comparable performance with smaller computations. But for now, I am inclined to not change my score.

###################


Bayesian models maintain the posterior distribution for predictions, which might bring up big computational costs of multiple forwards or big memory costs of multiple particles. To resolve the computational and memory issues at predictions, this paper proposes to distill Bayesian models into an amortized prediction model, avoiding the original multiple forwards. Specifically, in classification, they distill the predictive probabilities into an amortized Dirichlet distribution. They evaluated different distillation metrics, including KL divergence, Earth moving distance, and Maximum mean discrepancy.  Empirically, they evaluate the proposed method over out-of-distribution detection. They demonstrate that their method achieves comparable performance with much speedup.

Strengths, 
1, This paper is well-written and the ideas are well-presented. They evaluated the proposed method over different Bayesian models (MCDP & SGLD) as well different metrics (KL, EMD, MMD), and demonstrate the effectiveness of their method. Overall, this paper is very comprehensive.
2, As evaluated and validated in the experiments, the proposed method vastly reduces the inference time at test phase. 

Weakness,
1, The paper kind of lacks of novelty. Basically the proposed method distills a Bayesian models into an amortized Dirichlet distribution, which is straightforward. 
2, The baselines such as MCDP-KL, MCDP-EMD are strange, it is wired why you would distill the predictive distribution of a single point to a Dirichlet distribution. And I think it is probably unfair, as distilling the single-point distribution to the Dirichlet under KL, EMD, MMD might require large amount of particles, which they don't have.
3, Related to (2), more baselines should be compared with to better demonstrate the method's effectiveness. 1) performance of the un-distilled MCDP and SGLD models. 2) BDK and DPN for the MCDP models. 3) MCDP and SGLD with fewer particles. The paper claims to achieve 500x speed up, while I reckon the performance of MCDP and SGLD won't deteriorate a lot if you use only fewer particles. 
4, It would be interesting to see experiments other than out-of-distribution detection, such as calibration. 

Minor Issues,
1, The paper has several un-complied references, such as above eq(4) and appendix D.
2, The \Tau(x | \theta) in Figure 1 is a typo.
3, Assumption 1 should be put forward to the main articles for comprehensiveness of Lemma 1.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Overall I liked several results presented in this paper. The findings in Figure 2 gives clear illustration on how Bayesian classification models distinguish between in-distribution difficult-to-classify data and out-of-distribution data, namely uncertain predicted mean and large predicted variance. Though I believe this eventually depends on what kind of "kernel"s are used to correlate data points in the prior, throughout the paper I assume meaningful "kernel"s are used (for Bayesian NNs this is rooted in the inductive bias of neural networks). 

Another result that I liked is in experiments we can clearly see the advantage of considering the bayesian predictive distribution over a single predictive mean. As demonstrated by BDK-SGLD vs. BDK-DIR-SGLD. 

The proposed idea is a simple and meaningful improvement over previous works. Though the contribution is quite limited, the authors present it with great clarity, which I appreciated. However, I do find some discussions in the paper unnecessary and would expect for more technical contributions. For example, I didn't see the argument for the whole section discussing amortization gap. Everything seems straightforward given the hypothesis F is of enough capacity, which obviously does not hold in practice.

Many other concerns are summarized below:
* What if the teacher predictive distribution is far unlike a Dirichlet? How much is the discrepancy between teacher and student predictive distribution? Theoretical or empirical evidence is needed for this modeling choice.
* One importance advantage of Bayesian classification models is that they can capture the covariance between predictions  of different data points. By amortization this advantage no longer exists.
* In the paper the authors keep mentioning that the method can be applied to GPs but I don't see experiments or algorithms for it?
* The concentration model is parameterized using an exponential activation, how does this activation affect the performance?
* The distilling process is done on a held-out dataset. Which may not be wanted because an advantage of Bayesian classification models (eg. GPs) is that all hyperparameters can be automatically selected by marginal likelihoods and don't need a held-out validation set.
* MMD/wasserstein distances are cool but they require also samples from the student, which adds more variance to the distillation process.
* The experiment setup is extremely unclear to me. What is "uncertainty measures", are they used as metrics for detecting out-of-distribution data, how are AUROC/AUPR calculated using the uncertainty measures? I can guess the meaning but the paper should be more clear about this.
* I found most numbers convincing except that sometimes BDK-SGLD outperforms BDK-DIR-SGLD, if I understand it right, the predicted mean of BDK-DIR-SGLD should be as good as BDK-SGLD?

Minor:
* On page 4, above Eq. (4) there is a broken figure link.
* On Page 7, "To save space, we only present the best performing uncertainty measure (E, P or C)". What is "C" here?
review_title: Official Blind Review #4

Meta-review: This paper proposes to speed up Bayesian deep learning at test time by training a student network to approximate the BNN's output distribution. The idea is certainly a reasonable thing to try, and the writing is mostly good (though as some reviewers point out, certain sections might not be necessary). The idea is fairly obvious, though, so the question is whether the experimental results are impressive enough by themselves to justify acceptance. The method is able to get close to the performance achieved by Monte Carlo estimators with much lower cost, although there is a nontrivial drop in accuracy. This is probably worth paying if it achieves 500x computation reduction as claimed in the paper, though the practical gains are probably much smaller since Monte Carlo methods are rarely used with 500 samples. Overall, this seems a bit below the bar for ICLR.

Decision: Reject