Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents an empirical evaluation of the effect of training with noise by dropout or dropconnect, on the predictive uncertainty of neural networks and convnets. The conclusion seems to be that applying both training and inference with noise can help the network produce better uncertainty estimates in terms of better calibrated predictions.

Although the experiments were thorough, the issue with an empirical paper like this is that it is very difficult to ensure that the lessons learned will generalize across problems and domains. This paper only investigated two simple image datasets and two neural network architectures on the task of classification. There are several ways in which I think this paper could be made stronger. First, the problem is not well motivated: why do we care about uncertainty (although I believe we do)? A good application, or a motivation section would be beneficial here. Next, what about investigating a different domain such as text or speech recognition? Or other problems such as regression? Do the results hold across different domains? Finally, if we really care about uncertainty, there are a number of other techniques for inference in Bayesian neural networks such as stochastic variational inference using the reparameterization trick, or MCMC methods like stochastic gradient Langevin dynamics. The advantage of dropout is that it’s simple and fast, but do we lose anything by doing this in terms of calibration? Other than the empirical comparison, there is little novelty to this paper, and therefore I think the conclusions drawn need to be more general or the motivation more compelling. Even addressing a subset of these suggestions would make the paper stronger in my opinion.

In Figure 4 the calibration MSE does not look so robust when MC sampling is not used. Do you have any hypothesis on why MC sampling is so important here?

In Figure 5, the legend says “CNN” when it should just be “NN”. Also the caption says convolutional, when it should say fully connected.

In Table 3, why are the results so poor for non-MC-based dropout? Although this agrees with Gal and Ghahramani, this seems to be in direct contradiction to Table 4 of the original JMLR dropout paper by Srivastava et al from 2014.

How is the noise in the test set applied for Figures 5-10? Is it Gaussian noise applied to the pixels?

review_title: Good empirical evaluation but  lacking in novelty or a strong motivation

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper investigates whether the variational inference interpretation of dropout, as introduced in [Gal & Ghahramani (2016), and Kingma et al (2015)], can lead to good estimates of mode uncertainty outside of the training distribution. This is an area of research that indeed warrants more experimental investigation. One very interesting finding is that MC integration leads to much calibration, thus probably much better out-of-sample prediction, than the more usual. 
 
Critique:
- As explained in Kingma et al (2015), when using continuous posterior distributions over the weights, the dropout rate can be optimized, leading to better regularization. While the paper is cited in the introduction, this adaptive form of dropout is missing from experiments, without clarification.

- Only the dropout rate p=0.5 was used across experiments, while the optimal rate is problem dependent, as found by earlier published work.

- No new ideas are presented, and the analysis in the paper is quite limited. As it stands, this would be more appropriate for a workshop.
review_title: No new ideas, limited analysis, and interesting (yet limited) empirical results

final_score: rating: null
review_content: The results of this analysis are based on a very specific scenario of MNIST with dropout 0.5. Do these results hold across other, more complicated datasets? Even though 0.5 is common for dropout, this is not always the value that is used. What do these results look like at different values of the dropout parameter?

Can you describe in more detail the calibration test that you apply?
review_title: Cross-dataset, different dropout values, and calibration

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors study the calibration of neural networks using different variants of dropout and weight noise. They find that sampling during training and testing improves calibration.

Most of the results are not novel and have been discussed previously by Yarin Gal and other authors. What this paper adds is a more systematic evaluation of multiple different variants of dropout.

- A comparison against bootstrapped uncertainty estimates would be useful

- Ian Osband had a paper at NIPS investigating exactly this kind of uncertainty representation in neural nets. He found that dropout represents the risk of the model, but not really the uncertainty. This difference becomes apparent when e.g. the target of the model is bi-modal.
review_title: investigates an important problem, but contains few novel results

Meta-review: The reviewers unanimously recommend rejecting this paper.
Decision: Reject