Venue: MIDL 2018 Conference

Reviews:

final_score: rating: 4: Top 50% of accepted papers, clear accept
review_content: The paper builds up on the work of Kendall ang Gal [2017]. Where it has been proposed to decompose the predictive network uncertainty into model dependent noise (epistemic) and the noise inherent to the data (aleatoric). Authors of the reviewed paper, propose an alternative noise decomposition to Kendall and Gal [2017] for classification, by noting that the variance of the multinomial rv is a function of its mean, with a subsequently provided derivation. Kendall and Gal [2017] model the variability only of a linear predictor, but not the noise of the predictive probabilities as compared to the authors of the reviewed paper, through an additional output layer to predict the data dependent heteroscedastic noise. Derivation of the authors does not require additional output layer, which allows to freely use already existing architectures that allow sampling the predicted probabilities. 

Overall, great contribution to the investigation of uncertainty quantification in deep learning and its importance in medical imaging. However, a detailed evaluation of uncertainty quality, as well as the benefits of having predictive uncertainty, over multiple datasets of medical imaging domain would be relevant to fully demonstrate the advantage of the proposed method.

When using a GP for example, we might incorporate input dependent noise using additional GP, making the model heteroscedastic. Which is what is done in the work of Gal and Kendall, where noise is input dependent. If I understand correct the aleatoric noise in your case is not heteroscedastic? 

review_title: Original approach towards uncertainty in classification using DNNs

final_score: rating: 4: Top 50% of accepted papers, clear accept
review_content: -	The paper addresses the problem of measuring the uncertainty of a segmentation that is done by a neural network.  It quantifies uncertainty in classification problems by decomposing predictive uncertainty into two parts, aleatoric and epistemic uncertainty.  The proposed method is based on Bayesian neural network and is a variant of a  method proposed by  Kendal and Gal in NIPS 2017.  They use a very similar approach.  The only difference is the parametric model that is used to describe the uncertainty in classification problems. In the original paper they used a Gauassian distribution for the values before the softmax operation. In this paper they use a parametric modeling that is explicitly aware of the discrete nature of the classification operation.


-	The paper is clearly written, and it applies an uptodate deep learning method to medical imaging.  The problem of quantifying uncertainty of a lesion segmentation that is done by a neural network is an important goal. The paper shows several examples (in figure 1) to demonstrate the performance of the proposed method compared to the original methods of Kendall and Gal. What is missing is a quantitative comparison between the two methods that is done in a systematic way on a large dataset.  If the authors could show that their method works better than the baseline the usefulness of it was clearer.


review_title: Measuring the uncertainty of a segmentation that is done by a neural network. The problem of quantifying uncertainty of a lesion segmentation that is done by a neural network is an important goal.

final_score: rating: 3: Marginally above acceptance threshold
review_content: The paper is well written, the literature related to the paper is thoroughly described, considering the limited amount of space available in the paper. The experiments give some empirical evidences favoring the proposed model, although it is difficult to compare. 
It would have been nice to see a couple of plots for aleatoric, and epistemic uncertainty vs frequency of misclassified pixels, to see if any of the estimated uncertainties can predict the probability of misclassified pixels.
It could also be interesting to see how the model performs for different values of T, and how this impacts the inference time.

In any case, given how important uncertainty quantification (of both kinds) is in medical applications, MIDL is a nice venue for this paper, so I recommend this for publication.
review_title: In this paper the authors build on the work of Kendall and Gal for quantifying the aleatoric and epistemic uncertainty for each pixel, in segmentation problems. They improve upon that work by avoiding to learn extra components to model the variance by exploiting that the variance of a multinomial is a function of its mean. That also avoids impossible outcomes like having mean near 0 or 1 in the binomial case and high variance.

Decision: Oral