UNCERTAINTY QUANTIFICATION USING VARIATIONAL INFERENCE FOR BIOMEDICAL IMAGE SEGMENTATION
Authors: Abhinav Sagar
Venue: ICLR 2022 Submitted (2022)

Abstract: Deep learning motivated by convolutional neural networks has been highly successful in a range of medical imaging problems like image classification, image
segmentation, image synthesis etc. However for validation and interpretability, not
only do we need the predictions made by the model but also how confident it is
while making those predictions. This is important in safety critical applications
for the people to accept it. In this work, we used an encoder decoder architecture
based on variational inference techniques for segmenting brain tumour images. We
evaluate our work on the publicly available BRATS dataset using Dice Similarity
Coefficient (DSC) and Intersection Over Union (IOU) as the evaluation metrics.
Our model is able to segment brain tumours while taking into account both aleatoric
uncertainty and epistemic uncertainty in a principled bayesian manner.

Under review as a conference paper at ICLR 2022
UNCERTAINTY QUANTIFICATION USING VARIATIONAL INFERENCE FOR BIOMEDICAL IMAGE SEGMENTATION
Anonymous authors Paper under double-bling review
ABSTRACT
Deep learning motivated by convolutional neural networks has been highly suc- cessful in a range of medical imaging problems like image classification, image segmentation, image synthesis etc. However for validation and interpretability, not only do we need the predictions made by the model but also how confident it is while making those predictions. This is important in safety critical applications for the people to accept it. In this work, we used an encoder decoder architecture based on variational inference techniques for segmenting brain tumour images. We evaluate work the publicly available BRATS dataset using Dice Similarity Coefficient (DSC) and Intersection Over Union (IOU) as the evaluation metries. Our model is able to segment brain tumours while taking into account both aleatoric uncertainty and epistemic uncertainty a principled bayesian manner.
INTRODUCTION
Medical image segmentation is a challenging task for medical practitioners. is costly, takes time and is prone to error. Hence there is a need to automate the manually done segmentation. Lately Neural Networks have shown great potential on variety of medical image segmentation problems. The challenge with the approaches used in literature is that the model doesn't predict the uncertainty associated. This is where Bayesian methods come into play as it gives principled way of measuring uncertainty from the model predictions. Measuring uncertainty the output predictions made by neural networks is important for interpretation and validation. Rather than learning the point estimates, Bayesian Neural Networks (BNN) learns the distribution over the weights. The training process of BNN involves first initializing the parameters of the neural network. Next the weights sampled from some distribution (like gaussian with zero mean and unit variance) and both the forward pass and backward pass done to update the weights using the conventional backpropagation algorithm.
Monte Carlo dropout networks (Kingma et al., 2015) use dropout layers to approximate deep Gaussian processes which still lack theoretical understanding. Bayesian Convolutional Neural Network (Gal and Ghahramani, 2015) use variational inference to learn the posterior distribution over the weights given the dataset. The problem with this approach is that it requires lot of computation involving lot of parameters, making this technique not scalable in practice. Variational Autoencoder (Kingma al., 2015) which is based on generative models solves the above problems and has been successful in number of tasks like generating images, texts, recommender systems etc. This approach comes with several challenges in own right which have been successfully tackled in the literature. random variable sampled from posterior distribution has no gradient so the conventional backpropagation techniques can't 't be applied it. Local Reparameterization Trick (Kingma et al., 2015) was proposed to tackle this by converting the random variable to deterministic one for computation. The second challenge was the huge computational requirement since it required weight updates in every iteration. Bayes by Backprop algorithm (Blundell et al., 2015) tackled this by calculating gradients in back- propagation using a scale and shift approach by updating the posterior distribution in the backward pass.
to

---

Under review as a conference paper at ICLR 2022
2
RELATED WORK
2.1 MEDICAL IMAGE SEGMENTATION
The problem of segmenting medical images have been successfully tackled in literature using mainly two techniques, first using Fully Convolutional Network (FCN) (Long et al., 2015) and second those which are based on U-Net (Ronneberger al., 2015). The main characteristic of FCN architectures is that it doesn't 't use fully connected layers at the end which have been used successfully for image classification problems. U-Net methods on the other hand uses an encoder-decoder architecture with pooling layers in the encoder and upsampling layers in the decoder. Skip connections connect the encoder layers to the decoder layer to create an additional path for the flow gradients back in the backpropagation step. This helps in reducing overfitting due to many parameters involved while training the network.
2.2 BAYESIAN NEURAL NETWORK
Lately, there has been a revival of interest in bayesian methods as some of the inherent problems with deep learning could be solved using it. It is scalable approach of avoiding overfitting in neural networks and the same time gives us a measure of uncertainty. This very important in critical applications where not only do we require the predictions made from the model, but also how confident it is while making its predictions. BNN can be considered as an ensemble of neural networks (Gal, 2016). It has two advantages overfitting and second it gives a measure of uncertainty involved.
the standard neural networks, first avoids
Instead of point estimates, the neural network learns posterior distribution over the weights given the dataset as defined in Equation 1.
p(w D) = p(([w)p(w) p(D)
I P L (1) p(w)
(1)
p(D)
The predictive distribution can be calculated by approximating the integral as defined in Equation
p 1 D) (v"10" ,w) D)w)
(2)
The challenge is that the posterior often intractable in nature. To combat this, (Neal, 1993) used Markov Chain Monte Carlo (MCMC) for learning the weights over the bayesian neural networks. Also (Graves, 1), (Blundell et al., 2015) and (Louizos and Welling, 2016) proposed independently technique using variational inference techniques for approximating the posterior distribution. KL Divergence between the posterior and the true distribution can be calculated using Equation 3.
go go(w) du
KT. (go (po(w))p())
go (w) log p(w
Alternatively minimizing the KL divergence can be written in another form by maximizing the Evidence Lower Bound (ELBO) which is tractable. This is shown in Equation 4.
- -f go(w) log log w)w + KL {go (w) [p[(W)
(4)
2.3 VARIATIONAL INFERENCE
Variational inference finds the parameters of the distribution by maximizing the Evidence Lower Bound. ELBO consists of sum two terms Kullback-Leibler (KL) divergence between two distribu- tions and the negative log- lig-liieldoood (NLL) defined in Equation 5.
(go min KT. (go (w=) (Ip(w[))
(5)
2

---

Under review as a conference paper at ICLR 2022
The KL divergence is defined in equation 6.
p(x)
(L(((=))lp()))= - - g(x)) log
(6)
((x)
The posterior in the above equation contains an integral which is intractable in nature. The equation can be re written in Equation
KL (go (Pp(())) = (10) log (o(w)p(D)
p(D) w p)w))
(w) log go(w)
log)(DD + E (w))
gs (w) log pogp(Dlut (D/w)
p(w)
= (()) (
The above equation can be decomposed into two parts one of which is the KL divergence between the exact posterior and its variational approximation which needs to be minimized and the second is ELBO term which needs to be maximized. This is shown in Equation 8.
max log P(D) = max [KL (go(w)) [l((w))) + C(0) 0
(8)
8
KL divergence is zero exact posterior is equal to variational approximation. Since the KL divergence always greater than zero hence the equation can be approximated by maximizing only the ELBO (Kingma et al., 2015) as defined in equation 9.
((0) (w) log ((Dw)) Eno(w) log go(w) p(w)
Lo KL (go(w))l((w)
2.4 ALEATORIC UNCERTAINTY AND EPISTEMIC UNCERTAINTY
There are two types of uncertainty- aleatory and epistemic uncertainty where variance is the sum of both these. Bayesian Neural Networks can be considered ensemble of neural networks initialized randomly which averages the test results in parallel (Gal, 2016). For final predictions, single mean and variance can be estimated as shown in Equation 10 and Equation 11 respectively.
M
1
(x(
(x(x))
(10)
M
i=1
(x)
1 M
Σ
(x)))
-d(()) (x)x
(11)
The first term in variance denotes aleatoric uncertainty while the second denotes epistemic uncertainty. Bayesian Neural Network model for uncertainty estimation was done by (Kendall and Gal, 2017) with the last layer representing the mean and variance logits. The predictive distribution approximating the posterior distribution which gives measure of uncertainty is defined in Equation 12.
/ p (y"|a"" (ww
La (v"1=") -
(12)
Aleatoric uncertainty is a measure of the variability of the predictions from the dataset hence is inherent in the data present. The aleatoric uncertainty is the uncertainty that exists inside the dataset. captures noise inherent in the observations, which arises from the distribution of data. is highly dependent bias and distribution inside the input data but not the size of training samples.

---

Under review as a conference paper at ICLR 2022
Epistemic uncertainty the other hand is measure of the variability of predictions from the model which is tied to various metrics used for evaluation like accuracy, loss etc. It represents the uncertainty inside the model. The epistemic uncertainty will decrease with enough training samples. It is resulting from the limitation of knowledge and data of the system.
METHOD
3.1 DATASET
To validate the performance of our proposed approach to generalization, publicly available datasets were used for brain tumour segmentation BRATSI8 (Menze al., 2015) and (Bakas al., 2018). contains MRI scans of 175 patients with glioblastoma and lower grade glioblastoma. The images were of resolution x40x 155 pixels. The ground truth labels were created by expert neuroradiologists. The sample from the dataset is shown in Figure 2.
Figure Example MRI slices and ground truth segmentation
3.2 DATA AUGMENTATION
The following data augmentation methods were used to increase the size of dataset:
Rescaling: We rescale the pixels values by rescaling factor 1/255.
2. Rotation:
Random rotations with the setup degree range between [0, 360] were used.
3. Height and Width Shift: Shifting the input to the left or right and up or down was performed.
4. Shearing Intensity: It refers to shear angle (unit in degrees) in counter-clockwise direction.
Brightness: uses brightness shift value from the setup range.
3.3 NETWORK ARCHITECTURE
The prior distribution helps to incorporate learning of the weights over the network. Variational Autoencoder has been used successively as a kind of generative model by sampling from the prior distribution in the encoder. The decoder uses the mean vector and standard deviation vector from the latent space to reconstruct the input. Our model uses a similar encoder decoder architecture as that used in VAEs with the input to the encoder coming from pre trained image segmentation architecture. We tried difffrent backbones which have previously enjoyed success and found original U-Net gave the best results. The input the encoder only needs the mean vector, the standard deviation vector of the conditional distribution expressing the confidence with which the pixels are correctly predicted. After passing through the encoder, the parameters get converted to a latent representation which is again sampled in mean and standard deviation vector. The decoder later recovers this back to the original distribution. The conventional backpropagation algorithm used for training the model with gradient descent. The network architecture is shown in Figure 1.
Let dataset be denoted by D {x,, W variational approximation of the posterior distribution by qe (w). encoder as ( (z]w) and decoder as (w)).
The objective function used in this work is defined in Equation 13:
4

---

Under review as a conference paper at ICLR 2022
muz
muw
mua sigmaw
sigmar z
sigmar
Encoder
Decoder
Backward
pass
Figure Illustration of our network architecture
Capprox
Σ
log qo:
(0)1)) log (apc (8)()(1) + log Poro (w))13
(13)
The network trained using Gradient Descent as defined in Equation and Equation 15:
0 0=0+ave and v 3/ + BVvA
(14)
m
log Pa (2(0) (2())) (e0),,ll)) el()))
96
(15)
171
We need as predictions the posterior distribution of the model parameters which is denoted as go (ww).
The complete algorithm used in our work shown below:
Algorithm 1: Uncertainty Quantification using Variational Inference for Biomedical Image
Segmentation
Input: Dataset D (1 yi)};²
Input: Variational approximation of the posterior distribution qe (w)
Input: encoder (2/w)) (w) and decoder (w)
while not converged do
Sample minibatch: D ((x,, w
Sample weights with reparametrization: 60 qo, (1o(0)
Sample latent variables with reparametrization: 200) ~ Teg (2/(0(0)
Compute stochastic gradients of the objective:
Lapprox = Σ; log qa; (w(())
log poo (2) + log Paro (w0(1
Update parameters 0 = + a VeL and t 1 + V.LL
Σm V log p& (2/(1) l20 (x)))) ((())
Io +
end
Output: qo (w) posterior distribution of the model parameters
3.4 EXPERIMENTAL DETAILS
The hyperparameters used in our model are specified Table 1.
In addition to the above hyperparamaters, cyclical learning rate schedulers and ReduceLROnPIaaeee
was used. In gradient descent, the value momentum was taken as 0.9, y value of 0.1 and weight decay of 0.0005.
5

---

Under review as a conference paper at ICLR 2022
Table 1: Hyperparameters details
Parameter
Value
Batch Size
16
Optimizer
Adam
Learning Rate
0.001
LR scheduler patience
10
LR scheduler factor
0.1
Latent Variable Size
10
Max epochs
500
(rotout(Both Training and Test)
0.3
3.5 LOSS FUNCTIONS
combination of binary cross entropy and dice losses have been used to train the network. The first part binary cross entropy a commonly used loss function for classification problems as shown by (Goodfellow al., 2016). Every pixel in the image needs to be classified and hence loss function be written as shown in Equation 18.
LCE = ΣVi, yi. log Di.j + (1 Vi.,) log (1 Di.j)
(16)
The problem with binary cross entropy loss that it doesn't 't take into account the class imbalance as the background is the dominant class. This is one of fundamental challenges in semantic segmentation problems. Dice Loss is robust to the aforementioned problem which based on Dice Similarity Coefficient as defined in Equation 19.
FN, + FP, 2TP, + FN, + FP,
N (1-DSCC))
EDICE
(17)
Both the loss terms were combined single term with more weight given the Dice Loss term since it handles the class imbalance problem better. This is shown in Equation 20.
= LDICE 0.1 LcE
(18)
3.6 EVALUATION METRICS
Evaluation metrics for semantic segmentation problems which have often been used in literature are Dice Similarity Coefficient (DSC) also known as F1-score and Intersection over union (IoU). The corresponding equations are shown in Equation 16 and Equation 17 respectively.
2TP
DSC
(19)
2TP + FN FP
TP
IoU
(20)
TP+
True positive (TR), false negative (FN) and false positive (FP) number of pixels is calculated separately for each image and averaged over the test set. The ground truth is labelled manually by experts which compared against.
RESULTS
Figure displays how accuracy (y-axis) behaves based how much the data (x-axis) holds.

---

Under review as a conference paper at ICLR 2022
1.00
0.99
0.98
0.97
0.96
0.55
094
MC Dropout
Dropout
0.3
Deep Ensemble
Retained data
Figure 3: Accuracy data distribution of our network using uncertainty quantification approach validation/test sets.
We have generate the uncertainty value of each input based on the predicted output. This can be done using the below uncertainty quantification methods:
MC dropout: This is method based on Bayesian Neural Networks and activated dropout function at inference time. This can be defined using the below equation:
Uncertainty
Σ
Ps log
/M;
(21)
med
where M is MC sample size, is the number of classes, i, j the indexes of M, c and p; is the probability distribution of each image.
2. Deep ensembles: We collect the results of each image trained on five models and compute the average mean. We then calculate the entropy of these mean values as the uncertainty value on each image.
3. Ensemble MC dropout: This method is a combination of MC dropout and deep ensembles. It contains Bayesian Neural Networks and thus following similar way to compute MC dropout but with the average value of these five models. This can be defined using the below equation:
D
M
U'ncertaintyyess-mee =
Σ
Pj log P,)/ / DD
(22)
where D presents how many models are there in ensemble MC dropout, k; i, j are the indexes of D, M, c and other variables same as MC dropout uncertainty formula above.
Table 2: Uncertainty mean value of samples in the validation/test sets by our network. Uncertainty(Test)
Model
Uncertainty
MC dropout
0.1607
0.1217
Deep ensembles
0.1204
0.1146
Ensemble MC dropout
0.2235
0.1213
The Mean Dice Similarity value for various backbone architectures compared against different train size values are shown in Table
The IOU value for various backbone architectures compared against different train size values shown in Table 3.
The predicted segmentation along with uncertainty involved in segmentation is shown in Figure 4:

---

Under review as a conference paper at ICLR 2022
Table 3: Mean Dice Similarity metrics for the experiments
Train Size
DSC
53.1
10
56.6
15
60.8
20
64.3
Table Intersection over Union metries for the experiments
Train Size
IOU
5
48.4
10
50.6
15
53.1
20
55.8
Figure 4: Examples of models predictions on test samples. compared to ground truth segmentation. First column: input image, second column: ground truth segmentation, third column: predicted segmentation, fourth column: aleatoric uncertainty and fifth column: epistemic uncertainty.
The darker(dull) color denotes confidence while the lighter(brighter) color means the model is less confident while segmenting those areas. Another point to be noted is that aleatoric uncertainty and epistemic uncertainty are co-related i.e with the increase in confidence taking into account alcatoric uncertainty while segmenting brain tumours results in increase of epistemic uncertainty and vice versa.
5 CONCLUSIONS
In this work, we proposed way to quantify uncertainty in the context of medical image segmentation. Our network is based of the network represent distributions instead of point estimates and thus give a principled way of measuring uncertainty the same time while making the predictions. Our model uses bayesian neural networks for both the encoder and decoder. The inputs to encoder come from pre trained backbones like U-Net, V-Net and FCN sampled from conditional distribution representing the confidence with which pixels are labelled correctly. We evaluated our network on publicly available BRATS dataset with our model outperforming previous state of the using DSC and IOU evaluation metrics.
encoder decoder framework similar to that used by VAEs. The weights
ACKNOWLEDGMENTS
We would also like to thank Nvidia for providing the GPUs.
8

---

Under review as a conference paper at ICLR 2022
REFERENCES
M. S. Ayhan and P. Berens. Test-time data augmentation for estimation of hetcroscedastic alcatoric uncertainty in deep neural networks. 2018.
Bakas, M. Reyes, A. Jakab, S. Bauer, M. Rempiler. A. Crimi, R. T. Shinohara, C. Berger, S. M. Ha, M. Rozycki, al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv: arXiv: 1et.::1111..029.. 2018.
C. Blundell, Coreeiiee, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty neural networks. arXiv preprint 1505:0550000524 2015.
A. Bromiley, N. A. Thacker, M. L. Scott, M. Pokrić, A. Lacey, and 'T. F. Cootes. Bayesian and non-bayesian probabilistic models for medical image analysis. Image and Vision Computing, 21 (10):811-864, 2003.
Y. Gal. Uncertainty in deep learning. University of Cambridge, 1(3), 2016.
Y. Gal and Z. Ghahramani, Bayesian convolutional neural networks with bernoulli approximate variational inference. arXiv preprint arxiv:1506.021555 2015.
Goodfellow, Y. Bengio, and A. Courville. Deep learning. MIT press, 2016.
A. Graves. Practical variational inference for neural networks. In Advances in neural information processing systems, pages 2346-235,, 2011.
M. Havaei. A. Davy, D. Warde-Farley, Biard, A. Courville, Y. Bengio, C. Pal. P.-M. Jodoin, and H. Larochelle. Brain tumor segmentation with deep neural networks. Medical image analysis, 35: 18-31, 2017.
J. Jiang, P. Trundle, and J. Ren. Medical image analysis with artificial neural networks. Computerized Medical Imaging and Graphics, 34(8)=617-631, 2010.
A. Jungo and M. Reyes. Assessing reliability and challenges of uncertainty estimations medical image segmentation. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 48-56. Springer, 2019.
A. Jungo, R. Meier, E. Ermis, M. Blatti-Moreno, E. Herrmann, R. Wiest, and M. Reyes. On the effect of inter-observer variability for reliable estimation of uncertainty of medical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention. pages 682-690- Springer, 2018.
A. Kendall and Gal. What uncertainties do we need in bayesian deep learning for computer vision? In Advances in neural information processing systems, pages 5574-5584, 2017.
D. P. Kingma and J. Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.6900. 2014.
D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint 1r1iv:1312.6114, 2013.
D. P. Kingma, T. Salimans, and M. Welling. Variational dropout and the local reparameterizzaion trick. In Advances neural information processing systems, pages 2575-2583, 2015.
Y. Kwon, J.-H. Won, B. Kim, and M. C. Paik. Uncertainty quantification using bayesian neural networks in classification: Application to biomedical image segmentation. Computational Statisties & Data Analysis, 142: 1068 16, 2020.
M. L,,, J. Unkelbach, N. Ayache, and H. Delingette. Sampling image segmentations for uncertainty quantification. Medical image analysis, 34:42-51, 2016.
C. Leibig, V. Allken, M. S. Ayhan, Berens, and Wahl. Leveraging uncertainty information from deep neural networks for disease detection. Scientific reports, 7(1):1114, 2017.

---

Under review as a conference paper at ICLR 2022
Y. Li and Y. Gal. Dropout inference in bayesian neural networks with alpha-divergences. arXiv preprint S.t3:17003022555 2017.
G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. Van Der Laak, B. Van Ginneken, and C. Sánchez. survey on deep learning in medical image analysis. Medical image analysis, 42:60-88, 2017.
J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
C. Louizos and M. Welling. Structured and efficient variational deep learning with matrix gaussian posteriors. In International Conference on Machine Learning, pages 1708-1716, 2016.
Ma, Lin, S. Wesarg, and M. Erdt. A novel bayesian model incorporating deep neural network and statistical shape model for pancreas segmentation. In International Conference on Medical Image Computing and Computer-Assistie
Intervention, pages 480 487. Springer, 2018.
O. Maier, B. H. Menze, J. von der Gablentz, L. Häni, M. P. Heinrich, M. Liebrand, S. Winzeck, A. Basit, P. Bentley, L. Chen, et al. Isles 2015-a public evaluation benchmark for ischemic stroke lesion segmentation from multispectral mri. Medical image analysis, 35:250-269, 2017.
B. H. Menze, K. Van Leemput, D. Lashkari, T. Riklin-Raviv, Geremia, Alberts, Gruber, S. Wegener, M.-A. Weber, G. Székely, et al. A generative probabilistic model and discriminative extensions for brain lesion segmentation wwith application to tumor and stroke. IEEE transactions on medical imaging, 35(4):933-944 946, 2015.
F. Milletari, N. Navab, and S.-A. Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pages 565-571. IEBE, 2016.
A. Myronenko. 3d mri brain tumor segmentation using autoencoder regularization. In International MICCAI Brainlesion Workshop, pages 31 1-320. Springer, 2018.
Nair, Precup, Amold, and T. Arbel. Exploring uncertainty measures deep networks for multiple sclerosis lesion detection and segmentation. Medical image analysis, 59:101557, 2020.
R. M. Neal. Bayesian learning via stochastic dynamics. In Advances in neural information processing systems, pages 475-482, 1993.
A. Norouzi, M. S. M. Rahim, A. Altameem, T. Saba, A. E. Rad, A. Rehman, and M. Uddin. Medical image segmentation methods, algorithms, and applications. IETE Technical Review, 1199)1199-21,, 2014.
O. Ronneberger, Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer, 2015.
A. Saad, G. Hamarneh, and T. Möller. Exploration and visualization of segmentation uncertainty using shape and appearance prior information. IEEE Transactions on Visualization and Computer Graphics, 16(6):1366-1377) 2010a.
A. Saad, T. Möller, and G. Hamameh. Probexplorer: Uncertainty-guict exploration and editing of probabilistic medical image segmentation. In Computer Graphics Forum, volume 29, pages 1113-1122. Wiley Online Library, 2010b.
A. Sagar. Bayesian multi scale neural network for crowd counting. arXiv preprint arXiv:2007.12245. 2020.
A. Sagar and J. Dheeba. Convolutional neural networks for classifying melanoma images. bioRxiv, 2020a.
A. Sagar and J. Dheeba. On using transfer learning for plant disease detection. bioRxiv, 2020b.
10

---

Under review as a conference paper at ICLR 2022
K. Shridhar, F. Laumann, and M. Liwicki. Uncertainty estimations by softplus normaliza- tion bayesian convolutional neural networks with variational inference. arXiv preprint arXiv:1806.05978. 2018.
N. Srivastava, G. Hinton, A. Krizhevsky, Sutskever, and R. Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929-1958, 2014.
A. Taha and A. Hanbury. Metrics for evaluating 3d medical image segmentation: analysis, selection, and tool. BMC medical imaging, 15(1):29, 2015.
D. Tran, M. Dusenberry, M. van der Wilk, and D. Hafner. Bayesian layers: A module for neural network uncertainty. In Advances Neural Information Processing Systems, pages 14660-14672, 2019.
Veredas, H. Mesa, and L. Morente. Binary tissue classification wound images with neural networks and bayesian classifiers. IEEE transactions on medical imaging, 29(2)4410-477, 2009,
G. Wang. W. Li, M. Aertsen, J. Deprest, S. Ourselin, and T. Vercauteren. Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks. Neurocomputing, 338:34-45, 2019.


Reviews:
[{"review_id": "ymvLcG5y_d9", "reviewer": "Reviewer_MfyG", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The novelty is quite limited, as VAE has been explored in the medical domain [1]. Plus, it is a common practice in medical image segmentation to estimate the uncertainty of segmentation result by performing multiple times of dropout at test time [2]. There is not any improvement on their VAE architecture as well. \n\nClarity: The paper is clearly written and easy to follow, but has some typos. \n\nOverall, the novelty of the paper does not meet the standard of ICLR, and there is not any meaningful contribution in this paper. So I vote for rejection.\n\n[1] Sedai, Suman, et al. \"Semi-supervised segmentation of optic cup in retinal fundus images using variational autoencoder.\" International Conference on Medical Image Computing and Computer-Assisted Intervention\n\n[2] Wang, Guotai, et al. \"Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks.\" Neurocomputing 338 (2019): 34-45. \n", "summary_of_the_paper": "This paper propose to use VAE to perform the brain tumor segmentation task and uncertainty estimation. ", "summary_of_the_review": "The paper does not have any meaningful contribution.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "j2oZu4QbhG8", "reviewer": "Reviewer_UXjb", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There are multiple major issues about the paper. \n\n- Methods description is very difficult to understand. It is not very clear to me how the proposed method works\n- Experimental evaluations are quite limited. There are some Dice score and IoU values, but there is no comparison with another method in the literature.", "summary_of_the_paper": "The paper proposes a method for uncertainty quantification for biomedical image segmentation. The proposed method takes mean and std of segmentation from a backbone segmentation model and trains a VAE on top of it. The method is evaluated on only BRATS dataset.", "summary_of_the_review": "There are multiple major issues about the paper. \n\n- Methods description is very difficult to understand. It is not very clear to me how the proposed method works\n- Experimental evaluations are quite limited. There are some Dice score and IoU values, but there is no comparison with another method in the literature.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "fihsBPxLk5S", "reviewer": "Reviewer_5veL", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is a number of weaknesses:\n\n- No attempt is made to put the work in the context of related work. There many, many works on uncertainty estimation in the context of biomedical image segmentation (see below). The authors give the impression that they are the first to tackle this challenge.\n\nStochastic Segmentation Networks, Monteiro et al, NeurIPS 2020\nA probabilistic U-net for segmentation of ambiguous images, Kohl et al, NeurIPS 2018\nmany more\n\n- The paper has no contribution section where the contributions of the paper are explicitly listed. From the description of the methodology segmentation it seems that that there no novel contributions, all of the components of the method are well known (augmentation, loss function, variational inference, etc).\n\n- The results look rather underwhelming, no attempt is made at comparing the results to those of other methods that have been evaluated on the BRATS challenge.\n\n- No attempt is made at evaluating the usefulness of the uncertainty estimates for the users.\n", "summary_of_the_paper": "The paper propose a deep learning based method for taking into account both aleatoric uncertainty and epistemic uncertainty in biomedical image segmentation tasks. The proposed method is based on variational inference techniques with a standard encoder-decoder CNN architecture. The method is applied to brain tumour MR images from the standard BRATS segmentation challenge.", "summary_of_the_review": "The paper makes no significant technical contribution or innovation and does not offer any strong results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "M6PynPdDyGF", "reviewer": "Reviewer_YKhZ", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n-- U-Net architecture as backbone is a strong choice for the medical image segmentation and the proposed method was evaluated on publicly available dataset.\n\n-- The paper provides results of 3 different uncertainty quantification methods (\"MC dropout\", \"Deep Ensembles\", \"Ensemble MC dropout\"), , and DSC/IOU of varying train sizes.\n\n-- The paper includes background of the existing theory and sentences are usually clear.\n\n\nWeaknesses:\n\n-- The novelty of the paper is quite limited. The authors are encouraged to re-evaluate the study from the methodological and/or clinical perspective, and provide clear and specific points about their novel contributions at the end of the introduction. Related work section does not include any information about the existing related studies in the literature (e.g. [1, 2, 3]).\n\n-- The paper could possibly be considered as an application paper but the contributions in the paper have marginally low significance because the paper provides very limited empirical insights with already available methods with slight changes. \n\n-- The organization of the paper should be improved. For example, the majority of the paper contains just definitions of the known formulas. Most equations presented in the paper are missing references. Can the authors clarify whether there are any novel equations in the paper? Also, some methodological details are missing. The authors stated \"Our model uses a similar encoder decoder architecture as that used in VAEs with the input to the encoder coming from a pre trained image segmentation architecture.\". What type of VAE architecture was this? Were there any modifications? With which data was the encoder pre-trained? The authors are encouraged to add these details to the paper. \n\n-- I also highly recommend that the authors compare their methodology with the previous works (e.g. [1]). In its current form, the paper does not provide any comparison with the existing studies in the literature, or a comparison of different backbones even though the authors stated that \"We tried different backbones which have previously enjoyed success and found original U-Net gave the best results.\" in the paper. Can the authors clarify what those backbones are? I assumed these are U-Net, V-Net and FCN as stated in the conclusion (without any references) but the authors are encouraged to list them in the method section (with references to the specific architectures unless they are novel).\n\n\n\n[1] Kwon, Yongchan, Joong-Ho Won, Beom Joon Kim, and Myunghee Cho Paik. \"Uncertainty quantification using Bayesian neural networks in classification: Application to biomedical image segmentation.\" Computational Statistics & Data Analysis 142 (2020): 106816.\n\n[2] Nair, Tanya, Doina Precup, Douglas L. Arnold, and Tal Arbel. \"Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation.\" Medical image analysis 59 (2020): 101557.\n\n[3] Ghoshal, Biraja, Allan Tucker, Bal Sanghera, and Wai Lup Wong. \"Estimating uncertainty in deep learning for reporting confidence to clinicians in medical image segmentation and diseases detection.\" Computational Intelligence 37, no. 2 (2021): 701-734.", "summary_of_the_paper": "The paper proposes a method to quantify uncertainty in medical imaging, which is an important task for clinical applications, with variational inference. It uses the U-Net architecture and BRATS18 dataset for evaluation. It quantifies uncertainty with 3 methods and evaluates the predictions with Dice score (DSC) and Intersection over Union (IOU). ", "summary_of_the_review": "The novelty of the paper is quite limited, and the paper should be rewritten to include important metholodogical details as well as comparisons with the existing studies to assess its impact and significance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a method for uncertainty quantification for medical applications, which quantifies both aleatoric and epistemic components.

The paper initially received three strong reject recommendations. The main limitations pointed out by reviewers relate to the limited contributions (either methodological or applicative and clinical), the lack of positioning with respect to related works, the presentation needing improvement and the lack of experimental comparison with respect to recent relevant baselines. 
No rebuttal was provided. \
The AC carefully read the submission and agrees that the paper is premature for publication in the current form. Therefore, the AC recommends rejection.
Decision: Reject