Venue: MIDL 2022 Conference

Reviews:

final_score: rating: 3: Borderline
review_content: summary: 1. This paper explores adding variational inference to a multi-headed U-Net to improve model calibration and uncertainty estimation of a U-Net segmentation. 
2. Experiments on BRATS-18 dataset shows the proposed model can improve segmentation performance and uncertainty quantification over five baseline methods.
strengths: 1. The experimental designs are reasonable and complete: segmentation performance (dice score), calibration performance (ECE), and synthetic OOD performance (dice score). 
2. The insight of this paper is interesting.
weaknesses: 1. Though I think this paper provides some insights, I am worried about the novelty of the paper. The current presentation of the paper is incremental to a U-Net segmentation. The variational inference is not new and the multi-headed approach is also not new. I would encourage the author to not only do experiments to show that adding MH VI can improve U-Net segmentation, but also for other SOTA backbones. There is no evidence indicated in this paper that the idea can apply to other SOTA segmentation models.
2. Some experimental details are missing. Please see comments in the next sections.
3. The motivation of this paper is not well-established.
detailed_comments: 1. The author might consider adding significance test in table 1 for better interpretation. Some of the results are very similar and a significance can tell a better story.
2. To improve the strength of the paper, it is better to include other SOTA architecture backbones other than U-Net to test the benefit of VI and MH.
3. It is difficult to get any useful information from “We conduct all our experiments on the BRATS dataset with a 60% training, 10% validation and 30% test split.” The author may want to use absolute numbers for reporting this data split.
4. It might be better to include more information for this sentence “we sample the resolution to 256 × 256 × 155 voxels”. For example, the original image is of size (xxx), voxel spacing (yyy), downsampling to (zzz) with what methods (linear mapping, crop outliers, intensity range etc.) 
5. There is no mention about the appendix B in the main manuscript. How does it relate to the main story? The author might consider adding a few sentences to explain it.
6. I do not know how to interpret Figure 3. I do not understand why “VI MH with λ = 0.5 delivers an interpretable and well-calibrated uncertainty estimate”. For me, what I see in VI MH $\lambda$=0.5 has an obvious wrong prediction area inside the blue highlighted region. I do not think the descriptions in qualitative results paragraph convince me.
7. The author can draw a figure details about the dice score and ECE with respect to $\lambda$ ranging from 0 to 1 step with 0.1 in Appendix. In this way, it tells a better story than the current paper that only includes ($\lambda$=0, 0.5, 1).
8. Honestly, I did not find Figure 1 informative. It is the conceptual ideas without experimental results back-up. I would suggest the authors to use an example results to validate the points of variational methods capture the local minimal and multi-headed ensemble identifying different modes. This could be done by taking a close look at a local patch inside the image and visually and quantitatively demonstrate the points.
recommendation: []
review_title: null

final_score: rating: 4: Weak accept
review_content: summary: The manuscript proposes a variational inference multi-head network to improve on uncertainty prediction and calibration applied to multi-class tumour segmentation using the BRATS challenge in a context where only one rater has performed the annotations
The experiments are composed of an ablation study (addition of the VI part for multi head) and comparison to state of the art solutions (Prob-Unet, Monte-Carlo DropOut, Stochastic segmentation network and PhiSeg. Performance is also evaluated in the context of image artefacts (considered as out of distribution examples) 
strengths: - Well written paper with a clear motivation and substantial experimental design
- Good assessment of existing solutions and fair benchmarking
- Investigation of multiple aspects of the performance both in terms of segmentation performance and uncertainty modelling


weaknesses: - The loss function for the MH only model is unclear (how can the $\lambda$ value play a role) and where the ground truth play a role
- The expected calibration error does not appear to be explained
- Some of the figures are difficult to read/interpret due to the choice of markers/colors
detailed_comments: The paper is very interesting and overall well written. It is unfortunate that some key details are hidden in the supplementary material in particular the discussion regarding the choice for $\lambda$.

Regarding the experiments, 4 VIMH is compared to 12 MH without VI. For fairness, the same number of heads should probably have been chosen and an additional experiment would have been to look into the effect of the number of heads on the performance. The authors state that VI doubles the number of parameters so one would have expected the same ratio across number of heads.

Regarding Figure 2 and Figure 5, some colours are extremely similar making their distinction difficult. For Figure 5 a suggestion would be to choose also different markers
 
In terms of the loss function, it is not clear where the ground truth segmentation is used for the prediction. 

As a remark, considering the proposed artefacts as fully OOD may be a bit exaggerated as some degree of artefacts are probably observed across the chosen sample and this could explain the relatively high performance observed across models. Further, the OOD nature of the artefacts applied would probably be related to their strength. 

It is difficult from Table 2 to assess whether any of these results are statistically significant. 
recommendation: Oral
review_title: null

final_score: rating: 4: Weak accept
review_content: summary: This paper describes a strategy based on a multi-headed variational u-net design for image segmentation that is aimed at dealing with issues of global instance exploration  (i.e. out-of-distribution shifts) while handling uncertainty during a segmentation task. The final design is applied to the BRATS2018 brain tumor public dataset with experimentation on different data distribution shifts with promising results, in comparison to some other SOTA methods.
strengths: The integration of  a u-net based architecture, global exploration and a variational inference design on the heads into the basic multi-headed design of the EnsembleNet is new and interesting and offers potentially improved performance on a variety of medical image segmentation tasks. 

The detailed design of the VI (variational inference) modules is nicely done, including the introduction of the local re-parameterization “trick” originally proposed by Klingma et al.   In addition, the design of the two losses (one over all heads and one for each head), using Bayesian convolutions throughout the underlying U-Net for the overall VI MH (multihead)  architecture is an apparently unique and useful approach.  The ideas are reasonably motivated and the reduced number of parameters and relatively sped-up training is appreciated in comparison to the traditional ensemble approaches. 

The test results shown in Table 1 in terms of DICE scores show decent performance improvement over reasonable alternatives (VI U-Net, Probabilistic U-Net, etc) using an appropriate training/ validation/ testing split on the BRATS dataset.   The out of distribution (OOD) testing and performance comparisons in Table 2 and figure 5 is very helpful and more clearly shows the advantages of the proposed method under these different conditions (motion, spikes, ghosting, noise)

weaknesses: It is rather difficult to understand how the proposed architecture affects the final uncertainty in the segmentations either from the standard deviations of the DICE test results in Table 1 or in the qualitative results shown in figure 3 (i.e. the Prob U-Net uncertainty result doesn’t look much different that the VI MH  results). It would be helpful for the authors to more clearly point out the differences (and/ or point to the differences on the images in figure 3). They try to do this in section 4 in describing the different errors, but (as noted by the authors) it is all hard to interpret ---although quantitative calibration error is more clearly separated. 

Also, there is a concern that the design is tuned quite specifically for use with the BRATS2018 dataset only. In other words, will it easily extend for application to other datasets/ segmentation problems?
detailed_comments: More discussion on the meaning of improvements in uncertainty quantification due to the new design would be helpful. Perhaps more details on how the expected calibration error is obtained and what it means related to figure 3 or Table 1 would be helpful.
recommendation: Oral
review_title: null

Meta-review: metareview: The authors investigate the value of adding variational inference to a multi-headed U-Net in terms of the quality of predictive uncertainty estimation and out-of-distribution performance on the BRATS2018 segmentation challenge dataset. 

The reviewers agree that the evaluation strategies---segmentation performance (dice score), calibration performance (ECE), and synthetic OOD performance (dice score)---are quite thorough, and empirical improvements over the competing methods are significant. 

However, I would like to note that as Reviewer 1 has commented, the technical novelty might be limited as the contribution seems to be the addition of variational inference to a multi-headed U-Net architecture. 

Overall, given strong empirical results, I would like to recommend acceptance. I would like to, however, encourage the authors to describe this technical contribution more precisely in the introduction by mentioning the appropriate prior works and describing how to combine them (e.g. multi-head U-Net + VI). 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)