Venue: MIDL 2018 Conference

Reviews:

final_score: rating: 3: Marginally above acceptance threshold
review_content: 
This work presents an uncertainty driven sanity check for brain tumor cavity segmentation. Overall the paper is well written and presented and the results of tumor segmentation is promising. There are however few issues that needs to be addressed;

1. Predicted entropy or the doubt metric is dependent on the quality of segmentation. The metric however does not incorporate any segmentation errors in it. Please explain.

2. Please explain why a 3D CNN was not used. Considering the images are all isotropic after data pre-processing (1x1x1 mm), 3D UNet and DeepMedic has often used for brain lesion and tumor segmentation, the model needs to be compared to those models.


review_title: The authors present an uncertainty driven sanity check for brain tumor cavity segmentation. 

final_score: rating: 4: Top 50% of accepted papers, clear accept
review_content: The work uses a fully convolutional DenseNet for brain tumor cavity segmentation and proposes to use a doubt score for uncertainty estimation.The score is derived from network uncertainty using the Monte Carlo dropout method. The derived sanity check is supposed to help understanding of neural network outputs and to add to their quality assurance by automatic identification of segmentation errors which need expert review.

Pros:

- Automatic (and unsupervised) quality assurance of fully-automated segmentation methods can be of high significance, especially when large datasets need to be processed.

- The proposed method does not need a reference segmentation for computing the "doubt score" which makes it possible to apply also to unseen data without reference segmentation available, in contrast to other metrics such as the Dice score. Therefore it is also independent from a human observer.

- The proposed method for the sanity check is fairly simple and could also be applied to a variety of different network architectures, not just the one used in the paper.

- Reference data quality: the authors use contours for training and evaluation produced through a majority vote of 3 clinical experts which is likely to be of higher quality than data from a single observer and will lead to more robust quantitative results.

- Overall, the paper is clearly written and has a logic structure which is easy to follow.

Cons:

- Size of the dataset: A main motivation by the authors is that "deep learning methods still lack robustness when applied to real-world clinical data" which can be subject to "considerable variability" and thus lead to "significant mispredictions". However, they evaluate their sanity check method on a rather small dataset of 30 patients only, which will most probably not capture most of the variability that could be expected (also during training). Also, variability in MR images is often due to use of different scanners and different protocols in different clinics, not just anatomical variability. This is not adressed and it is not explicitely stated whether all 30 patients were imaged in the same clinic and scanner.

- With a sensitivity of 4/7 and a false discovery rate of 5/9, the doubt score as a means of identifying bad segmentations (with a Dice <0.75) is in its current form not completely convincing. The specificity is higher, but it seemed that the main motivation was to detect the bad segmentations with a high confidence for expert review. Then again, the dataset used is probably too small to draw too many conclusions from these numbers.

- The "importance of the spatial prior" is not completely evident from the results. In some cases, the distance weights in the doubt score seem to introduce false positives (e.g. case 3: high doubt score due to far off region with high uncertainty, even though the segmentation quality seems acceptable) and false negatives (e.g. case 17: on the shown slice about half of the cavity is missing, but as this is close to the boundary, the doubt score is still low). It would be interesting to see results such as in Figure 4 when omitting the distance weight or using a non-linear transform of the euclidean distance.

Other comments:

- It is unclear how the number of Monte Carlo dropout networks was chosen to be T=20 and how much this number impacts the resulting doubt score. Is there some sort of convergence as T grows? The choice of threshold for the doubt score seems also not very intuitive.

- The need for registration to an atlas is not obvious, resampling and cropping could be done without. Is there any other reason to use the atlas?

- The authors state that a 2D approach is favored over 3D also because of the anisotropic resolution in the unprocessed images. This is not very convincing as resampling to isotropic voxels is done during pre-processing before the network sees any images.

- The notation "dense layer" to refer to a sequence of BatchNorm, Conv, ReLU etc. within a "dense block" might be a bit confusing as "dense layer" often refers to fully connected layers.

- In the last section of the discussion the notations false/true positives/negatives are used to refer to the segmentation result. This is a bit confusing, as before these terms were used to describe the sanity check results. Maybe rephrase that last part?

- It could be nice to see the doubt scores in the figures 3 and 5 for direct comparison.
review_title: The authors propose a fairly simple approach for unsupervised quality control of deep neural network segmentations which seems overall convincing as a first step towards auto quality assurance. A weakness is the relatively small data set size of 30 patients only (significance of results?). The spatial prior in the quality score is not yet convincing. It would be interesting to see future evaluations on larger data sets and modifications to the spatial prior.

final_score: rating: 3: Marginally above acceptance threshold
review_content: The paper describes a deep neural network applied to post-operative MR images for the segmentation of brain tumor cavities.  In addition it estimates the certainty of the segmentation result across the image to potentially identify images/segmentations which would require manual checking in a clinical environment.  The paper is well written and methods are clearly described.  The dataset is appropriately split into training/test using crossfold validation, but no validation set is used to set network architecture and parameters.  This implies that the network is tuned to obtain optimal results on the test set used here and may not generalise with similar performance.
The methods described (network for segmentation and dropout for uncertainty estimation) are not novel in themselves, only the application.  The segmentation results on a dataset of 30 subjects are good and it is demonstrated that the uncertainty estimate is useful (if not foolproof) in identifying possible incorrect segmentations for manual checking. 
While I would not reject the paper immediately I would give only a tentative recommendation to accept based on the low novelty value of the methodology.
review_title: A well constructed paper describing application of deep learning methodologies to brain tumor cavity segmentation.

Decision: Poster