Efficiently correcting patch-based segmentation errors to control image-level performance in retinal images

Authors: patrick  köhler
jeremiah  fadugba
lisa m koch
Venue: MIDL_io_2024_Conference
Published: 2024-01-09 10:29:24

Abstract: Segmentation models which are deployed into clinical practice need to meet a quality standard for each image. Even when models perform well on average, they may fail at segmenting individual images with a sufficiently high quality. We propose a combined quality control and error correction framework to reach the desired segmentation quality in each image. Our framework recommends the necessary number of local patches for manual review and estimates the impact of the intervention on the Dice Score of the corrected segmentation. This allows to trade off segmentation quality against time invested into manual review. We select the patches based on uncertainty maps obtained from an ensemble of segmentation models. We evaluated our method on retinal vessel segmentation on fundus images, where the Dice Score increased substantially after reviewing only a few patches. Our method accurately estimated the review’s impact on the Dice Score and we found that our framework controls the quality standard efficiently, i.e. reviewing as little as necessary.

Main paper: 
 introduction: 
 Segmentation is a central task in medical image analysis, as it often builds the foundation for surgical planning (Li et al., 2021) , diagnosis and disease progression monitoring (Soomro et al., 2019 ). In ophthalmology for example, segmenting retinal blood vessels from fundus images provides geometric characteristics such as branching angles or vessel diameters in a non invasive fashion. Unfortunately, their manual segmentation requires three to five hours per image (Jin et al., 2022) , making it unfeasible to annotate entire images routinely for every patient. Recently, medical image segmentation algorithms have achieved a performance that is sufficient for clinical deployment (Isensee et al., 2021 ). Yet, even the best models are not guaranteed to perform well on all images and may fail silently on individual ones.In medical contexts, quality standards are often crucial for safety, fairness and efficacy of therapeutic decisions. One strategy to implement such standards is to predict a Figure 1 : Overview. We obtain multiple predictions from an ensemble (a-c) and compute an uncertainty map (d) from which we select patches with highest uncertainty (e). Afterwards, we estimate the DSC and how it would change if the patches were to be reviewed manually (f ) to satisfy a target quality (g).quality metric such as the Dice Score Coefficient (DSC) per image without knowing the ground truth and exclude low quality segmentations from downstream analyses. This can be done by training an auxiliary DSC regression network in addition to the segmentation model itself (Robinson et al., 2018; Williams et al., 2021; Fournel et al., 2021) , or using probabilistic segmentation model outputs (Li et al., 2022) . Similarly, Galdran et al. (2018) learn to predict the normalized mutual information between the ground truth (GT) and the model segmentation, leveraging manually degraded GTs as a training set. However, this quality control paradigm defers entire images with subpar DSC for manual review. This may cause more manual labour than necessary because the low performance may be caused predominantly by specific image regions. For a quality assessment which is more granular than image level, Zaman et al. ( 2023) train a model to predict segmentation error maps. While such approaches identify segmentation errors they do not discuss effective strategies for corrective interventions. This is done by interactive segmentation methods (Liu et al., 2022a; Luo et al., 2021) , which incorporate manual annotations to refine their prediction. Similarly, Benenson et al. (2019) propose an error correction mechanism by training an additional network to correct the predicted segmentation. This approach relies on a manually curated dataset of corrections which is representative for the model's failure modes. Neither of these methods has attempted to quantify the effect of the correction in advance.In contrast, here we propose a combined quality control and error correction framework for vessel segmentation in retinal fundus images. Our framework proposes local candidate regions for manual review instead of deferring entire images. Then, we extend a recently proposed DSC estimator to provide an estimate of the correction's impact (Li et al., 2022) . This allows the annotator to assess how many patches should be reannotated in order to meet the required segmentation quality standards. 
multi-disease dataset for challenging vessel segmentation: 
 We used the FIVES dataset (Jin et al., 2022) , comprised of 800 high-resolution retinal fundus images from the Second Affiliated Hospital of Zhejiang University (SAHZU), China. The images were taken from healthy individuals and patients with glaucoma, age-related macular degeneration (AMD) and diabetic retinopathy (DR) (200 images each). In each image, retinal blood vessels were annotated manually by two junior annotators and verified by experienced senior annotators in a standardized procedure (Jin et al., 2022) . The presence of disease lesions made the segmentation task more challenging because these could interfere with the blood vessels. We used the original splits provided with the dataset, i.e. 600 training images, from which we used 120 for validation, and 200 test images. All images were pre-processed by applying Contrast Limited Histogram Equalization (Pizer et al., 1987) with a clip limit of 2 and a grid size of 8 x 8. The images and segmentation masks with original resolution of 2048 x 2048 were resampled to 512 x 512 pixels.We used the state-of-the-art vessel segmentation model FR-Unet (Liu et al., 2022b) to develop our framework. The architecture was optimized for the intricacies of retinal vessel segmentation, namely thin foreground structures and low-contrast regions. The model's hidden representations expanded horizontally and vertically through a multiresolution convolution mechanism to retain the full image resolution. This allowed aggregating features from different scales to supplement high-level contextual information to the low-level regimes and vice versa. In experiments with a limited range of common datasets, FR-Unet has been shown to outperform other architectures with fewer parameters (Liu et al., 2022b) .We used an ensemble of m FR-Unets, which were trained with different random seeds (Ganaie et al., 2022) . An image of size n × n was passed through the ensemble, resulting in m predicted probabilistic segmentations Y (1) p , . . . , Y (m) p (Fig. 1 a-c). The final probabilistic segmentation Y p = {p i : p i ∈ [0, 1], i = 1, . . . , n 2 }Given a fundus image and a probabilistic segmentation, our goal was to refer a minimal number of patches to manual review such that a desired segmentation quality could be guaranteed (Fig. 1 ). Our proposed framework consisted of two major components: (1) We computed pixel-wise uncertainties from the outputs of the FR-Unet ensemble to select patches as candidates for manual review (Fig. 1 a-e). (2) We estimated each patch's impact on the segmentation quality if it was to be reviewed by an expert (Fig. 1 f, g ). This allowed us to control the desired segmentation performance efficiently, i.e. only referring as few patches for review as necessary.To obtain an uncertainty map U , we computed the pixelwise entropy across all i = 1, . . . , m probabilistic segmentations Y (i) p (Fig. 1 d ) We expected that high uncertainty regions correspond to erroneous segmentations and that reviewing those would lead to efficient quality improvements. To identify local regions with potentially low segmentation quality and therefore high potential for improvement, we convolved U with a square kernel of size P × P , effectively computing the mean uncertainty in the patch around each pixel. We then selected the k non-overlapping patches with the highest uncertainty (Fig. 1 e ).Having identified candidate patches, we wanted to estimate their impact on the image's segmentation quality in terms of DSC if they were to be re-segmented manually. The DSC is defined asEQUATIONwhere, TP, FP and FN denote the number of true/ false positive and false negative pixels. Therefore, the DSC can only be computed using the GT segmentation. As GT labels are not available at test time, we can not compute Eq. 1 directly to assess the impact of reannotating a given patch. However, DSC can be estimated only having access to the probabilistic model output Y p (Li et al., 2022) . This approach relies on calibrated output probabilities, which means that for all π ∈ [0, 1] exactly π • 100% of the pixels with predicted probability π actually belong to the foreground, such that the predicted probabilities reflect the correctness of the prediction accurately.If the outputs were perfectly calibrated, summing over them would yield the number of pixels that belonged to the GT foreground (by definition). Li et al. (2022) leverage this property to construct their estimatorEQUATIONwhere 1 [.] denotes the indicator function. Hence, summing over the output probabilities of all pixels that were classified as foreground yields an estimator for TP (enumerator of Eq. 1). Analogously, we can estimate the total number of GT foreground pixels, i.e. TP + FN, by summing over all output probabilities (Eq. 2, first summand in denominator). We applied temperature scaling (TS) to the model outputs as in Li et al. (2022) to calibrate p i .The DSC estimator provided a quality assessment for an individual image. While this may be useful for quality control in itself, it does not provide actionable insight for estimating the effect of patch-based error correction. Here, we were interested in the following question: What would the DSC be if an expert reviewed specific high-uncertainty patches (Fig. 1 f )? Therefore, we propose an estimator for the DSC of the corrected segmentation Y corr that is composed of two image parts: The high-confidence regions where the model output is accepted Y model , and the manually reviewed patches Y manual in the high-uncertainty regions. The DSC of the combined segmentation Y corr can be expressed as as a linear combination of the DSC estimates of its components:DSC( Y p, corr ) = w model DSC( Y p, model ) + w manual DSC( Y p, manual ) .(EQUATIONThe DSC for Y model can be estimated with Eq. 2 by considering the foreground probabilities p i only for regions where the model output was accepted. For Y manual , we assumed perfect manual segmentation performance for simplicity, i.e. DSC( Y manual ) = 1. The weights w i correspond to the fraction of predicted foreground within Y i . We give a theoretical justification for this choice in App. A. 
segmentation performance before error correction: 
 We trained an ensemble of m = 5 FR-Unets on the training fold of the FIVES dataset (see Sec. 2.2) for 80 epochs with the DSC Binary Cross Entropy loss 1 . This loss was a combination of the softDSC loss (Milletari et al., 2016) with Cross-Entropy and has been observed to produce better generalization than softDSC alone (Liu et al., 2021; Ma et al., 2021; Galdran et al., 2022) . To improve generalization, we augmented the data with random flips and rotations. We used the Adam optimizer with a learning rate of 10 -4 and a cosine annealing scheduler (number of iterations ≤ 40). Afterwards, we selected the model with the highest validation DSC, leading to an average DSC of 0.887±0.094 (mean ± SD) on the test set (n = 200). For 10% of the images, the DSC was below 0.828 or above 0.934 (Fig. 2 a ). We found that many of the segmentation errors occured in the finer vessel structures and we mainly observed discontinuities of vessels and missing segments (see. Manually segmenting high-uncertainty regions increased the segmentation quality (Fig. 2 b ). Here, we selected the top k ∈ {0, . . . , 5} non-overlapping patches of size 81 × 81 in the uncertainty map (as in Fig. 1 e ) and replaced the predicted output with the GT to model a "perfect" human annotator. The impact of patch size is discussed in App. C.We observed a gradual increase in DSC when correcting more patches. For example, the median DSC across all images increased by more than 0.02 after correcting five patches. This was much more than could be achieved by a simple baseline approach of selecting random patches (from inside the retinal fundus) instead of high entropy patches, which increased segmentation quality at a lower rate (grey points in Fig. 2 ). For further examination, we split the test set into well-segmented images (above median DSC, Fig. 2 c ) and poorly segmented images (below median DSC, Fig. 2 d ). The segmentation quality improved more for images which were initially poorly segmented (Fig. 2 d , similar trends were observed in a subgroup analysis in App. D). Here, we observed a median increase in DSC of approximately 0.04 when reannotating k = 5 patches. For some images, the segmentation quality was improved by up to 0.1. Even for images that were already well segmented the DSC could be further improved by 0.018 on average.In summary, patch-wise error correction based on uncertainty led to a substantial increase in performance on average. However, using this simple patch selection strategy did not by itself provide an a-priori estimate for the effect of error correction.Therefore, we next predicted the effect of patch-wise error correction on the resulting segmentation quality, without access to the GT. We first assessed the estimator proposed by Li et al. (2022) (Eq. 2) to predict DSC before error correction. With a mean absolute error of 0.02, we found that it reliably estimated the segmentation quality for individual images (Fig. 3 a, red points), even though we observed a slight bias towards overestimating the true performance (mean error DSC -DSC = 0.012). Moreover, accurate DSC estimation relied heavily on calibration with temperature scaling. For uncalibrated model outputs, DSC overestimated consistently (grey points in Fig. 3 a ).Ultimately, we were interested in predicting how the DSC changed if the segmentation in high uncertainty regions of the images was corrected. Hence, we evaluated our proposed DSC estimator for a patch-wise corrected segmentation (Eq. 3) and observed constantly low estimation errors across all number of patches (Fig. 3 b ). The bias towards overestimation observed in the estimator from Li et al. (2022) carried over to our correction estimate. When we correct only one or two patches, the DSC's are being overestimated for more than half of the images. In general, we preferred conservative estimates over those liberal ones, because we rather wanted to ensure the desired quality target with a higher probability at the cost of reviewing more patches. Hence, we introduced a correction term in the subsequent analysis to reduce overconfidence post-hoc.In conclusion, the estimator from Li et al. (2022) was not only useful for its initial purpose of estimating average performance over entire datasets but also allowed us to assess how DSC would change if an expert was to review a specific set of patches.Reviewing segmentations with human experts is typically time intensive and costly. Hence, we wanted to ensure that our method is efficient, i.e. that we only request as little human resources as necessary to reach the desired quality.To quantify robustly the quality standard that has been reached by a correction strategy we chose the 5 th percentiles over the images' DSCs. In contrast to the minimum DSCs, this accounted for outliers for which the model's segmentation is fundamentally incorrect and cannot be fixed with partial review. In our framework, the remaining 5% should be referred to full review or repeated acquisition. We calibrated our DSC estimates to prevent overconfidence (cf. Sec. 3.3) by subtracting ϵ = 0.02. ϵ was optimized on the validation set such that the 5 th percentiles matched the quality target.We determined for each image individually how many patches needed to be reviewed in order to reach each of the three quality targets of DSC = 0.88, 0.90 and 0.92, where 0.92 was reported as human performance on this data set (Jin et al., 2022) . With Eq. 3, we calculated the estimated DSC after reviewing k = 1, . . . , 14 patches and select the lowest number of patches such that the estimated DSC exceeded the target. As a baseline, we chose a fixed number of patches which were reviewed for each image.Our adaptive strategy led to 3.2 reviewed patches per image on average to reach a quality standard of almost 0.90 (Fig. 4 ). The actual performance (5 th percentiles were 0.87, 0.89, 0.91) differed slightly from the desired quality standards because of imperfect DSC estimation. For the baseline, approximately three times more patches were reviewed per image to achieve a similar quality standard. for each image an equal number of patches is reviewed (grey). For our workflow, we chose three quality targets and estimated for each image individually how many patches needed to be reviewed to achieve the targets. 
discussion and conclusion: 
 We presented a framework to correct segmentation errors locally in order to control the segmentation quality per image. Correcting patches with high uncertainty led to an increase in segmentation quality for retinal blood vessels. Furthermore, we could accurately predict this increase using a DSC estimator which did not require access to the GT segmentation. Therefore, our workflow allowed to allocate more resources to images with poor segmentations and not waste resources where outputs already satisfied the quality criterion. Our method estimates DSC, which is a measure of overlap with broad applicability in many segmentation tasks. However, particularly to account for topological consistency in thin vessel structures, predicting customized metrics such as the centerline DSC (Shit et al., 2021) is an important next step for certain downstream tasks.In this paper, we assumed perfect expert performance for the patch review. While the performance may be higher than in large scale annotation settings because the reviewer can focus their attention on few small areas, this simplification ignores potential inter-rater variability. One mitigation would be to evaluate our method with multi-annotator data, where the oracle patch could be provided by a different annotator.We used deep ensembles to quantify uncertainty and suggest candidate patches for manual review. Our modular framework allows replacing this step with any other approach that generates uncertainty maps such as Monte-Carlo dropout (see Fuchs et al. (2022) for an overview and App. E for a comparison of uncertainty maps). As uncertainty estimation typically generates computational overhead, other strategies could be pursued to suggest candidate patches purely based on image statistics, e.g. by identifying low-contrast regions.Beyond covering large demands in hospitals more efficiently, our method can be applied in clinical trials, when automatic volumetric measurements inform on effect sizes of drugs. In that case, our framework could improve the efficiency of drug development by ensuring that each volumetric estimate is accurate enough for the downstream analysis of interest. Appendix A. Justification for the choice of w i in Equation 3Let us assume we are given a predicted segmentation X and the respective ground truth Y . Given two disjoint subsets of those, which we denote with subscripts, we would like to combine their individual DSC linearly to obtain the overall DSC(X, Y ). Let us consider the DSC definition in terms of set sizesEQUATIONand solve for w i . Furthermore, let the subsets be disjoint and their union the entire imageEQUATION)Multiplying each of the summands in Eq. 5 with 1 allows us to write them over the same denominator as on the right side of Eq. 4. For i = 1 set j = 2 and vice versa. ThenEQUATIONwhere the denominator of the right side equals to |X| + |Y | because of the prerequisits 6. Now, let us choose w i such that the enumerator of Eq. 4 equals the enumerator of Eq. 5 after Eq. 7 as been applied, i.e.EQUATIONSinceEQUATIONbecause of the prerequisites 6, solving Eq. 8 for w i yieldsEQUATION)In words, this is the sum of predicted foreground and GT foreground within the subset i over the sum of predicted foreground and GT foreground in the entire image. As we have no access to the amount of GT foreground we approximate it with the predicted amount of foreground. Hence, we approximate w i with the number of predicted foreground pixels within the subset i over the total number of predicted foreground pixels. 
 
 Appendix:  


Decision: This paper proposes a strategy to correct segmentations in a smart way so that the dice score increases as quick as possible. This paper is orginal and well-written. All reviewers agreed to accept it.

Reviews: review: paper_summary: paper_summary: This paper proposes a strategy to correct segmentations in a smart way so that the dice score increases as quick as possible. For this, an ensemble of methods is used to generate a pixelwise uncertainty map, and this is then used to source patches that are potentially wrong. Among those patches, the authors are able to estimate how much would the global dice score increase if someone corrected each of them (this idea is brilliant, I think) . Therefore in test time, the model can extract the patches that would be more efficient to correct to achieve a desired quality.
main_review: main_review: Nothing to add here, I had already given this one a Strong Accept before the rebuttal, and if any, the paper has improved. The paper is indeed strong, and what is better, it has extremely interesting directions to expand, which warrants a very nice future extended work (the MELBA special issue, maybe?). Anyway, a pleasure to read.
strength_weakness: strength_weakness: - My main question is, how relevant is the correction of a few dice score points to downstream clinical applications? I am thinking of a "bad" vessel segmentation in a fundus images, and I am not sure that I would want to improve its dice, becuase that could lead me to spend time making some bigvessels a bit thicker or thinner, according to the preferences of a particular annotator. But in the end, what matters most is to get the vessel tree right, without breaking vessels, don't you think? Let us discuss this in rebuttal, I am willing to be convinced otherwise. Also, please see below a suggestion.

- Not a great concern, but I wonder if training on 512x512 for images that are 4x that size would not already lose vessels during downsampling? Also, does all the work happen on the downsampled images, or are they only downsampled for training but then the method works at native resoution, as it should? I mean, do you measure dice at native resolution upsampling the result of the model, or do you stay at 512x512 from beginning to end.
questions: questions: Nothing in particular, maybe clarify my doubts above?
paper_summary: paper_summary: This paper addresses automatic segmentation uncertainty for retinal blood vessel segmentation from fundus images. The objective is to select a minimum number of smaller patches (e.g 81x81 in 512x512 images) that could benefit from "patch review", i.e. manual correction, a time intensive task. 

To evaluate this aspect, the method is the following : 
- From an 5-ensemble of a state of the art UNet retinal vessel segmentation model, a model-based pixel level uncertainty map is obtained (pixelwise entropy) 
- It is observed that locally replacing the $k$ segmentation output patches having the highest average uncertainty with the ground truth improves segmentation quality. This is shown to be better (quite unsurprisingly) than randomly selecting patches (Fig. 2). A linear DSC gain is achieved with $k$ 

Another aspect of the paper is predicting increase in DSC performance without ground truth. An estimator of DSC proposed earlier (Li et al. 2022) is used to this end, which is shown to overestimate segmentation quality. This estimator is then calibrated to better reflect performances (Fig. 3).

The last aspect covers adapting the number of patches to be selected based on the estimator : i.e. selecting the lowest number $k$ that would beat a quality target.
main_review: main_review: This is a very well written, sound paper that proposes an ad hoc solution to the improvement of vessel structure segmentation in retinal fundus image. While not particularly original in terms of methodology, this is a nice pipeline that could readily be implemented in clinical workflows. However, positioning with respect to other label noise / segmentation uncertainty literature is almost empty, which is I think the weaker spot of this work. I however recommend acceptance considering the solid work performed.
strength_weakness: strength_weakness: Other aspects could have been covered by the paper and should be mentioned at least in the discussion : 
- The analysis of vessel like structures with the DSC is questionable, as global volumetric overlaps are typically biased towards higher volumes. There is a literature on better segmentation quality estimators, such as e.g. clDice (CVPR 2021) [1]
- The impact of the patch size is not discussed
- The impact of other epistemic uncertainty estimators beyond ensembling (e.g. MC dropout) is not discussed
- The assumption that a ground truth exists (and thus correction can be achieved by replacing with this perfect ground truth) means that there is no inter reader variability in annotating vessel structures, which I guess is not true even if I am not an expert in this field.

[1] https://github.com/jocpae/clDice
questions: questions: - limitations of the Dice score 
- existence of other epistemic uncertainty measures beyond entropy of ensembles
- mention inter reader variability and limitation of the study in this regard
paper_summary: paper_summary: Authors propose a method to manually correct for segmentation results
with minimal effort. They first present a method for selecting patches
with potentially high segmentation error, then propose a way to
estimate how much DSC increase would one have by correcting the
patches. The first part is addressed by estimating segmentation
entropy at each pixel given an ensemble of segmentation results. The
second part is addressed by estimating DSC assuming calibrated
outputs. Experiments with retinal blood vessels on fundus images were
performed and consistent improvements over DSC with the correction
procedure were reported.
main_review: main_review: It is a very good article - and my opinion remained the same after the rebuttal. 
This could be an oral presentation, however, I recommend a poster, since oral presentations should be reserved for perhaps more technically novel work.
strength_weakness: strength_weakness: 1. The title and abstract gives the impression that the proposed
   techniques are generally applicable to all medical image
   segmentation tasks. While this may still be possible, there is
   strong focus on a particular task and experiments only for that
   task are presented. Perhaps adjusting the title and the abstract
   may be useful.
2. Generation of the uncertainty maps is a computationally heavy
   mechanism. I am wondering whether one can get away by simply
   choosing areas with low contrast. Essentially, the uncertainty maps
   may be mostly highlighting low-contrast areas.
questions: questions: I think the paper is quite good. I do not see a lot to discuss in the rebuttal.

Comments: title: Revised Fig. 2
Full Response
Version with highlighted changes
Forgot to a link to a reference I mentioned
comment: Dear all,

during our internal review, we have identified a minor bug in the code that displays the results of the baseline method (random patch selection) for Fig. 2 c, d.
Instead of showing the baseline performance for the respective subgroups (well and poorly segmented images), we've accidentally shown the baseline performance for the entire test set in all panels b, c and d.

We have fixed that and find, as expected, that the baseline performance is higher for poorly segmented images than across the entire test set (panel d) and lower for the well segmented images.
However, this does neither affect the overall result nor the interpretation.
 
We have uploaded a revised version of the manuscript with the corrected figure panels.
Oh, that is a great idea! If I was able to train a model at low resolution, and then (learn to) do the corrections at high-res, patch-wise, that would be a nice thing to have. You see, a problem I almost always encounter (with high-res or 3d imaging) is that I am forced to train with patches in order to get good quality solutions. But when training with patches the "big picture" is lost to the model. I a recent project I have a large scan with an object always in the middle, I cannot feed the entire scan to the model due to OOM in the gpu, so I have to train patchwise, and then the prior info that there is a single object in the middle is lost, which is very meh.

Looking forward to read about your estimator of the clDice score, it would be a really useful tool to have. Also thanks for paying attention to my comments, even though you had no chance that I would increase my rating :)
Thank you for your helpful feedback and inputs.

We have now adjusted the title and abstract to more clearly reflect the fact that the analyses in our paper were conducted on retinal vessel segmentation. The title now reads “Efficiently correcting patch-based segmentation errors to control image-level performance in retinal images”, and the abstract was slightly modified to emphasise the segmentation task. Please also note that we highlight our application area with the keywords “Retinal Blood Vessels” and “Fundus” below the abstract. We hope this avoids any further misunderstandings.

The generation of the uncertainty maps is indeed computationally heavy during training, as it requires training multiple models. However, during test time, multiple forward passes are relatively cheap. Nevertheless, we agree in principle that it would be interesting to explore other strategies for candidate patch selection, either with different uncertainty techniques, or using low-level image statistics as the reviewer suggests. We have now added a brief discussion of the computational load of uncertainty estimation, and included the reviewer’s idea of selecting low contrast regions.
Thank you for taking the time to read our work carefully and providing constructive as well as  helpful feedback.

We agree that clDSC is a particularly useful metric for the task of vessel segmentation, as has also been suggested by reviewer N9Um. We now included clDSC and the limitations of DSC  in our discussion, and are planning to implement a clDSC estimator as an extension of this work. Nevertheless, we believe our work using DSC estimation can be a useful starting point for many, as it is generally applicable in many segmentation tasks beyond tube-like structures. 

We agree with the reviewer that a discussion of other uncertainty estimators will strengthen the paper. We used entropy of deep ensembles as one example technique, but of course, our modular framework allows replacing this step with any other approach for generating uncertainty maps. As suggested by reviewer fLiB, one could even consider candidate patch selection based on image statistics alone. We have now included a discussion of different approaches to uncertainty estimation. In addition, we have also qualitatively compared the uncertainty maps produced by our ensemble with MC dropout-generated uncertainty maps (added in App. F). Both methods resulted in very similar uncertainty maps and we would not expect the experimental results to change substantially in this case. 

For simplicity, we assumed a perfect oracle, but we agree that a perfect groundtruth without inter-rater variability may be unrealistic. However, we hypothesise that the expert performance may be fairly high and consistent in an error correction framework where attention is drawn to small and few areas. However, as a proof of concept it could for example be interesting to evaluate our method with multi-annotator data, where the oracle patch would then be provided by a different annotator than the reference annotation. Another interesting future avenue would be to actually measure and report manual performance on patches in an error correction setting. We have now included a discussion of inter-rater variability and the limitation of our evaluation in this regard.

Finally, the reviewer has commented on the choice of patch size in our pipeline. We have heuristically chosen a patch size of 81 as a tradeoff between annotation effort and field of view: we assumed that it may be useful to have a large enough patch to include some contextual information, while keeping the area as small as possible to reduce the manual annotation burden. We have now included an additional analysis (App. D) where we address the question “Is it better to review multiple smaller patches or fewer larger patches?”. We compared the increase in DSC when using different patch sizes, where we fixed the total size of the reviewed area. That is, we compared the effect of reviewing 16 patches of size 41x41, 4 patches of size 81x81, and 1 patch of size 161x161. We found that reviewing 16 patches of size 41x41 had a similar effect as reviewing 4 patches of size 81x81, while reviewing only a single large patch unsurprisingly yielded smaller DSC improvements.
Thank you for taking the time to read our paper in depth and for your thoughtful feedback.

We believe applying our  method to DSC estimation is a useful starting point with general applicability in many segmentation tasks. However, we agree that clDSC is a particularly useful metric for the task of vessel segmentation and have now included a discussion of clDSC and the limitations of DSC in our paper. We have now also started to investigate how we could extend our estimator to clDSC for an extension of this work. 

We agree that subgroup analysis is relevant and interesting here. We have now evaluated our method  in the individual subgroups and found similar results as in our original analysis. The subgroup with the lowest segmentation performance (Glaucoma) profited most from the patch review. Furthermore, we found that the pathological subgroups contained individual images where the review was particularly helpful, whereas such “outliers” did not exist for the normal subgroup. We have added these supplementary results to the appendix.

Regarding your comment about image resolution: We indeed carried out all our experiments on a 512x512 resolution. In principle, the method could also work in native resolution, using upsampled predictions and uncertainty maps, and inserting high-resolution oracle data. However, we now found that simply upsampling the results of our segmentation model to native resolution led to a DSC drop by 0.05, indicating that indeed, some information was lost in the original downsampling step. Therefore we feel that conceptually, it is more meaningful to evaluate our method in 512x512 resolution to not confound the effects of downsampling with the effects of our method. We welcome further comments on this matter, and also believe this would actually raise an interesting follow-up question for future extension: Is it useful to correct lower-resolution models with high-resolution manual reviews?

Thank you for pointing us to the MICCAI’18 paper, it is indeed very related. We have included it as related work in the introduction.
Dear reviewers, thank you very much for your thorough and helpful feedback. We are very happy that you appreciated our work and its presentation. You have raised relevant and interesting questions and concerns which we believe have improved our paper. 

We have submitted a revised version of the manuscript. In addition, we’ve uploaded marked-up version on google drive (https://drive.google.com/file/d/1ZGP9o4nQJqZZwvS9B8c4utDyVxC4cQ9Y/view?usp=sharing), where new material is marked in red. Please note that we also made a few minor modifications to keep the paper within the page limit after incorporating the reviewer comments.

We have addressed your comments in the individual posts below and look forward to a fruitful discussion.
Dear reviewer, thanks for the link! We had searched for it and assumed that was the paper you were referring to. We will post our response to your review shortly, and look forward to the discussion phase.
Sorry, while re-reading my own review I noticed that in the third detailed comment, I forgot to add the actual link to the reference I mentioned, here it is: https://doi.org/10.1007/978-3-030-00928-1_10