Structure Size as Confounder in Uncertainty Based Segmentation Quality Prediction

Authors: kai  geißler
jochen g hirsch
stefan  heldmann
hans  meine
g  hirsch
hirsch heldmann geißler
  meine
monte carlo dropout
Venue: MIDL_io_2024_Conference
Published: 2024-01-31 16:21:46

Abstract: Various uncertainty estimation methods have been proposed for deep learning-based image segmentation models. An uncertainty measure is treated useful if it can be used to accurately predict segmentation quality. Therefore, structure-wise uncertainty measures are frequently correlated with measures like the Dice score. However, it is known that the Dice score highly depends on the size of the structure of interest. It is less well-known that popular structure-wise uncertainty measures also correlate with structure size. Therefore, the structure size acts as confounding variable when trying to quantify the performance of such uncertainty measures via correlation. We investigate this for the popular uncertainty measures structure-wise epistemic uncertainty, mean pairwise Dice and volume variation coefficient based on test-time-augmentation, Monte Carlo Dropout and model ensembles. We propose to use a partial correlation coefficient to address structure size as confounding variable and arrive at lower correlation estimates which better reflect the true relationship between segmentation quality and structure-wise uncertainty.

Main paper: 
 introduction: 
 Estimation of model uncertainty in deep learning based image segmentation can be done with various proposed measures such as mean predictive entropy (Gal et al., 2017) , mutual information (Kendall and Gal, 2017) , volume variation coefficient (VVC) (Roy et al., 2018) , or mean pairwise Dice (MPD) (Roy et al., 2018) . Given a specific model input, there is also a choice of methods to produce a set of outputs, like test time augmentation (TTA) (Wang et al., 2019) , Monte Carlo dropout (MCD) (Gal and Ghahramani, 2015; Gal et al., 2017) or model ensembles (Lakshminarayanan et al., 2017) , which we will call uncertainty sources. Uncertainty measures then translate such sets of outputs into an uncertainty prediction, i.e. an estimate of the likelihood of making a wrong prediction. In order to assess an uncertainty measure, it is commonly correlated with a measure for the prediction quality. In the case of image segmentation, this is usually the Dice score, which is known to be higher in general for larger structures (Reinke et al., 2021) . If model uncertainty was also influenced by structure size, then the structure size would be a confounding variable, potentially tainting correlation estimates.Figure 1 : A proper uncertainty measure for structure segmentation quality should be well correlated with the segmentation quality. We assess the confounding effect of the structure volume on this relationship.In this work, we analyse this relationship for the task of segmenting diverse anatomical structures in whole body MRI. Our contributions are threefold:• We show empirically that the popular uncertainty measures structure-wise epistemic uncertainty, MPD and VVC all correlate notably with structure size.• To compute correlation estimates that correct for this confounding variable, we propose to use partial Spearman's rank correlation (Liu et al., 2018) and assess its changes to the correlation estimates and to the resulting ranking of uncertainty measures.• We also confirm that the average symmetric surface distance has a lower correlation with structure size and is therefore less severely affected its confounding nature.Our results are achieved in the context of domain transfer between computed tomography (CT) which is our source domain and magnetic resonance imaging (MRI) which is our target domain. A segmentation model is trained on the TotalSegmentator dataset (Wasserthal et al., 2023) , a large collection of CT images with masks for anatomical structures. During training it utilizes GIN augmentation (Ouyang et al., 2023) to allow domain transfer. It is evaluated on a subset of the MR Imaging Study within the German National Cohort Study (Bamberg et al., 2015) for which segmentation masks are manually created by radiological technologists. 
related work: 
 Different ways have been proposed to estimate the uncertainty of deep neural networks, like model ensembles (Lakshminarayanan et al., 2017) , test time augmentation (Wang et al., 2019) , using the raw model outputs (Smailagic et al., 2018) , Monte Carlo dropout (Gal and Ghahramani, 2015, 2016) or deterministic methods (Liu et al., 2020a) . In addition there are uncertainty aware models using variational autoencoders to learn the model uncertainty (Kohl et al., 2018) and models based on Dempster-Shafer theory (Huang et al., 2022) or There are also models which try to disentangle the uncertainty coming from different possible uncertainty sources (Shaw et al., 2021) . Mehta et al. (2022) conducted the QU-BraTS challenge assessing the uncertainty prediction performance of 14 participating teams and proposing a novel metric to evaluate uncertainty maps. Camarasa et al. (2021) evaluated how to best calculate voxel-wise uncertainty maps in multi-class segmentation settings.Based on sampled model outputs, there are different ways to compute uncertainties for each structure. One way is to compute the voxel-wise uncertainty of the image, such as entropy, variance or variation ratio over the samples (Gal et al., 2017) and average it over the predicted structure mask. For segmentation tasks, there are also structure level measures which can be computed directly from a set of masks, like the volume variation coefficient and the mean pairwise Dice score between the predictions (Roy et al., 2018) . Kendall and Gal (2017) introduced an uncertainty decomposition into two components: Aleatoric uncertainty captures uncertainty inherent to the data, like measurement or labeling errors, while epistemic uncertainty captures the uncertainty in the model parameters by computing the mutual information (Mukhoti and Gal, 2018) .Many publications use correlations between segmentation quality and uncertainty measures to compare different methods (Roy et al., 2018; Wang et al., 2019; Hoebel et al., 2020; Lin et al., 2022a,b; Sahlsten et al., 2023) while others report correlations on their own to assess the quality of the uncertainty measure (Hiasa et al., 2019) . Hoebel et al. (2020) found no significant correlation between uncertainty measures and structure size in their experiments on lung nodule segmentation in CT scans, computing uncertainties based on MCD and model ensembles using structure-wise mean entropy over the predicted samples, VVC and MPD. Furthermore, Wang et al. (2019) claim that the VVC is agnostic to structure size.In contrast to this, we find non-negligible correlations between the structure size and the uncertainty measures structure-wise epistemic uncertainty, VVC and MPD based on model ensembles, MCD and TTA in our experiments. 
data: 
 For model training the publicly available TotalSegmentator dataset is used. It contains 1204 CT images with 104 annotated structures (Wasserthal et al., 2023) , accounting for 27 organs, 59 bones, 10 muscles, and eight vessels.Our evaluation dataset consists of 30 selected cases out of 10828 whole body MR volumes obtained as part of the MR Imaging Study within the German National Cohort Study (GNC, 2014 (GNC, -2019) ) (Bamberg et al., 2015) from volunteers. The data were acquired on MAG-NETOM Skyra 3 T (Siemens Healthineers, syngo VD13C) systems. The MRI sequence is a two-point Dixon volumetric interpolated breath-hold examination (VIBE) with T1 weighting. Axial slices were acquired with a 320×260 in-plane matrix (resolution 1.4×1.4 mm 2 ) and a slice thickness of 3 mm. The volume consists of four acquired table positions with a total of 316 slices. The test cases are selected to span a diverse sample of the subject distribution with regard to study center, sex, age, weight and height. Details of the study population are depicted in Table 1 .Three radiological technologists annotate segmentation masks for 28 anatomical structures, comprising 8 abdominal organs (liver, spleen, kidneys, stomach, pancreas, adrenal glands), 5 thoracic organs (heart, lungs, esophagus, trachea), 12 bones (scapulas, claviculas, hips, sacrum, femurs, vertibrae L1-5, T1-12 and C3-7), 2 muscles (autochthon) and one vessel (aorta) on these cases. The number of annotated structures per structure type is shown in Table 6 in the appendix. The structure types are chosen as a subset of the TotalSegmentator structures. This variety was selected to be able to draw conclusions about anatomical structures of various shapes and sizes.A segmentation model is trained from scratch on the TotalSegmentator data using our reimplementation of the nnU-Net framework (Isensee et al., 2021) . As training configuration, we approximate the TotalSegmentator low resolution (3 mm) model. The images are resampled to a voxel size of 3×3×3 mm 3 , a patch size of 80×80×80 is used and the nnU-Net non-CT normalization is performed. The models are trained for 250,000 iterations with a batch size of 2. The model used for MCD has a dropout layer with dropout rate 0.1 after each block of convolution, normalization and non-linearity in the up-and downpath. For basic data augmentation we utilize the batch generators library (Isensee et al., 2020) .In addition all models are trained using GIN data augmentation (Ouyang et al., 2023) to allow for domain transfer between CT and MR images. In GIN, a convolutional neural network (CNN) applies a random non-linear intensity value transformation on the training patches. Therefore, the model needs to focus more on the shape and less on the intensity values of structures and generalizes better to different imaging modalities and contrasts. We re-implement GIN augmentation. In our experiments, GIN augmentation is applied with a probability of 0.9 to each patch. The CNN for the augmentation has 4 layers with 2 channels each and uses ReLU activations. In each layer of the GIN augmentation network a random 1×1×1 convolution is sampled. As we train our models on a low resolution we decided to use only 1×1×1 convolutions to avoid too much smoothing that could arise from larger filters.We asses three alternative uncertainty sources to predict a set of different predictions for the same input image: model ensembles, TTA and MCD. For model ensembles one trains a set of deep learning models to obtain models with different learned weights for the prediction task. In MCD this process is approximated by training only one model with dropout layers (Srivastava et al., 2014) ; in our experiments, we use spatial dropout (Tompson et al., 2015) which is more suited for convolutional neural networks. During inference, the dropout layers are then activated to use the model as Bayesian neural network that approximates the model posterior with different predictions for each dropout layer configuration. In TTA, there is only one fixed model, but for each inference pass the input image is augmented with either reversible geometric augmentations or intensity transformations to obtain a set of different predictions for the same input. In our experiments 3D rotations, scaling, additive and multiplicative brightness, contrast transformation and gamma transformation are used. These are inspired by the data augmentations used in the batch generators library.The epistemic uncertainty (Kendall and Gal, 2017; Mukhoti and Gal, 2018) is first computed for each voxel by Equation 1, where p ct (x) is the predicted probability for class c, voxel x and sample t. T is the number of samples used for TTA, MCD and model ensembles, which is set to 10 in our experiments.EQUATIONIn order to then get to a structure-wise epistemic uncertainty, we average the epistemic uncertainty over all voxels which are predicted as this structure.The MPD (Roy et al., 2018) for class c is computed as the mean of the pairwise Dice scores of all predicted segmentation masks (Eq. 2). m ct is the binary mask for class c and sample t.MPD c = 2 T (T -1) i>j Dice(m ci , m cj ) (2)Finally, the VVC (Roy et al., 2018) for class c is computed as the variance of the predicted structure volumes divided by their mean (Eq. 3).VVC c = Var t (vol(m ct )) / E t (vol(m ct ))(3)When computing the correlation between two variables X and Y, which are both highly correlated with a third variable Z, the correlation between X and Y gets tainted by the confounding nature of Z and potentially overestimated. In our case X and Y are the segmentation quality and structure-wise uncertainty while the potential confounder Z is the structure size. Partial correlation coefficients allow to remove the confounding effect of Z when assessing the correlation between X and Y, providing a better estimate of their actual relationship.As we do not observe a linear relationship between our segmentation quality metrics and uncertainty measures, we cannot use partial Pearson's correlation for our purpose, but have to rely on rank based correlations. Kendall (1942) defined a partial correlation based on Spearman's rank correlation, but stated that it is hard to theoretically justify. It was also criticized because it is nonzero even under conditional independence of the correlated variables given the controlling variable (Korn, 1984) . Therefore we propose to use the partial Spearman's rank correlation defined by Liu et al. (2018) which they base on probability scale residuals. To compute it we use the R package PResiduals (Liu et al., 2020b) .  
evaluation across all structures: 
 At first, we consider Spearman's correlation between structure volume and uncertainty, as well as between structure volume and segmentation quality. The results across all structures and patients are presented in Table 2 . We observe strong correlation for all tested uncertainty measures and for the Dice score, which is contrary to the observations of Hoebel et al. (2020) who observed no significant correlations between structure volume and uncertainty.For the average symmetric surface distance (ASSD), we observe only weaker correlations.The correlation between uncertainty measures and segmentation quality is shown in Table 3 . We observe moderate to high correlations in all cases, ranging from 0.520 to 0.958, again higher when using the Dice score as segmentation quality measure.As we observed before, there is a high correlation between both the segmentation quality and structure volume as well as uncertainty and structure volume. Therefore, one can see the structure volume acting as confounding variable between segmentation quality and uncertainty. To correct for this, we use the partial Spearman's rank correlation between segmentation quality and uncertainty, accounting for the structure volume. It removes the influence of structure volume from the correlation between segmentation quality and uncertainty. We show the results in Table 4 . One can observe that the correlations drop in all cases, sometimes considerably by up to 0.260. We can also compare Table 3 and Table 4 to check if the ranking of different uncertainty methods (column wise) or uncertainty measures (row wise) changes when switching from correlation to partial correlation. This Table 4 : Partial correlation between uncertainty measures and segmentation quality controlling for structure volume. Differences to regular correlation (Table 3 ) are displayed in gray, measures that rank better (worse) within their column are marked with △ (▽) and uncertainty sources that changed rank within their row with ▲(▼).Bold values mark the best method/measure combination for each segmentation quality metric. is important, because it means that we potentially arrive at different conclusions about the utility of uncertainty methods/measures when we assess them either with the correlation or partial correlation. We observe that the ranking of different methods is relatively stable (only one column changes rankings) while for the different measures the rankings differ more (all three rows changing for Dice and one row changing for ASSD).From these observations we can conclude that the structure volume has a strong confounding effect on the correlation between uncertainty measures and the Dice score and a weak confounding effect on the correlation between uncertainty measures and ASSD. The partial correlation allows to address this issue and arrive at correlation estimates less tainted by this confounding variable.In the previous section we observed a strong correlation between structure-wise uncertainty measures and structure volume when computing it across a wide variety of structure types. All these structure types differ in their typical volume (cf. Table 6 in the appendix), which is why the structure type itself could be the confounding variable, with the structure volume being only an intermediate dependent variable on that. So the remaining question is if the structure volume is the true confounding variable or if it is actually the structure type.Considering correlation between structure size and uncertainty or segmentation quality, the median, 1 st and 3 rd quartiles over all 28 structure types are shown in Table 5 . The correlations between structure volume and uncertainty are weaker when evaluated per structure type, but still many structure types show correlations that deviate considerably from 0.When correlating the uncertainty measures with the Dice score per structure type, the medians of the distribution of correlations move towards 0 in all nine cases we are assessing when switching from correlation to partial correlation. The lower and upper quartiles also move towards 0 in eight of nine cases each. For the ASSD the medians and quartiles sometimes rise and sometimes fall when switching from correlation to partial correlation. Figure 2 in the appendix summarizes the distribution over correlations and partial correlations per structure type and Figure 3 and Figure 4 in the appendix provide all the individual correlations. This shows that the confounding nature of structure volume is still present when assessing individual structure types and the partial correlation should be used. When correlating with the ASSD this effect is much less severe. This highlights its usefulness to assess the quality of uncertainty measures, as it is less effected by the confounding nature of structure volume.To test if partial correlation also leads to consistent reductions when considering structure types individually, we perform a one-sided Wilcoxon signed-rank test over all types. The threshold for significance is set to p < 0.0028 based on Bonferroni correction to achieve a family wise error of 0.05. We find significance for TTA using either MPD or VVC as uncertainty measure and Dice score as segmentation quality metric. 
conclusion: 
 We evaluated the influence of structure volume as confounding variable on the correlation between model uncertainty and segmentation quality in medical image segmentation. It was confirmed that the Dice score has a strong correlation with structure volume, and in contrast to prior work, we also found model uncertainty to be strongly correlated with structure volume. This effect was evaluated both across various structure types as well individually per structure type. It appears stronger in the first case, but is still visible per structure type.To counteract this issue, we propose to use the partial correlation coefficient when correlating segmentation quality and model uncertainty, which removes the confounding effect of the structure volume from the correlation estimate. In addition, we observe that the average symmetric surface distance suffers less from this issue, as it has only a very weak correlation with structure volume. This makes it a suitable segmentation quality measure that can be used in addition to the Dice score to assess the performance of uncertainty measures.  
 
 Appendix:  


Decision: The paper proposes to investigate structure size as a confounder variable in utilizing uncertainty estimates for segmentation quality prediction. It tackles an important research problem. 
During the rebuttal phase, the authors adequately addressed many of the concerns raised, which helped in improving the quality of the paper.

Reviews: review: paper_summary: paper_summary: The authors notice that the dice score is not independent of the size of the structure to be segmented, this is, that a model could achiveve lower dice scores just becasue the foreground is small and not because it is bad at segmenting. Then, since we might agree that uncertainty measures should in some way be inversely correlated to segmentation accuracy, it might make sense if structure size is also a confounder for uncertainty quality, in the same way as it is for segmentation.
main_review: main_review: As I said, the paper is interesting and has been somehow improved during rebuttal, although the problem of ignoring false positives is still there and the lack of public python implementation hinders the adoption of the metric that is robust to confounding. Borderline acceptance sounds like a reasonable final rating, I would not be bothered if the paper is accepted but I am not going to fight for it.
strength_weakness: strength_weakness: - Dice score measures the overlap of a prediction and the manual ground-truth. On the other hand the authors compute voxel-wise uncertainty maps, and then average over voxels predicted as the structure they are analyzing. I wonder if we are not neglecting the "False Negatives"? I mean, what happens with pixels where the model is very uncertain but predicts background and not structure? We don't care about those?
- Figure 1 does a very poor job in motivating the paper. The main idea is nice, but then this figure has lots of white space, some graphs with a very small fontsize and a caption that does not inform much about what we are seeing here. I find this to be a missed opportunity to engage the reader, by showing an example of a structure that is large or small, and the impact of that into uncertainties being judged as bad when they are actually good.
- I find the use of the partial spearman correlation coefficient a bit esoteric (and it uses R, which is a bit out of our world in medical computer vision). May the authors clarify why the SRCC does not work for them? Did they try and they did not see meaningful results?
- "we propose to use the partial correlation coefficient that controls for the structure volume when correlation segmentation quality and model uncertainty." I don't understand this sentence.
questions: questions: - Even if the research question is interesting and meaningful, I end the paper not having very clear what I should do. Like, use Hausdorff distance instead of Dice? What is the recommendation that the reader should take home? I feel this should be emphasized more in the text, and also in the abstract.
- Please please please improve Fig. 1
paper_summary: paper_summary: The paper proposes to investigate structure size as a confounder variable in utilizing uncertainty estimates for segmentation quality prediction. To correct for this, they propose to use partial Spearman’s rank correlation. The experimental results indicate that there is a strong correlation between uncertainty, dice, and structure size, while that is not the case for distance-based segmentation quality measure.
main_review: main_review: The paper tackles an important research problem. During the rebuttal phase, the authors adequately addressed many of the concerns raised, which helped in improving the quality of the paper. Overall, I am happy to accept the paper in its current form.
strength_weakness: strength_weakness: * Missing citations in the relevant work section [1] [2] [3]. 
* It took a while to understand that the MR data was only utilised for testing purposes. Maybe the authors want to clarify this further.
* It is not clear what bold values in Table 3 and Table 4 represent.
* Overall, I see the benefit of the performed analysis and results showing that uncertainty measures and dice scores correlate with structure size, and there is a strong correlation between uncertainty and dice scores. However, the conclusion of the paper is not clear to me. Do authors recommend using ASSD instead of Dice (and partial correlation instead of correlation) to measure the segmentation performance when developing uncertainty measures in the future? 
* The authors may want to write the conclusion after each experimental setting with a bit more clarity and motivate the necessity of the next set of experiments better. For example, at the end of Section 4.1, it is not clear what the conclusion is and how it leads to experiments in Section 4.2.
* The necessity of different (filled and empty) arrows in Table 4 is not clear. Why the analysis of how ranks differ between Table 4 and Table 5 is necessary? What do different ranks in these two tables signify?  
*  Also, it would be beneficial to provide graphs (per structure for each of the correlations reported in the paper) in the appendix, as it would be easy to visualize this relationship and see the per structure effect of them. 


[1] Mehta, R., Filos, A., Baid, U., Sako, C., McKinley, R., Rebsamen, M., Dätwyler, K., Meier, R., Radojewski, P., Murugesan, G.K. and Nalawade, S., 2022. QU-BraTS: MICCAI BraTS 2020 challenge on quantifying uncertainty in brain tumor segmentation-analysis of ranking scores and benchmarking results. The journal of machine learning for biomedical imaging, 2022.

[2] Camarasa, R., Bos, D., Hendrikse, J., Nederkoorn, P., Kooi, M.E., van der Lugt, A. and de Bruijne, M., 2021. A quantitative comparison of epistemic uncertainty maps applied to multi-class segmentation. arXiv preprint arXiv:2109.10702.

[3] Shaw, R., Sudre, C.H., Ourselin, S., Cardoso, M.J. and Pemberton, H.G., 2021. A Heteroscedastic Uncertainty Model for Decoupling Sources of MRI Image Quality. Machine Learning for Biomedical Imaging, 1(MIDL 2020 special issue), pp.1-23.
questions: questions: Please address concerns raised in the weakness and detailed comment section.
paper_summary: paper_summary: This paper investigates the interesting point for the popular uncertainty measures structure-wise epistemic uncertainty, mean pairwise Dice and volume variation coefficient based on test-time-augmentation, Monte Carlo Dropout and model ensembles. They propose to use a partial correlation coefficient to address structure size as confounding variable and arrive at lower correlation estimates. The research in this paper has really aroused my interest. However, the uncertainty estimation methods in this paper are not sufficiently investigated and the experiments are insufficient. If the author can better respond to the shortcomings, I may change the grade.
main_review: main_review: Thanks to the author's positive improvements, I believe several issues that were concerning to me have been addressed. 
They extended the related works.
They adapted Figure 1.
But I still reserve the opinion that some methods should be compared to show that the authors' findings apply to a variety of models and measures. 
As a result, I've decided to adjust my rating accordingly.
strength_weakness: strength_weakness: 1. Contributions to this article are not listed in the intro.
2. The uncertainty estimation methods compared in this paper are insufficient.
3. The output in Figure 1 requires a clear finding and conclusion.
4. Insufficient research on uncertainty estimation.
5. How about transformer and SAM?
6. More indicators on uncertainty estimation, such as ECE and UEO[1], are not introduced.

[1] Assessing reliability and challenges of uncertainty estimations for medical image segmentation
questions: questions: See Weaknesses and detailed Comments.

Comments: title: Reponse to Rebuttal
Initial response
We thank you for your valuable feedback and adressed it in our draft as follows:
We thank you for your valuable feedback and adressed it in our draft as follows:
We thank you for your valuable feedback and adressed it in our draft as follows:
comment: Ok, I had a look at the revision you made available below and it seems the paper improved noticeably. Sorry for asking a silly question, and thanks for clarifying.

I still think that the impact of your work would be greater if you were able to release a python version of the partial SRCC, and I find the answer about how false negatives are disregarded a bit like "we do not have a solution for that problem, sorry", which is not super-satisfying. It seems to me that the fair thing to do here is to upgrade my rating to borderline-accept. Good luck!
Overall, I am really happy with the rebuttal provided by the authors. It greatly improved the paper and made the reading experience much better. 

As per my understanding, the conclusion made by the authors is that Dice is greatly affected by the structure size, and as such, the correlation between dice and uncertainty is also affected by it. In this case, using the Partial correlation coefficient would somewhat circumvent the issue. Distance-based ASSD metric is less affected by the size of the structure, and as such, the correlation between ASSD and uncertainty is almost similar irrespective of the chosen correlation calculation method. 

Considering all this, I would be happy to provide acceptance of the paper.
Dear Reviewer, please find the changes between the initial submission and our current draft here: https://owncloud.fraunhofer.de/index.php/s/selaS0qOZz4Xi5G
Could you please provide a PDF where all the changes are highlighted in a different colour? In the current state it is really difficult to see where the changes are made without access to the previous version of the paper.
1. We listed our contributions with bullet points for clarity.
2. We adapted Figure 1 to better represent our main message, as you were not the only reviewer pointing out that it needed to be improved.
3. Our paper is not about finding the best uncertainty method, but we want to point out the methodological pitfall of structure size being a confounding variable in this setting (which is hopefully also more clear from the better Figure 1 and its new caption).
4. We extended the related work section with more uncertainty estimation methods.
5. As already written above, our contribution is not to assess and compare as many models or uncertainty measures as possible, and we hope this becomes much more clear now in our revised paper. We have merely selected a number of uncertainty estimation methods to show that the problem we highlight and address does not only affect a single one, but that our findings hold across a variety of models and measures. We expect other base networks to behave similarly as the methods we looked at, as they are based on the creation of a set of predictions from model ensembles, TTA or MCD and deriving uncertainties from these predictions, but we did not test all base networks from the literature yet.
6. While the indicators you proposed (expected calibration error (ECE) and uncertainty-error overlap (UEO)) are valuable measures for assessing uncertainty estimation methods, they are not directly suited to evaluate structure-wise uncertainty predictions, as they act on the uncertainty maps and not on the structure-wise uncertainties. As we are looking at the latter in this work, we did not include an evaluation of these metrics.
> Missing citations in the relevant work section [1] [2] [3]
>
> It took a while to understand that the MR data was only utilised for testing purposes. Maybe the authors want to clarify this further.
>
> It is not clear what bold values in Table 3 and Table 4 represent.
>
> The authors may want to write the conclusion after each experimental setting with a bit more clarity and motivate the necessity of the next set of experiments better. For example, at the end of Section 4.1, it is not clear what the conclusion is and how it leads to experiments in Section 4.2.

We incorporated the missing literature into our paper and clarified the parts and sections you pointed out as hard to understand or suboptimally phrased, cf. the description of which data is used for training and evaluation and the conclusion of section 4.1 and motivation for section 4.2.

---

> The necessity of different (filled and empty) arrows in Table 4 is not clear. Why the analysis of how ranks differ between Table 4 and Table 5 is necessary? What do different ranks in these two tables signify? 

Regarding the analysis of rankings in Table 4 and 5: Oftentimes, an assessment like the correlation of uncertainty with segmentation quality is used to decide which uncertainty method works best. If we switched from the "normal" correlation to partial correlation and these rankings stayed the same, then using the partial correlation would not influence the conclusions drawn from assessment based on regular correlation measures. However, we observed that these rankings do change, which shows that the conclusions drawn from partial correlations can be indeed different ones (and we argue that they are also better ones, as they remove the influence of the counfounding variable).

---

> It would be great if the authors could clarify what they mean by the following sentence: "The number of annotated structures per structure type ranges from 6 (right adrenal gland) to 26 (liver), with a mean of 19."

This sentence was referring the fact how many annotation masks were available for the different structure types, as the annotation process is progressing and the number of masks differs for all the structures. We now removed this sentence and included a table for the number of masks in the appendix.
> Dice score measures the overlap of a prediction and the manual ground-truth. On the other hand the authors compute voxel-wise uncertainty maps, and then average over voxels predicted as the structure they are analyzing. I wonder if we are not neglecting the "False Negatives"? I mean, what happens with pixels where the model is very uncertain but predicts background and not structure? We don't care about those?

Regarding the averaging of voxels over the structure mask for structure-wise epistemic uncertainty: We are aware and agree that this approach has the weakness you stated. We tried dilated masks to catch more potential false negatives, but this lead to similar results and introduces a hyperparameter of the dilation kernel size, which is why we decided against it for the sake of simplicity. As our image volumes are very large (whole body examinations) and each structure is only a very small part of it, we did not find or develop a better strategy to cope with it yet. Nevertheless this problem is to some degree adressed by the MPD and VVC as they act directly on the sets of predicted masks.

---

> Figure 1 does a very poor job in motivating the paper. The main idea is nice, but then this figure has lots of white space, some graphs with a very small fontsize and a caption that does not inform much about what we are seeing here. I find this to be a missed opportunity to engage the reader, by showing an example of a structure that is large or small, and the impact of that into uncertainties being judged as bad when they are actually good.

We improved figure 1 (and revamped its caption), which was rightfully and thankfully pointed out by multiple reviewers not to have conveyed our main contribution before.

---

> I find the use of the partial spearman correlation coefficient a bit esoteric (and it uses R, which is a bit out of our world in medical computer vision). May the authors clarify why the SRCC does not work for them? Did they try and they did not see meaningful results?

Why Spearman's rank correlation coefficient (SRCC) is not sufficient is meant to be the main message of our paper: As the segmentation quality and structure-wise uncertainty both heavily correlate with the structure size, the correlation between segmentation quality and structure-wise uncertainty gets overestimated and captures strongly the relationship of both variables with the structure size and not with each other. Therefore we propose to use partial correlation coefficients, which reduce the influence of the confounding variable on the correlation estimate between the two variables of interest. We are using the R implementation as it was the only implementation of the partial Spearman correlation introduced by Liu et al. (2018), but as our research motivates its use in this context it could make sense to re-implement it in other frameworks of course. Using the official implementation for now prevented us from making mistakes when implementing it ourselves.

---

> Even if the research question is interesting and meaningful, I end the paper not having very clear what I should do. Like, use Hausdorff distance instead of Dice? What is the recommendation that the reader should take home? I feel this should be emphasized more in the text, and also in the abstract.

Based on your feedback, we concluded that our contributions did not become clear enough before, and we improved the draft in many places to better reflect our main message and recommendations to the reader.