Leveraging Probabilistic Segmentation Models for Improved Glaucoma Diagnosis: A Clinical Pipeline Approach

Authors: anna m wundram
paul  fischer
stephan  wunderlich
hanna  faber
lisa m koch
christian f baumgartner
wundram  fischer
wunderlich faber koch
berens  baumgartner
Venue: MIDL_io_2024_Conference
Published: 2024-01-19 13:44:11

Abstract: The accurate segmentation of the optic cup and disc in fundus images is essential for diagnostic processes such as glaucoma detection. The inherent ambiguity in locating these structures often poses a significant challenge, leading to potential misdiagnosis. To model such ambiguities, numerous probabilistic segmentation models have been proposed. In this paper, we investigate the integration of these probabilistic segmentation models into a multistage pipeline closely resembling clinical practice. Our findings indicate that leveraging the uncertainties provided by these models substantially enhances the quality of glaucoma diagnosis compared to relying on a single segmentation only.

Main paper: 
 introduction: 
 Glaucoma is a chronic eye disease in which nerve fibers gradually degenerate, leading to damage in the optic nerve head. It is the second leading cause of blindness worldwide affecting one in ten people over the age of eighty (Sevastopolsky, 2017) . Diagnosis involves the exact localization of the optic disc and cup in the fundus image. Their shape, size, and relationship to each other are crucial disease markers. Therefore, automated segmentation of the cup and disc is an important step for computer-aided diagnosis frameworks. However, delineation of those structures, in particular the optic cup, is highly challenging and subject to large uncertainties with large disagreements even among experts. ing uncertainty can enhance downstream task performance. However, relying on variance maps instead of the full distribution limits the downstream task to neural networks, as the variance must be added as a channel. Recently, Feiner et al. (2023) and Fischer et al. (2023) proposed two related approaches for propagating uncertainty arising in MR reconstruction to subsequent classification, regression, or segmentation tasks. Both these approaches directly propagate samples from the conditional distribution through the pipeline rather than relying on the variance. Similar, to those works, we propose a sampling-based uncertainty propagation approach. However, we go one step further by using uncertainty to enhance downstream performance. This is achieved by marginalizing over possible outcomes in the highly uncertain segmentation stage. In summary, our contributions are as follows:1. An interpretable pipeline for glaucoma diagnosis which incorporates clinical knowledge and closely mirrors clinicians' reasoning.2. The first application of segmentation uncertainty quantification techniques to cupand-disc segmentation in fundus images with a comparative analysis of four widely used techniques.3. A sampling-based approach for propagating uncertainty through a multi-stage pipeline as well as a strategy for leveraging this approach to enhance downstream performance.4. Introduction of RTC as a clinically motivated alternative to CDR in computer-aided glaucoma diagnosis pipelines and empirical verification of its superior performance. 
methods: 
 The proposed pipeline consists of five steps as illustrated in Fig. 1 . We deterministically extract the region of interest (ROI) containing the cup and disc to obtain the close-up images x. For the highly uncertain step of optic disc and cup segmentation, we use probabilistic segmentation to approximate the distribution p(s|x) of the segmentations s, and we produce segmentation samples s i . The samples (which capture the uncertainty) are propagated through a deterministic rim thickness extraction function g : s i → r i producing RTCs r i . Following this, a deterministic classification function f : r i → d i takes the RTCs as input and produces a predicted diagnosis d i ∈ {0, 1} for each r i . Lastly, we marginalize over the samples to obtain a final probability p(d|x) of the image being "glaucoma suspect". All of these steps will be introduced in detail below.The region of interest (ROI) for glaucoma diagnosis only includes a small area of the fundus containing the optic nerve head, in particular the optic disc and cup (see Fig. 1a ).Prior work has shown that a two-step segmentation approach consisting of ROI extraction, followed by segmentation can improve segmentation results (Kim et al., 2019; Liu et al., 2021) . Following these works we use a U-Net for cup and disc segmentation on full-view fundus images. Next, a padded quadratic bounding box is placed around the segmentations, and the resulting ROI is cropped and resized to 320 × 320 pixels. All experiments in this paper were performed with ROI images x obtained in this manner.We use the improved U-Net first employed by Kohl et al. (2018) which operates on 7 rather than 5 resolution levels and uses bilinear upsampling instead of transposed convolutions for all experiments, to ensure consistency with the Prob. U-Net (Kohl et al., 2018) and PHiSeg (Baumgartner et al., 2019) baselines described in the next section.The cup and disc segmentation step is characterized by large uncertainties and variability even among human experts. Therefore, segmentations resulting from a deterministic approach, such as a U-Net, may be insufficient. It may for example produce segmentations with a very thin rim in a subject where a thick and a thin rim are equally likely. The full distribution of possible segmentations p(s|x) matching a given image x contains valuable information for the downstream classification task. In recent years, several techniques have been proposed which allow to approximate this conditional probability distribution. In this work, we compare four such techniques: The probabilistic U-Net (Kohl et al., 2018) , PHiSeg (Baumgartner et al., 2019) , MC Dropout (Kendall et al., 2016) , and ensembles (Lakshminarayanan et al., 2017) .The probabilistic U-Net (Kohl et al., 2018) is a combination of the conditional VAE (Sohn et al., 2015) approach with a U-Net architecture. PHiSeg further extends the idea by a hierarchical latent space and was shown to provide closer approximations of p(s|x). Both techniques estimate the aleatoric uncertainty.Ensembles are implemented by training ten standard U-Nets with different random seeds (Ensemble seeds ). We additionally create an ensemble by training a U-Net for each of the 11 expert annotators in our data (Ensemble experts ). The widely used MC Dropout technique produces probabilistic segmentation samples by repeatedly predicting segmentations for the same image with dropout enabled. We use a dropout rate of 0.2 on the activation maps for training and testing. Dropout is applied to all layers except the final four segmentation layers. Ensembles and MC Dropout estimate epistemic uncertainty. We refer the reader to (Abdar et al., 2021) for definitions of aleatoric and epistemic uncertainty.We use the improved U-Net architecture first proposed in Kohl et al. (2018) for all approaches with the exception of PHiSeg. PHiSeg employs the same U-Net encoder, but requires a specific decoder. Crucially, all examined probabilistic segmentation methods allow the generation of segmentation samples s i from the estimated distribution of p(s|x).Next, we extract RTCs r i from segmentation samples s i . Rim thickness is defined as the width of the rim between the borders of the optic cup and disc (Spaeth et al., 2002; Hwang and Kim, 2012) . To compute the RTC, a beam centered at the optic cup is rotated by 360 degrees. At every half-degree interval, the points where the beam intersects with the borders of the optic disc and optic cup are determined. The rim thickness is calculated as the Euclidean distance between these two intersections. This process results in a data vector of length 720 for each segmentation sample s i . The resulting RTCs were visualized as polar plots (see Fig. 1 and 3 ). We denote the rim-thickness extraction procedure as a deterministic function g : s i → r i . A visual explanation is shown in Appendix C.The RTCs r i are classified as "glaucoma suspect" or "not glaucoma suspect". We use a logistic regression classifier since, in preliminary experiments, more powerful classifiers such as SVMs or Random Forests did not lead to improvements. To prevent overfitting, the RTC data is reduced by grouping the values into 72 bins and calculating their mean. We train an individual classifier for the RTCs obtained from each of the examined segmentation methods. The optimal decision threshold for each classifier is obtained by maximizing the Youden index (sensitivity + specificity -1) on the validation set. This results in a deterministic classifier f : r i → d i that maps each rim thickness sample to a binary diagnosis.The final probability p(d|x) of an input image x being "glaucoma suspect" is obtained by marginalizing over all possible segmentations, that is p(d|x) = p(s|x)f (g(s))ds. We approximate this integral using the Monte Carlo method with samples from the respective segmentation techniques. We use 100 samples for all methods, except for ensembles which are limited by the number of networks trained.The probability p(d|x) is a natural measure for uncertainty as it can be interpreted as a respective agreement or disagreement of the predictions resulting from the segmentation samples. In that sense it is comparable to expert disagreement. We obtain the final robust prediction of our pipeline by thresholding the above probability at 0.5. 
data: 
 We used two publicly available fundus image datasets for experiments. The Cháks . u dataset (Kumar et al., 2023) contains 1345 fundus images with cup and disc annotations by five experts for each image. Additionally, each expert also provided a diagnosis of "glaucoma suspect" or "not glaucoma suspect". The dataset also contains consensus segmentations obtained using the STAPLE algorithm (Warfield et al., 2004) . The RIGA dataset (Almazroa et al., 2018) consists of 750 fundus images with cup and disc annotations by six experts for each image. In contrast to Cháks . u it contains no diagnosis labels, and was only used for training the segmentation networks in our study. We split the training portion of the Cháks . u data into a training and validation set according to an 80/20 split. We used the official test split of the Cháks . u data for all our evaluations. Additionally, we split the RIGA dataset into a training, and a validation portion according to a 80/20 split.We trained the segmentation-based stages of our pipelines with combined RIGA and Cháks . u datasets (see Sections 2.1 & 2.2), and the classifiers using only the Cháks . u dataset which contains Glaucoma suspect labels (see Section 2.4). More details about the training and model selection can be found in Appendix A.In order to show that accounting for the uncertainty in the segmentation step leads to improved performance in downstream tasks, we included a deterministic U-Net for the cup and disc segmentation as an additional baseline. Again, we used the improved architecture proposed in Kohl et al. (2018) . Moreover, we included a classifier trained directly on the expert annotations, as well as a black box ResNet50 network trained on the ROIs of the Cháks . u dataset to get a sense of the maximum achievable performance. Values close to 1 or 0 indicate large agreement for "glaucoma suspect" or "not glaucoma suspect", respectively. Values close to 0.5 denote high disagreement.We observed that using probabilistic segmentation techniques consistently led to improvements in the downstream classification performance compared to the deterministic U-Net (Tab. 1), with Ensemble seeds achieving the highest overall AUC score.Notably, the black box ResNet50 model performed slightly worse than the best probabilistic pipeline approaches. This suggests that our interpretable approach has the potential to outperform black-box models in certain settings. We note that black box approaches may perform better in a data-richer setting (De Vente et al., 2023) .Most methods showed similar performance in the segmentation task, as indicated by the Dice scores in Table 1 . However, despite its high Dice score, the deterministic U-Net fell short on AUROC scores, indicating that accurate segmentation alone is insufficient, and that considering the entire probability distribution of the upstream task leads to improvements.In order to confirm the hypothesis that RTCs are better suited for glaucoma diagnosis than the widely used CDR, we additionally extracted the area CDR from all segmentations and trained an additional set of logistic regression classifiers on those values. We observed that the best AUROC scores were achieved with RTC, with particularly large improvements over CDR for the highly performing Ensemble seeds .We additionally calculated the Pearson's correlation coefficient (CC) between the mean expert prediction (i.e. all expert predictions averaged) and the mean pipeline prediction p(d|x) in the last column of Tab. 1. PHiSeg achieved the highest CC with Ensemble seeds also performing very well. Visual inspection of the mean expert and pipeline predictions confirmed these findings (see Fig. 2 ). This indicates that the propagated uncertainty correlates with the expert disagreement, and thus is an informative measure for prediction uncertainty. However, further improvements may be achieved in future work by specifically optimizing downstream calibration.The entropy maps and RTCs in Fig. 3 confirm that the distributions of the annotator disagreement approximately matched the estimated segmentation and RTC uncertainties. We observed that the method correctly predicted higher uncertainties in areas where the rim was obscured by blood vessels. Additional samples and methods are shown in Appendix B. 
discussion and conclusion: 
 Here, we proposed a pipeline for human-interpretable glaucoma prediction. We showed that probabilistic segmentation techniques are suitable for capturing uncertainties in the location of the cup and disc, and demonstrated an approach for propagating these uncertainties through the pipeline steps to the final prediction. Knowledge about the uncertainty adds an additional level of interpretability to the individual pipeline steps. We furthermore proposed a simple strategy for obtaining robust predictions by marginalizing over the distribution of possible segmentations, and showed that accounting for the uncertainty in this manner led to improved downstream predictions. Our analysis of different probabilistic segmentation techniques revealed that a simple random seed ensemble provided the best balance. However, PHiSeg provided the best qualitative results and correlation with expert disagreements.A limitation of our work is the sole focus on rim thickness as diagnostic feature. Future work will focus on incorporating diagnostic markers such as the color and intensity of the optic cup, and whether fundus corresponds to the right or left eye into our pipeline. These features are known to be diagnostically important and may further improve performance.  
 
 Appendix:  


Decision: All reviewers unanimously agree that the authors have effectively addressed my feedback and provided clarity on various aspects of their paper during the rebuttal phase. The paper presents a novel pipeline that incorporates uncertainty quantification into optic cup and disc segmentation in fundus images for glaucoma diagnosis. The approach is supported by thorough experimentation, and its contribution is noteworthy. Therefore, I highly recommend its acceptance.

Reviews: review: paper_summary: paper_summary: In this paper, the authors introduce a new pipeline that integrates uncertainty quantification into the segmentation of the cup and disc in fundus images for diagnosing glaucoma. The proposed pipeline comprises four main stages: Initially, a U-Net model is trained to identify the regions of interest (ROI) by segmenting the cup and disc in the global image. Subsequently, a probabilistic model is applied to these ROIs to generate multiple segmentation samples of the cup and disc. For each segmentation proposal, the rim thickness curve (RTC) is computed and input into a linear classifier for diagnosis prediction. The final diagnostic score is derived by aggregating the predictions from all segmentations.
main_review: main_review: The authors provided detailed responses to my concerns and those of other reviewers. The explanations provided, especially concerning the interpretability of the pipeline, have significantly clarified many aspects and addressed several of my previous comments.
strength_weakness: strength_weakness: - The literature review on the role of uncertainty in enhancing clinical diagnosis appears incomplete. The assertion that only a few studies have shown the benefits of integrating uncertainty might overlook significant contributions in the field, such as those focusing on probabilistic attention [1], and robustness in medical image classification[2]. It would also be beneficial to include references that clearly define "epistemic" and "aleatoric" uncertainty, to help the reader better understand these concepts that are not defined in the paper [3].
- The paper analysis of the results produced by the benchmarked methods lacks a discussion in terms of their focus on epistemic versus aleatoric uncertainty. An extended discussion in this context, potentially combined with an examination of the correlation between models and experts uncertainty, would enhance the paper's value. (e.g. Do "epistemic approaches" outperform "aleatoric ones", what could we conclude from this, ... ?)
- There is insufficient discussion regarding how individual components of the pipeline contribute to the overall performance improvement. Identifying whether the observed performance enhancements are due to the consideration of multiple diagnostic proposals per patient or solely due to the improved performance of the classifier would be insightful. A suggested approach would be to train the classifier directly on the ground truth segmentations, and then apply this trained classifier to the segmentations produced by the evaluated stochastic segmentation models. This would allow for a direct assessment of the segmentations' impact on performance (in particular when comparing ensembling vs other benchmarked methods as the classifier is not trained on the same number of samples), or alternatively, an ablation study could be conducted.
- The details provided about the experimental setup, such as the number of samples used to train the classifier, need to be more explicit (e.g., is the classifier trained on 100 samples per training image for all methods except ensembling ?).
- The paper does not adequately address the performance of the first ROI identification step and the rationale behind the choice of classifier. Additional details on these mentioned "preliminary experiments" could clarify these choices.
- The claim of interpretability is questionable, given that the proposed stochastic method might obscure which specific segmentation contributes to the final predictions. A discussion on the interpretability of stochastic models versus deterministic models could clarify this point.



[1]: Schmidt, A., Morales-Álvarez, P., & Molina, R. (2023). Probabilistic Attention Based on Gaussian Processes for Deep Multiple Instance Learning. IEEE Transactions on Neural Networks and Learning Systems.
[2]: Abdar, M., Salari, S., Qahremani, S., Lam, H. K., Karray, F., Hussain, S., ... & Nahavandi, S. (2023). UncertaintyFuseNet: robust uncertainty-aware hierarchical feature fusion model with ensemble Monte Carlo dropout for COVID-19 detection. Information Fusion, 90, 364-381.
[3]: Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., ... & Nahavandi, S. (2021). A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information fusion, 76, 243-297.
questions: questions: The authors may focus on addressing the concerns highlighted in the weaknesses section, especially the insufficient exploration of how each component within the pipeline individually contributes to the enhancement of overall performance, and a more comprehensive analysis of the results.
paper_summary: paper_summary: The authors build a diagnostic pipeline for glaucoma from retinal images that puts to use a probabilstic unet-like system into a clinically-inspired multi-step process. Since the net can produce multiple samples from a posterior distribution of the OD and the cup, those are used to propagate such uncertainty into the measurement of a relevant biomarker for detecting glaucoma. Interestingly, the popular OD/cup ratio is not the measure of choice, but another one called rim-to-disc ratio, which seems to be more valid (kumar 2019)
main_review: main_review: Not only my questions but those of other reviewers have been properly addressed. This is a solid paper that deserves to be accepted in my opinion. OpenReview still wants me to write fifty more chars...
strength_weakness: strength_weakness: - It is okay to go from images to biomarkers and then use logistic regression to reach a diagnosis, as a way of increasing transparency. However, I believe this should be done without sacrificing (much of) the accuracy that a standard CNN would provide. Maybe the authors should spend a bit of time training an end-2-end CNN that takes OD crops and returns glaucoma diagnostic to check that the AUROC is not abismally higher.
- For glaucoma analysis, a more realistic setup could be a screening-like situation, with much more presence of the non-pathological class, and using other relevant metrics. I understand that extreme class imbalance does not make up for a nice situation, but maybe if we are to transition to real-life use-cases, then this should be the main assumption. Please see below for some suggestions :)
- Although using the rim-to-od ratio is a nice addition, I also find it a missing opportunity to verify that it is actually superior to the traditional od-to-cup ratio. The cited clinical literature supporting the benefits of the former is a 2002 ophthalmology paper (22 years ago this was proposed and it has not been widely picked up, one wonders why), and a NSR from the same authors of the Cháksu dataset, and this cannot be considered strong clinical evidence I think. It would have been great to see if using the conventional OD-to-cup ratio actually is worse through the eyes of the logistic regression by the end of the pipeline.
questions: questions: * I guess that it would be great if the authors could train a cnn to output glaucoma diagnostic from the same data (resnet50 pretrained on imagenet should do, and this would be a quick experiment I believe, given there are relatively few images for training a classifier)
* Also retrain the system to use OD-Cup ratio instead of OD-rim, although I am not sure if this makes lot of sense. I am hoping the authors will clarify if that is the case, or they can push back and tell me that it would be meaningless, no problem with that :)
* If the authors were interested in pursuing the "screening setup" direction, now or in the future, please check this out: 
1) https://www.sciencedirect.com/science/article/pii/S2666914523000325 
2) https://arxiv.org/abs/2302.01738

which also involves uncertainty but in the out-of-distribution side of things: train only with gradable images, then test with ungradable -diagnostically useless images:
- Related to this, the same organizers are running a challenge about explainable glaucoma in which they use a similarly large scale dataset. In a sizeable subdataset (6k) there are also explanations of diagnostics by means of multi-label annotatations, with possible clinical reasons for the given diagnostic. In addition, there are two raters involved, and if they disagree an expert gives their opinion to disambiguate diagnostic. Considering the interest of the authors on explainability and on retinal image-based disease diagnosis, I guess this and the above datasets must be mouth-watering ;)
paper_summary: paper_summary: The authors present research on using deep learning in glaucoma diagnosis while incorporating it into physicians' workflow. The method takes into account uncertainty in the segmentation of eye components in the image. The method also employs a different metrics compared to commonly used ratio for diagnosis of the disease.
main_review: main_review: Thank you team for the update and detailed explanations for my questions. I find the added details convincing and suggest a strong accept for the conference. I encourage the authors to continue exploring this area.
strength_weakness: strength_weakness: Taking a step further and comparing propagated uncertainty, expert disagreement and actual diagnosis (may not be available) could be helpful.
It seems that propagated uncertainty from CDR as metric could also resemble expert disagreement.
questions: questions: Taking a step further and comparing propagated uncertainty, expert disagreement and actual diagnosis (may not be available) could be helpful.
It seems that propagated uncertainty from CDR as metric could also resemble expert disagreement.
paper_summary: paper_summary: The paper presents a multi-stage pipeline for enhancing glaucoma diagnosis through probabilistic segmentation of optic cup and disc in fundus images. It introduces an approach that leverages uncertainty quantification to improve the reliability of glaucoma detection, contrasting with traditional methods that may not account for inherent ambiguities. The authors demonstrate how the integration of probabilistic models and the extraction of Rim Thickness Curve (RTC) instead of the traditional Cup-to-Disc Ratio (CDR) can provide better diagnostic performance. They validate their approach with experiments on the Chaksu test set, showing improved classification accuracy and alignment with expert disagreement.
main_review: main_review: The authors have addressed my comments and clarified aspects of their paper on the rebuttal process. I would like to thank them for their efforts. The paper introduces a new pipeline that integrates uncertainty quantification into the segmentation of the optic cup and disc in fundus images for glaucoma diagnosis. The method is backed by a rigorous set of experiments and the contribution is significant. I suggest a strong accept.
strength_weakness: strength_weakness: * It can be argued that little methodological improvement has been done from Fischer et al. work on uncertainty propagation.
* Reproducibility: code is not provided.
* Limitations discussion: The authors do not discuss the limitations of their approach.
* I suggest reviewing the grammar and typos of some sentences. I’ve provided a non exhaustive list in the detailed comments section.
questions: questions: See detailed comments above. I suggest reviewing the grammar and spelling of the entire paper.

Comments: title: Thank you
Rebuttal
Rebuttal
Rebuttal
Rebuttal
comment: Thank you for your thoughtful responses to my questions and comments.
We are glad to hear that the responses were satsifactory and helped to clarify your concerns! Thanks for the suggestion with the analysis of samples with differing diagnoses. That will be an interesting addition to a potential extension of this work. 

Thanks also for the paper on aleotoric/epistemic uncertainties. We had a hunch that the distinction might not be so clear-cut in practice for a while now. Great to see it confirmed!
We are glad to hear that the responses were satsifactory and helped to clarify your concerns. We apologise for this misunderstanding. We have indeed experimented with using the same classifier trained on GT segmentations for all segmentation methods during the initial development of our method. Back then we actively decided against including this ablation, since we believe that the maximum achievable performance with each segmentation network is more informative than the performance with a fixed classifier. In the end, this is also what one would use in practice. We found that the performance with classifiers specific to the segmentation network was marginally higher than a classifier trained on the segmentation networks' outputs. However, the overall picture is similar with both evaluation strategies.
Thank you for the detailed responses to my remarks and for clearly outlining the modifications made in the revised manuscript. 

All my concerns have been satisfactorily addressed, but I would like to add an additional comment:

**R1.3**:
My remark was rather referring to the comparison between the different probabilistic segmentation techniques. I think a little ablation study using the same logistic regression classifier, trained on expert annotations, for all the probalistic segmentation approaches could have provided clearer insights into which strategy yields the highest quality samples (as the performance of the logistic regression classifier could be biased by the number of samples used for training for instance). However, following your clarification, I understand that this analysis may exceed the intended scope of your study.
Hello,

Thanks for your answers, sorry to ask for something that you already had in the paper, my mistake for reading too fast, I think I am overworking these days. Anyway, I like the paper and the responses are satisfactory I believe. As a suggestion, it would be interesting to look into the samples where a black box prediction differs from the clinically-inspired system, and see why that is the case.

By the way, next time a reviewer pushes to discuss about the difference between aleatoric and epistemic methods, and its impact, you may point them towards this 2024 ICLR oral (not mine, unfortunately):

https://arxiv.org/abs/2401.08501

where they show that for uncertainty in segmentation problems, basically in practice the epistemic vs aleatoric distinction is BS. Good luck!
We sincerely thank the reviewer for their time, valuable feedback, and positive comments. We summarized and re-structured the reviewer's points with the identifiers R4.x in order to facilitate any subsequent discussion.

All changes to the revised PDF that we uploaded are highlighted in **orange**.

> **R4.1:** It can be argued that little methodological improvement has been done from Fischer et al. work on uncertainty propagation.

*Response:* While Fischer et al. have contributed towards uncertainty propagation, **this paper is building upon that work by further examining whether uncertainty propagation can *enhance* performance**. To this end, we propose a novel approach to marginalise over the distribution of the uncertain pipeline step, and show that this approach leads to superior performance compared to a deterministic method. **We have emphasized this more in the text**. 

> **R4.2:** Reproducibility: code is not provided.

*Response:* The code was not yet ready for publication by the time of our initial submission. It is now available at https://github.com/annawundram/glaucoma-diagnosis-pipeline.

> **R4.3:** Limitations discussion: The authors do not discuss the limitations of their approach.

*Response:* **We have included a brief discussion of the limitations of our approach** in the Discussion.

> **R4.4:** I suggest reviewing the grammar and typos of some sentences

*Response:* We thank the reviewer for taking the time to put together the list of typos. **We fixed all of them in the updated PDF.** We have furthermore **thoroughly checked the revised manuscript for spelling and grammar mistakes**.

> **R4.5:** Since you are using an improved version of the UNet architecture. Why did you choose the one provided by Kohl et al. and not nnUNet?

*Response:* Our choice of architecture was motivated by our wish to use the probabilistic U-Net and PHiSeg as originally proposed by the respective authors. In order to make the results of the other techniques comparable to those methods, we chose this modified U-Net as our base architecture for all experiments. **We have clarified this in the updated PDF.**
We sincerely thank the reviewer for their time, valuable feedback, and positive comments. We summarized and re-structured the reviewer's points with the identifiers R3.x in order to facilitate any subsequent discussion.

All changes to the revised PDF that we uploaded are highlighted in **orange**.

> **R3.1:** Taking a step further and comparing propagated uncertainty, expert disagreement and actual diagnosis (may not be available) could be helpful.

*Response:* We thank the reviewer for this suggestion. Unfortunately, we haven't been able to find a dataset that contains multiple expert annotations **and** an actual (i.e. clinically confirmed) diagnosis. Unfortunately, the majority of datasets that we are aware of only contain labels such as "glaucoma suspect" or "referable glaucoma" as it is the case for example in Chaksu and AIROGS. 

> **R3.2:** It seems that propagated uncertainty from CDR as metric could also resemble expert disagreement.

*Response:* We agree with the reviewer on this point. However, by using the rim thickness as a metric, the **CDR is also implicitly contained** as the CDR can be derived from the RTC.
We would like to sincerely thank the reviewer for their time, valuable feedback, and positive comments. We summarized and re-structured the reviewer's points with the identifiers R2.x in order to facilitate any subsequent discussion.

All changes to the revised PDF that we uploaded are highlighted in **orange**.

> **R2.1:** Maybe the authors should spend a bit of time training an end-2-end CNN that takes OD crops and returns glaucoma diagnostic to check that the AUROC is not abismally higher.
> 
*Response:* We agree with the reviewer and **have therefore trained and evaluated a ResNet50 on the Chaksu optic disc crops** with the same train/val/test split as used in the paper (see new results in Table 1). The training details are reported in Appendix A. This end-to-end network reached an AUROC score of 0.884, which is slightly worse than the best probabilistic pipeline approaches. This shows that our probabilistic pipeline approach has the potential to perform on a par, or even outperform a black box approach. We acknowledge, however, that a black box model may have better performance in a data-richer setting as is for example suggested by the AIROGS challenge results. 

> **R2.2:** For glaucoma analysis, a more realistic setup could be a screening-like situation, with much more presence of the non-pathological class, and using other relevant metrics. (...) 

*Response:* As our paper aims at closing the gap to real life applications, **we agree that realistic class-imbalance is a necessary factor** to take into account when evaluating such pipelines. In our case, **we have worked with a class-imbalance of about 80/20 for non-pathological/pathological cases**. It can, however, be argued that this is not sufficient and real life scenarios might be even more imbalanced. We will further explore this in future work, potentially using the datasets the reviewer pointed out under R2.4. 

> **R2.3:** Although using the rim-to-od ratio is a nice addition, I also find it a missing opportunity to verify that it is actually superior to the traditional od-to-cup ratio.

*Response:* If we understand correctly, then this point is asking for a comparison of RTC (rim thickness curve) to CDR (cup-to-disc ratio). We would like to emphasise that **our results include such a comparison** by drawing the reviewer’s attention towards **Table 1** which includes all metrics for both RTC and CDR as well as the section titled **“RTCs outperformed CDR for glaucoma diagnosis“ in Section 3.3.** If we have misunderstood the reviewer, we would kindly ask them to clarify their concern.

> **R2.4:** If the authors were interested in pursuing the "screening setup" direction, now or in the future, please check this out (...)

*Response:* We thank the reviewer for pointing us towards these interesting datasets. We were indeed unaware of the REGAIS dataset, and the many possibilities for future research it offers. We are very interested to pursue this in future work.
We sincerely thank the reviewer for their time, valuable feedback, and positive comments. We summarized and re-structured the reviewer's points with the identifiers R1.x in order to facilitate any subsequent discussion. 

All changes to the revised PDF that we uploaded are highlighted in **orange**.

> **R1.1:** The literature review on the role of uncertainty in enhancing clinical diagnosis appears incomplete. (...) include references that clearly define "epistemic" and "aleatoric" uncertainty(...) 

*Response:* We have broadened the scope of the literature review and **included the references** pointed out. Additionally, we **refer to the suggested paper for a definition of the terms for aleatoric and epistemic uncertainty**.

> **R1.2:** The paper analysis of the results produced by the benchmarked methods lacks a discussion in terms of their focus on epistemic versus aleatoric uncertainty.

*Response:* We compared the classification performance between methods for aleatoric and epistemic uncertainties and **did not observe systematic differences between the two types of models**. Even though we believe that further analysis of the different uncertainty types is an important and fruitful research avenue, we do not discuss this further in the paper due to lack of interesting insights and space constraints. 

> **R1.3:** There is insufficient discussion regarding how individual components of the pipeline contribute to the overall performance improvement. (...) A suggested approach would be to train the classifier directly on the ground truth segmentations(...)

*Response:* We thank the reviewer for the suggestion. We would like to note that **the purpose of the individual pipeline components was not to improve performance, but rather to improve interpretability by making the pipeline similar to clinical processes**. Nevertheless, it is an interesting question how replacing steps of the pipeline by black box components affects performance. As part of our response to Reviewer 2 (see R2.1) we have added a black box ResNet50 classifier trained directly on the ROIs to Table 1. As can be seen, **replacing the majority of the pipeline by a black box does not lead to improvements.** We believe that these results subsume the suggested ablation, as it is unlikely that replacing smaller subsets of the pipeline by black boxes would lead to larger improvements. 

> **R1.4:** The details provided about the experimental setup, such as the number of samples used to train the classifier, need to be more explicit (...) 
 
*Response:* We **added a more detailed explanation of the classifier training** (see Appendix A). 

> **R1.5:** The paper does not adequately address the performance of the first ROI identification step and the rationale behind the choice of classifier. (...) 

*Response:* ROI detection is a comparatively easy step, and we have never observed the ROI detection to fail in our experiments. Note that **a failure in the ROI detection would have been reflected in the Dice scores, and the final classification score**. For the classifier, we tested several popular classification methods (Logistic Regression, SVM, Random Forests). **Due to space limitations, we included only the best performing classification method (i.e. Logistic Regression)**. 

> **R1.6:** The claim of interpretability is questionable, given that the proposed stochastic method might obscure which specific segmentation contributes to the final predictions.(...) 

*Response:* We would like to clarify that **our claim for interpretability results from setting up the diagnosis framework as a *pipeline* that mirrors clinical practice** as opposed to a black box model. We would like to further emphasize that currently all samples contribute equally to the final prediction. We don't believe there is any sacrifice in interpretability compared to non-stochastic approaches. Both, the pipeline approach as well as the ability to visualise the prediction variance at each step of the pipeline make the model more interpretable than standard black box approaches. **We have further clarified this in the Discussion**. 

> **R1.7:** In Figure 1, the description should be corrected from "majority vote" to "average".

*Response:* For binary classification, computing the average and picking the majority vote is **equivalent**. We chose the term **“majority vote” for consistency with the expert majority vote**.

> **R1.8:** An open question (...) is how the introduced variance affects the performance of downstream tasks. 

*Response:* We thank the reviewer for this interesting suggestion for future work. We plan to address this question in a potential extension.

> **R1.9:** Availability of the code

*Response:* The code is now available at https://github.com/annawundram/glaucoma-diagnosis-pipeline.

> **R1.10:** Minor points regarding wording and ordering of references. 
 
*Response:* **We incorporated these points** in our updated draft.