Evaluating Uncertainty Quantification in Medical Image Segmentation: A Multi-Dataset, Multi-Algorithm Study  
[PERSON]  
1 Department of Biocyhenetes and Biomedical Engineering, AGH University, al. Mickiewicza 30, 30-059 Krakow, Poland; [EMAIL_ADDRESS] (M.S. ); [EMAIL_ADDRESS] (Z.T.) [PERSON]  
1 Department of Biocyhenetes and Biomedical Engineering, AGH University, al. Mickiewicza 30, 30-059 Krakow, Poland; [EMAIL_ADDRESS] (M.S. ); [EMAIL_ADDRESS] (Z.T.) [PERSON]  
2 Department of Radiology, Jagiellonian University Medical College, ul. Botancirna 3, 31-503 Krakow, Poland; [EMAIL_ADDRESS] (W.W.); [EMAIL_ADDRESS] (M.R. ); [EMAIL_ADDRESS] (K.K. ); [EMAIL_ADDRESS] (P.M.)  
[PERSON]  
3 Department of Diagnostic Imaging, University Hospital, ul. Jakubowskiego 2, 30-688 Krakow, Poland; [EMAIL_ADDRESS] (I.K. ); [PERSON]PERSON]  
1 Department of Biocyhenetes and Biomedical Engineering, AGH University, al. Mickiewicza 30, 30-059 Krakow, Poland; [EMAIL_ADDRESS] (M.S. ); [EMAIL_ADDRESS] (Z.T.) Footnote 1: email: {[EMAIL_ADDRESS], [EMAIL_ADDRESS], jakob@gmail. treats all inputs, both certain and uncertain, as equally reliable. This limitation can pose challenges for informed clinical decision-making [3]. Uncertainty quantification is critical for understanding how confident a DL model is in its predictions. Several techniques can be employed to equip a DL with the ability to express uncertainty. These methods include Bayesian Neural Networks (BNNs), Ensembles, Monte Carlo (MC) Dropout, Quantile Regression (QR), and Evidential Deep Learning (EDL) [4]. While uncertainty plays a significant role, how the medical images are segmented when preparing ground truth training data is another key factor affecting a DL's output. This process, often performed by experts, can introduce bias. Even with identical architecture and data, models trained on differently segmented samples by different physicists can produce significantly different results [5]. To address this, previous research has explored using multiple segmentation approaches. This can involve collaborative segmentation, where experts reach an agreement or segment independently and then combine the results for training. Yet, previous research does not clearly indicate how and to what extent multiple annotations influence the model performance. A segmentation model configuration also plays a crucial role, particularly in the settings related to uncertainty estimation. While we understand that adjustments to model parameters will affect performance, the magnitude of this impact remains unclear. A specific example is the dropout rate in MC models [6]. How does changing this rate influence both the model's accuracy and its uncertainty estimates? Furthermore, is this effect consistent across different models and datasets? In other words, does the dropout rate that yields the best performance (or uncertainty estimates) for one model (or dataset) have any bearing on other models (or datasets)? Again, previous research utilizes different configurations, yet a universal impact of this parameter should be found. Lastly, previous research on uncertainty measurement typically relies on a single dataset focusing on a specific image type and body part. While this approach may lead to good performance on that dataset, it does not guarantee that the uncertainty estimates will generalize to other datasets. This limitation hinders their generalizability and clinical applicability. This paper explores how multiple annotations for medical image segmentation tasks affect the uncertainty of DL models. We explore this by employing three well-established DL architectures and four datasets. These datasets, labeled by multiple experts, encompass various body parts and image types with different sample sizes and dimensionality. This diverse selection allows for unbiased experiments to assess the impact of multiple annotations on model performance. Additionally, we test MC dropout models with various dropout rates to analyze the influence of dropout on uncertainty and identify the optimal rate.