Title: Uncertainty quantification in clinical settings: A retinal fundus screening study and benchmarking
Primary area: datasets and benchmarks
Keywords: Responsible AI, Trustworthy AI, Uncertainty quantification, Computer-aided diagnosis, Benchmarking, clinical validation, Retinal fundus imaging

Abstract: We offer the most extensive benchmark for uncertainty quantification (UQ) in retinal AI screening, methodically assessing six well-known UQ techniques in three main diseases: glaucoma (114K+ images), age-related macular degeneration (28K+ images), and diabetic retinopathy (100K+ images). Our benchmark comprises two Vision Transformer topologies, standardized train/test/calibration splits, and evaluation on both public datasets and actual clinical data from a local hospital.
Results indicate that although UQ is helpful, gains are method- and disease-dependent, and we are still a long way from the ideal clinical target-risk operation. Coverage drastically decreases as risk limits increase, and no single approach is consistently dependable in all contexts.
While neither method consistently outperforms the others, Deep Ensembles and Test-Time Augmentation (TTA) are the two practical UQ approaches that most frequently enhance selective prediction and/or calibration. Conformal Prediction (CP) serves as a must-have safety rail, ensuring alignment between nominal and observed coverage. However, no method can reliably achieve the 2% target risk required for autonomous screening without sacrificing coverage. These findings highlight the need for more robust UQ methods, both for in-distribution scenarios and under domain shifts (out-of-distribution), as well as improved mechanisms for capturing disagreements and implementing policy-aware thresholding in human-in-the-loop workflows. To facilitate progress in this field, we release our benchmark, which includes standardized splits, trained model checkpoints, code, and an online demo for interactive exploration, thereby providing a reference for future UQ research in ophthalmic AI screening.