QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation – Analysis of Ranking Scores and Benchmarking Results
Authors: Mehta Raghav , Filos Angelos , Baid Ujjwal , Sako Chiharu , McKinley Richard , Rebsamen Michael , Dätwyler Katrin , Meier Raphael , Radojewski Piotr , Murugesan Gowtham Krishnan , Nalawade Sahil , Ganesh Chandan , Wagner Ben , Yu Fang F. , Fei Baowei , Madhuranthakam Ananth J. , Maldjian Joseph A. , Daza Laura , Gómez Catalina , Arbeláez Pablo , Dai Chengliang , Wang Shuo , Reynaud Hadrien , Mo Yuanhan , Angelini Elsa , Guo Yike , Bai Wenjia , Banerjee Subhashis , Pei Linmin , AK Murat , Rosas-González Sarahi , Zemmoura Ilyess , Tauber Clovis , Vu Minh H. , Nyholm Tufve , Löfstedt Tommy , Ballestar Laura Mora , Vilaplana Veronica , McHugh Hugh , Maso Talou Gonzalo , Wang Alan , Patel Jay , Chang Ken , Hoebel Katharina , Gidwani Mishka , Arun Nishanth , Gupta Sharut , Aggarwal Mehak , Singh Praveer , Gerstner Elizabeth R. , Kalpathy-Cramer Jayashree , Boutry Nicolas , Huard Alexis , Vidyaratne Lasitha , Rahman Md Monibor , Iftekharuddin Khan M. , Chazalon Joseph , Puybareau Elodie , Tochon Guillaume , Ma Jun , Cabezas Mariano , Llado Xavier , Oliver Arnau , Valencia Liliana , Valverde Sergi , Amian Mehdi , Soltaninejad Mohammadreza , Myronenko Andriy , Hatamizadeh Ali , Feng Xue , Dou Quan , Tustison Nicholas , Meyer Craig , Shah Nisarg A. , Talbar Sanjay , Weber Marc-André , Mahajan Abhishek , Jakab Andras , Wiest Roland , Fathallah-Shaykh Hassan M. , Nazeri Arash , Milchenko Mikhail , Marcus Daniel , Kotrotsou Aikaterini , Colen Rivka , Freymann John , Kirby Justin , Davatzikos Christos , Menze Bjoern , Bakas Spyridon , Gal Yarin , Arbel Tal 
Published: May 2023

Deep learning (DL) models have provided the state-of-the-art performance in a wide variety of medical imaging benchmarking challenges, including the Brain Tumor Segmentation (BraTS) challenges. However, the task of focal pathology multi-compartment segmentation (e.g., tumor and lesion sub-regions) is particularly challenging, and potential errors hinder the translation of DL models into clinical workflows. Quantifying the reliability of DL model predictions in the form of uncertainties, could enable clinical review of the most uncertain regions, thereby building trust and paving the way towards clinical translation. Recently, a number of uncertainty estimation methods have been introduced for DL medical image segmentation tasks. Developing scores to evaluate and compare the performance of uncertainty measures will assist the end-user in making more informed decisions. In this study, we explore and evaluate a score developed during the BraTS 2019-2020 task on uncertainty quantification (QU-BraTS), and designed to assess and rank uncertainty estimates for brain tumor multi-compartment segmentation. This score (1) rewards uncertainty estimates that produce high confidence in correct assertions, and those that assign low confidence levels at incorrect assertions, and (2) penalizes uncertainty measures that lead to a higher percentages of under-confident correct assertions. We further benchmark the segmentation uncertainties generated by 14 independent participating teams of QU-BraTS 2020, all of which also participated in the main BraTS segmentation task. Overall, our findings confirm the importance and complementary value that uncertainty estimates provide to segmentation algorithms, and hence highlight the need for uncertainty quantification in medical image analyses. Our evaluation code is made publicly available at <a href='https://github.com/RagMeh11/QU-BraTS'>https://github.com/RagMeh11/QU-BraTS</a>