QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Results
Creators: name: Mehta, Raghav
name: Filos, Angelos
name: Baid, Ujjwal
name: Sako, Chiharu
name: McKinley, Richard
name: Rebsamen, Michael
name: Datwyler, Katrin
name: Meier, Raphael
name: Radojewski, Piotr
name: Murugesan, Gowtham Krishnan
name: Nalawade, Sahil
name: Ganesh, Chandan
name: Wagner, Ben
name: Yu, Fang F.
name: Fei, Baowei
name: Madhuranthakam, Ananth J.
name: Maldjian, Joseph A.
name: Daza, Laura
name: Gomez, Catalina
name: Arbelaez, Pablo
name: Dai, Chengliang
name: Wang, Shuo
name: Reynaud, Hadrien
name: Mo, Yuan-han
name: Angelini, Elsa
name: Guo, Yike
name: Bai, Wenjia
name: Banerjee, Subhashis
name: Pei, Lin-min
name: AK, Murat
name: Rosas-Gonzalez, Sarahi
name: Zemmoura, Ilyess
name: Tauber, Clovis
name: Vu, Minh H.
name: Nyholm, Tufve
name: Lofstedt, Tommy
name: Ballestar, Laura Mora
name: Vilaplana, Veronica
name: McHugh, Hugh
name: Talou, Gonzalo Maso
name: Wang, Alan
name: Patel, Jay
name: Chang, Ken
name: Hoebel, Katharina
name: Gidwani, Mishka
name: Arun, Nishanth
name: Gupta, Sharut
name: Aggarwal, Mehak
name: Singh, Praveer
name: Gerstner, Elizabeth R.
name: Kalpathy-Cramer, Jayashree
name: Boutry, Nicolas
name: Huard, Alexis
name: Vidyaratne, Lasitha
name: Rahman, Md Monibor
name: Iftekharuddin, Khan M.
name: Chazalon, Joseph
name: Puybareau, Elodie
name: Tochon, Guillaume
name: Ma, Jun
name: Cabezas, Mariano
name: Llado, Xavier
name: Oliver, Arnau
name: Valencia, Liliana
name: Valverde, Sergi
name: Amian, Mehdi
name: Soltaninejad, Mohammadreza
name: Myronenko, Andriy
name: Hatamizadeh, Ali
name: Feng, Xue
name: Dou, Quan
name: Tustison, Nicholas
name: Meyer, Craig
name: Shah, Nisarg A.
name: Talbar, Sanjay
name: Weber, Marc-Andre
name: Mahajan, Abhishek
name: Jakab, Andras
name: Wiest, Roland
name: Fathallah-Shaykh, Hassan M.
name: Nazeri, Arash
name: Milchenko1, Mikhail
name: Marcus, Daniel
name: Kotrotsou, Aikaterini
name: Colen, Rivka
name: Freymann, John
name: Kirby, Justin
name: Davatzikos, Christos
name: Menze, Bjoern
name: Bakas, Spyridon
name: Gal, Yarin
name: Arbel, Tal
Publisher: name: arXiv
Year: 2021
Subjects: subject: Image and Video Processing (eess.IV)
subject: Computer Vision and Pattern Recognition (cs.CV)
subject: Machine Learning (cs.LG)
subject: FOS: Electrical engineering, electronic engineering, information engineering
subject: FOS: Electrical engineering, electronic engineering, information engineering
subject: FOS: Computer and information sciences
subject: FOS: Computer and information sciences

description: Deep learning (DL) models have provided state-of-the-art performance in various medical imaging benchmarking challenges, including the Brain Tumor Segmentation (BraTS) challenges. However, the task of focal pathology multi-compartment segmentation (e.g., tumor and lesion sub-regions) is particularly challenging, and potential errors hinder translating DL models into clinical workflows. Quantifying the reliability of DL model predictions in the form of uncertainties could enable clinical review of the most uncertain regions, thereby building trust and paving the way toward clinical translation. Several uncertainty estimation methods have recently been introduced for DL medical image segmentation tasks. Developing scores to evaluate and compare the performance of uncertainty measures will assist the end-user in making more informed decisions. In this study, we explore and evaluate a score developed during the BraTS 2019 and BraTS 2020 task on uncertainty quantification (QU-BraTS) and designed to assess and rank uncertainty estimates for brain tumor multi-compartment segmentation. This score (1) rewards uncertainty estimates that produce high confidence in correct assertions and those that assign low confidence levels at incorrect assertions, and (2) penalizes uncertainty measures that lead to a higher percentage of under-confident correct assertions. We further benchmark the segmentation uncertainties generated by 14 independent participating teams of QU-BraTS 2020, all of which also participated in the main BraTS segmentation task. Overall, our findings confirm the importance and complementary value that uncertainty estimates provide to segmentation algorithms, highlighting the need for uncertainty quantification in medical image analyses. Finally, in favor of transparency and reproducibility, our evaluation code is made publicly available at: https://github.com/RagMeh11/QU-BraTS.
description: Accepted for publication at the Journal of Machine Learning for Biomedical Imaging (MELBA): https://www.melba-journal.org/papers/2022:026.html