Understanding metric-related pitfalls in image analysis validation
Creators: name: Reinke, Annika
name: Tizabi, Minu D.
name: Baumgartner, Michael
name: Eisenmann, Matthias
name: Heckmann-Nötzel, Doreen
name: Kavur, A. Emre
name: Rädsch, Tim
name: Sudre, Carole H.
name: Acion, Laura
name: Antonelli, Michela
name: Arbel, Tal
name: Bakas, Spyridon
name: Benis, Arriel
name: Blaschko, Matthew
name: Buettner, Florian
name: Cardoso, M. Jorge
name: Cheplygina, Veronika
name: Chen, Jianxu
name: Christodoulou, Evangelia
name: Cimini, Beth A.
name: Collins, Gary S.
name: Farahani, Keyvan
name: Ferrer, Luciana
name: Galdran, Adrian
name: van Ginneken, Bram
name: Glocker, Ben
name: Godau, Patrick
name: Haase, Robert
name: Hashimoto, Daniel A.
name: Hoffman, Michael M.
name: Huisman, Merel
name: Isensee, Fabian
name: Jannin, Pierre
name: Kahn, Charles E.
name: Kainmueller, Dagmar
name: Kainz, Bernhard
name: Karargyris, Alexandros
name: Karthikesalingam, Alan
name: Kenngott, Hannes
name: Kleesiek, Jens
name: Kofler, Florian
name: Kooi, Thijs
name: Kopp-Schneider, Annette
name: Kozubek, Michal
name: Kreshuk, Anna
name: Kurc, Tahsin
name: Landman, Bennett A.
name: Litjens, Geert
name: Madani, Amin
name: Maier-Hein, Klaus
name: Martel, Anne L.
name: Mattson, Peter
name: Meijering, Erik
name: Menze, Bjoern
name: Moons, Karel G. M.
name: Müller, Henning
name: Nichyporuk, Brennan
name: Nickel, Felix
name: Petersen, Jens
name: Rafelski, Susanne M.
name: Rajpoot, Nasir
name: Reyes, Mauricio
name: Riegler, Michael A.
name: Rieke, Nicola
name: Saez-Rodriguez, Julio
name: Sánchez, Clara I.
name: Shetty, Shravya
name: van Smeden, Maarten
name: Summers, Ronald M.
name: Taha, Abdel A.
name: Tiulpin, Aleksei
name: Tsaftaris, Sotirios A.
name: Van Calster, Ben
name: Varoquaux, Gaël
name: Wiesenfarth, Manuel
name: Yaniv, Ziv R.
name: Jäger, Paul F.
name: Maier-Hein, Lena
Publisher: name: arXiv
Year: 2023
Subjects: subject: Computer Vision and Pattern Recognition (cs.CV)
subject: FOS: Computer and information sciences
subject: FOS: Computer and information sciences

description: Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.
description: Shared first authors: Annika Reinke and Minu D. Tizabi; shared senior authors: Lena Maier-Hein and Paul F. J\"ager. Published in Nature Methods. arXiv admin note: text overlap with arXiv:2206.01653