Metrics reloaded: Recommendations for image analysis validation
Creators: name: Maier-Hein, Lena
name: Reinke, Annika
name: Godau, Patrick
name: Tizabi, Minu D.
name: Buettner, Florian
name: Christodoulou, Evangelia
name: Glocker, Ben
name: Isensee, Fabian
name: Kleesiek, Jens
name: Kozubek, Michal
name: Reyes, Mauricio
name: Riegler, Michael A.
name: Wiesenfarth, Manuel
name: Kavur, A. Emre
name: Sudre, Carole H.
name: Baumgartner, Michael
name: Eisenmann, Matthias
name: Heckmann-Nötzel, Doreen
name: Rädsch, Tim
name: Acion, Laura
name: Antonelli, Michela
name: Arbel, Tal
name: Bakas, Spyridon
name: Benis, Arriel
name: Blaschko, Matthew
name: Cardoso, M. Jorge
name: Cheplygina, Veronika
name: Cimini, Beth A.
name: Collins, Gary S.
name: Farahani, Keyvan
name: Ferrer, Luciana
name: Galdran, Adrian
name: van Ginneken, Bram
name: Haase, Robert
name: Hashimoto, Daniel A.
name: Hoffman, Michael M.
name: Huisman, Merel
name: Jannin, Pierre
name: Kahn, Charles E.
name: Kainmueller, Dagmar
name: Kainz, Bernhard
name: Karargyris, Alexandros
name: Karthikesalingam, Alan
name: Kenngott, Hannes
name: Kofler, Florian
name: Kopp-Schneider, Annette
name: Kreshuk, Anna
name: Kurc, Tahsin
name: Landman, Bennett A.
name: Litjens, Geert
name: Madani, Amin
name: Maier-Hein, Klaus
name: Martel, Anne L.
name: Mattson, Peter
name: Meijering, Erik
name: Menze, Bjoern
name: Moons, Karel G. M.
name: Müller, Henning
name: Nichyporuk, Brennan
name: Nickel, Felix
name: Petersen, Jens
name: Rajpoot, Nasir
name: Rieke, Nicola
name: Saez-Rodriguez, Julio
name: Sánchez, Clara I.
name: Shetty, Shravya
name: van Smeden, Maarten
name: Summers, Ronald M.
name: Taha, Abdel A.
name: Tiulpin, Aleksei
name: Tsaftaris, Sotirios A.
name: Van Calster, Ben
name: Varoquaux, Gaël
name: Jäger, Paul F.
Publisher: name: arXiv
Year: 2022
Subjects: subject: Computer Vision and Pattern Recognition (cs.CV)
subject: FOS: Computer and information sciences
subject: FOS: Computer and information sciences

description: Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. Particularly in automatic biomedical image analysis, chosen performance metrics often do not reflect the domain interest, thus failing to adequately measure scientific progress and hindering translation of ML techniques into practice. To overcome this, our large international expert consortium created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. The framework was developed in a multi-stage Delphi process and is based on the novel concept of a problem fingerprint - a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), data set and algorithm output. Based on the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as a classification task at image, object or pixel level, namely image-level classification, object detection, semantic segmentation, and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool, which also provides a point of access to explore weaknesses, strengths and specific recommendations for the most common validation metrics. The broad applicability of our framework across domains is demonstrated by an instantiation for various biological and medical image analysis use cases.
description: Shared first authors: Lena Maier-Hein, Annika Reinke. arXiv admin note: substantial text overlap with arXiv:2104.05642 Published in Nature Methods