Medical Out-of-Distribution Analysis Challenge 25
Creators: name: Zimmerer, David
name: Köhler, Gregor
name: Full, Peter
name: Maier-Hein, Klaus
name: Reinke, Annika
name: Maier-Hein, Lena
name: Bercea, Cosmin
name: Schnabel, Julia
name: Wiestler, Benedikt
Publisher: name: Zenodo
Year: 2025
Subjects: Anomaly detection
Anomaly localization
Out-of-distribution detection
novelty detection
MICCAI 2025 challenge

description: Despite advancements in biomedical image computing, creating annotated training datasets remains a costly and time-intensive process, requiring the expertise of domain specialists. This dependency on labeled data limits scalability, particularly when dealing with new imaging modalities, evolving parameters, or shifting patient demographics. Moreover, standard algorithms often fail when faced with out-of-distribution (OOD) samples, resulting in inaccurate or overconfident predictions [20–23]. Even clinicians can overlook unexpected findings in medical images, a phenomenon known as "inattentional blindness" [1].

Unsupervised and weakly supervised anomaly detection presents a promising alternative, as it does not require labeled datasets and can generalize to previously unseen pathologies. This approach streamlines model adaptation to new imaging modalities and directs physicians' attention to overlooked abnormalities, potentially improving diagnostic accuracy. While anomaly detection has been a topic of active research [8–17], particularly in the medical domain [4–7], the field lacks a standardized benchmark for fair algorithm comparison in medical and radiological image analysis [2, 3].

To address this gap, we propose the Medical Out-of-Distribution (MOOD) Challenge, offering a benchmark and dataset for anomaly detection.

Insights from previous iterations of the challenge (MOOD 2020–2024) [25] revealed that while some methods performed well in specific cases, they often struggled with clinically significant anomalies [6, 18–19, 23]. To address this, we have broadened the scope to include more clinically relevant and synthetic examples. These updates have demonstrated improved performance for certain medical conditions, showcasing the potential of anomaly detection approaches. However, significant challenges remain in achieving clinical applicability, emphasizing the need for continued innovation.

We have transitioned to an object-centric metric for pixel-wise evaluation to better align with medical objectives [9, 26]. This shift incorporates insights from real-world evaluations and feedback from clinical partners. For the next year, we will continue refining our synthetic anomalies to create a more robust and representative challenge. Additionally, we are introducing a new "real-world" use-case task focused on anomaly detection. Starting in 2026, the MOOD Challenge will adopt a rotating leadership structure, with the University of Erlangen, Imperial College London, TU Munich, and DKFZ taking turns as the primary organizers each year. Each institution will bring unique datasets and medical objectives, reflecting diverse perspectives on out-of-distribution detection and anomaly localization while staying true to the MOOD spirit.

The MOOD Challenge is dedicated to driving methodological advancements in anomaly detection while benchmarking their effectiveness. It addresses critical challenges in medical image analysis, including limited labeled data and out-of-distribution errors.

References

[1] Drew, Trafton, Melissa L. H. Vo, and Jeremy M. Wolfe. "'The Invisible Gorilla Strikes Again: Sustained Inattentional Blindness in Expert Observers.'" Psychological Science 24, no. 9 (September 2013): 1848-53. https://doi.org/10.1177/0956797613479386.[2] Bergmann, Paul, Michael Fauser, David Sattlegger, and Carsten Steger. "MVTec AD -- A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection," 9592-9600, 2019. http://openaccess.thecvf.com/content_CVPR_2019/html/Bergmann_MVTec_AD_--_A_Comprehensive_Real-World_Dataset_for_Unsupervised_Anomaly_CVPR_2019_paper.html.[3] Hendrycks, Dan, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. "A Benchmark for Anomaly Segmentation." ArXiv:1911.11132 [Cs], November 25, 2019. http://arxiv.org/abs/1911.11132.[4] Chen, Xiaoran, Nick Pawlowski, Martin Rajchl, Ben Glocker, and Ender Konukoglu. "Deep Generative Models in the Real-World: An Open Challenge from Medical Imaging." CoRR abs/1806.05452 (2018).[5] Baur, Christoph, Benedikt Wiestler, Shadi Albarqouni, and Nassir Navab. "Deep Autoencoding Models for Unsupervised Anomaly Segmentation in Brain MR Images." ArXiv:1804.04488 [Cs], April 12, 2018. http://arxiv.org/abs/1804.04488.[6] Zimmerer, David, Fabian Isensee, Jens Petersen, Simon Kohl, and Klaus Maier-Hein. "Unsupervised Anomaly Localization Using Variational Auto-Encoders." In International Conference on Medical Image Computing and Computer-Assisted Intervention, 289-297. Springer, 2019.[7] Schlegl, Thomas, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. "Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery," n.d. https://arxiv.org/pdf/1703.05921.pdf.[8] Abati, Davide, Angelo Porrello, Simone Calderara, and Rita Cucchiara. "Latent Space Autoregression for Novelty Detection." ArXiv:1807.01653 [Cs], July 4, 2018. http://arxiv.org/abs/1807.01653. [9] Ahmed, Faruk, and Aaron Courville. "Detecting Semantic Anomalies." ArXiv:1908.04388 [Cs], August 13, 2019. http://arxiv.org/abs/1908.04388.[10] Akcay, Samet, Amir Atapour-Abarghouei, and Toby P. Breckon. "Skip-GANomaly: Skip Connected and Adversarially Trained Encoder-Decoder Anomaly Detection." ArXiv:1901.08954 [Cs], January 25, 2019. http://arxiv.org/abs/1901.08954.[11] Beggel, Laura, Michael Pfeiffer, and Bernd Bischl. "Robust Anomaly Detection in Images Using Adversarial Autoencoders." ArXiv:1901.06355 [Cs, Stat], January 18, 2019. http://arxiv.org/abs/1901.06355.[12] Bergmann, Paul, Michael Fauser, David Sattlegger, and Carsten Steger. "Uninformed Students: Student-Teacher Anomaly Detection with Discriminative Latent Embeddings." ArXiv:1911.02357 [Cs], November 6, 2019. http://arxiv.org/abs/1911.02357.[13] Choi, Hyunsun, Eric Jang, and Alexander A. Alemi. "WAIC, but Why? Generative Ensembles for Robust Anomaly Detection." ArXiv:1810.01392 [Cs, Stat], October 2, 2018. http://arxiv.org/abs/1810.01392.[14] Guggilam, Sreelekha, S. M. Arshad Zaidi, Varun Chandola, and Abani Patra. "Bayesian Anomaly Detection Using Extreme Value Theory." ArXiv:1905.12150 [Cs, Stat], May 28, 2019. http://arxiv.org/abs/1905.12150.[15] Maaloe, Lars, Marco Fraccaro, Valentin Lievin, and Ole Winther. "BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling." ArXiv:1902.02102 [Cs, Stat], February 6, 2019. http://arxiv.org/abs/1902.02102.[16] Piciarelli, Claudio, Pankaj Mishra, and Gian Luca Foresti. "Image Anomaly Detection with Capsule Networks and Imbalanced Datasets." ArXiv:1909.02755 [Cs], September 6, 2019. http://arxiv.org/abs/1909.02755.[17] Sabokrou, Mohammad, Mohammad Khalooei, Mahmood Fathy, and Ehsan Adeli. "Adversarially Learned One-Class Classifier for Novelty Detection." ArXiv:1802.09088 [Cs], February 25, 2018. http://arxiv.org/abs/1802.09088.[18] Goldstein, Markus, and Seiichi Uchida. "A Comparative Evaluation of Unsupervised Anomaly Detection Algorithms for Multivariate Data." PLOS ONE 11, no. 4 (April 19, 2016): e0152173.https://doi.org/10.1371/journal.pone.0152173.[19] Skvara, Vit, Tomas Pevny, and Vaclav Smidl. "Are Generative Deep Models for Novelty Detection Truly Better?" ArXiv:1807.05027 [Cs, Stat], July 13, 2018. http://arxiv.org/abs/1807.05027.[20] Hendrycks, Dan, and Kevin Gimpel. "A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks." ArXiv:1610.02136 [Cs], October 7, 2016. http://arxiv.org/abs/1610.02136.[21] Mehrtash, Alireza, William M. Wells III, Clare M. Tempany, Purang Abolmaesumi, and Tina Kapur. "Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation." ArXiv:1911.13273 [Cs, Eess], November 29, 2019. http://arxiv.org/abs/1911.13273.[22] Roady, Ryne, Tyler L. Hayes, Ronald Kemker, Ayesha Gonzales, and Christopher Kanan. "Are Out-of-Distribution Detection Methods Effective on Large-Scale Datasets?" ArXiv:1910.14034 [Cs], October 30, 2019. http://arxiv.org/abs/1910.14034.[23] Shafaei, Alireza, Mark Schmidt, and James J. Little. "A Less Biased Evaluation of Out-of-Distribution Sample Detectors." ArXiv:1809.04729 [Cs, Stat], August 20, 2019. http://arxiv.org/abs/1809.04729.[24] Maier-Hein, L., Eisenmann, M., Reinke, A. et al. "Why rankings of biomedical image analysis competitions should be interpreted with care." Nat Commun 9, 5217 (2018). https://doi.org/10.1038/s41467-018-07619-7[25] http://medicalood.dkfz.de/ , https://www.synapse.org/(hashtag)!Synapse:syn21343101/wiki/599515 [26] Zimmerer, D., Paech, D., Lüth, C., Petersen, J., Köhler, G., Maier-Hein, K. (2022). Unsupervised Anomaly Detection in the Wild. In: Maier-Hein, K., Deserno, T.M., Handels, H., Maier, A., Palm, C., Tolxdorff, T. (eds) Bildverarbeitung für die Medizin 2022. Informatik aktuell. Springer Vieweg, Wiesbaden. https://doi.org/10.1007/978-3-658-36932-3_6[27] Zimmerer, David, and Klaus Maier-Hein. "Beyond Heatmaps: A Comparative Analysis of Metrics for Anomaly Localization in Medical Images." International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging. Cham: Springer Nature Switzerland, 2024.