NOVA: A Benchmark for Anomaly Localization and Clinical Reasoning in
  Brain MRI
Authors: name: Cosmin I. Bercea; name: Jun Li; name: Philipp Raffler; name: Evamaria O. Riedel; name: Lena Schmitzer; name: Angela Kurz; name: Felix Bitzer; name: Paula Roßmüller; name: Julian Canisius; name: Mirjam L. Beyrle; name: Che Liu; name: Wenjia Bai; name: Bernhard Kainz; name: Julia A. Schnabel; name: Benedikt Wiestler
Published: 2025-05-20 08:10:57

Abstract: In many real-world applications, deployed models encounter inputs that differ
from the data seen during training. Out-of-distribution detection identifies
whether an input stems from an unseen distribution, while open-world
recognition flags such inputs to ensure the system remains robust as
ever-emerging, previously unknown categories appear and must be addressed
without retraining. Foundation and vision-language models are pre-trained on
large and diverse datasets with the expectation of broad generalization across
domains, including medical imaging. However, benchmarking these models on test
sets with only a few common outlier types silently collapses the evaluation
back to a closed-set problem, masking failures on rare or truly novel
conditions encountered in clinical use.
  We therefore present NOVA, a challenging, real-life evaluation-only
benchmark of sim900 brain MRI scans that span 281 rare pathologies and
heterogeneous acquisition protocols. Each case includes rich clinical
narratives and double-blinded expert bounding-box annotations. Together, these
enable joint assessment of anomaly localisation, visual captioning, and
diagnostic reasoning. Because NOVA is never used for training, it serves as an
extreme stress-test of out-of-distribution generalisation: models must bridge
a distribution gap both in sample appearance and in semantic space. Baseline
results with leading vision-language models (GPT-4o, Gemini 2.0 Flash, and
Qwen2.5-VL-72B) reveal substantial performance drops across all tasks,
establishing NOVA as a rigorous testbed for advancing models that can detect,
localize, and reason about truly unknown anomalies.