NOVA: A Benchmark for Anomaly Localization and Clinical Reasoning in
  Brain MRI
Authors: name: Cosmin I. Bercea, name: Jun Li, name: Philipp Raffler, name: Evamaria O. Riedel, name: Lena Schmitzer, name: Angela Kurz, name: Felix Bitzer, name: Paula Roßmüller, name: Julian Canisius, name: Mirjam L. Beyrle, name: Che Liu, name: Wenjia Bai, name: Bernhard Kainz, name: Julia A. Schnabel, name: Benedikt Wiestler

In many real-world applications, deployed models encounter inputs that differ
from the data seen during training. Out-of-distribution detection identifies
whether an input stems from an unseen distribution, while open-world
recognition flags such inputs to ensure the system remains robust as
ever-emerging, previously unknown categories appear and must be addressed
without retraining. Foundation and vision-language models are pre-trained on
large and diverse datasets with the expectation of broad generalization across
domains, including medical imaging. However, benchmarking these models on test
sets with only a few common outlier types silently collapses the evaluation
back to a closed-set problem, masking failures on rare or truly novel
conditions encountered in clinical use.
  We therefore present NOVA, a challenging, real-life evaluation-only
benchmark of sim900 brain MRI scans that span 281 rare pathologies and
heterogeneous acquisition protocols. Each case includes rich clinical
narratives and double-blinded expert bounding-box annotations. Together, these
enable joint assessment of anomaly localisation, visual captioning, and
diagnostic reasoning. Because NOVA is never used for training, it serves as an
extreme stress-test of out-of-distribution generalisation: models must bridge
a distribution gap both in sample appearance and in semantic space. Baseline
results with leading vision-language models (GPT-4o, Gemini 2.0 Flash, and
Qwen2.5-VL-72B) reveal substantial performance drops across all tasks,
establishing NOVA as a rigorous testbed for advancing models that can detect,
localize, and reason about truly unknown anomalies.