Enhancing Abnormality Grounding for Vision Language Models with
  Knowledge Descriptions
Authors: name: Jun Li; name: Che Liu; name: Wenjia Bai; name: Rossella Arcucci; name: Cosmin I. Bercea; name: Julia A. Schnabel
Published: 2025-03-05 09:02:33

Abstract: Visual Language Models (VLMs) have demonstrated impressive capabilities in
visual grounding tasks. However, their effectiveness in the medical domain,
particularly for abnormality detection and localization within medical images,
remains underexplored. A major challenge is the complex and abstract nature of
medical terminology, which makes it difficult to directly associate
pathological anomaly terms with their corresponding visual features. In this
work, we introduce a novel approach to enhance VLM performance in medical
abnormality detection and localization by leveraging decomposed medical
knowledge. Instead of directly prompting models to recognize specific
abnormalities, we focus on breaking down medical concepts into fundamental
attributes and common visual patterns. This strategy promotes a stronger
alignment between textual descriptions and visual features, improving both the
recognition and localization of abnormalities in medical images.We evaluate our
method on the 0.23B Florence-2 base model and demonstrate that it achieves
comparable performance in abnormality grounding to significantly larger 7B
LLaVA-based medical VLMs, despite being trained on only 1.5% of the data used
for such models. Experimental results also demonstrate the effectiveness of our
approach in both known and previously unseen abnormalities, suggesting its
strong generalization capabilities.