UniBiomed: A Universal Foundation Model for Grounded Biomedical Image
  Interpretation
Authors: name: Linshan Wu, name: Yuxiang Nie, name: Sunan He, name: Jiaxin Zhuang, name: Hao Chen

Multi-modal interpretation of biomedical images opens up novel opportunities
in biomedical image analysis. Conventional AI approaches typically rely on
disjointed training, i.e., Large Language Models (LLMs) for clinical text
generation and segmentation models for target extraction, which results in
inflexible real-world deployment and a failure to leverage holistic biomedical
information. To this end, we introduce UniBiomed, the first universal
foundation model for grounded biomedical image interpretation. UniBiomed is
based on a novel integration of Multi-modal Large Language Model (MLLM) and
Segment Anything Model (SAM), which effectively unifies the generation of
clinical texts and the segmentation of corresponding biomedical objects for
grounded interpretation. In this way, UniBiomed is capable of tackling a wide
range of biomedical tasks across ten diverse biomedical imaging modalities. To
develop UniBiomed, we curate a large-scale dataset comprising over 27 million
triplets of images, annotations, and text descriptions across ten imaging
modalities. Extensive validation on 84 internal and external datasets
demonstrated that UniBiomed achieves state-of-the-art performance in
segmentation, disease recognition, region-aware diagnosis, visual question
answering, and report generation. Moreover, unlike previous models that rely on
clinical experts to pre-diagnose images and manually craft precise textual or
visual prompts, UniBiomed can provide automated and end-to-end grounded
interpretation for biomedical image analysis. This represents a novel paradigm
shift in clinical workflows, which will significantly improve diagnostic
efficiency. In summary, UniBiomed represents a novel breakthrough in biomedical
AI, unlocking powerful grounded interpretation capabilities for more accurate
and efficient biomedical image analysis.