Title: UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysis
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: Multimodal Learning, Medical AI, Vision-Language Model, Unified Framework

Abstract: Clinical diagnosis demands models that can process multimodal medical inputs (images, patient histories, lab results) and generate diverse outputs including both textual reports and visual content (annotations, segmentation masks, and images). Despite this need, existing medical AI systems disrupt this unified process: medical image understanding models interpret images but cannot generate visual outputs, while medical image generation models synthesize images but cannot provide textual explanations.  This leads to gaps in data representation, feature integration, and task-level multimodal capabilities. To this end, we propose a multi-level framework that mirrors clinical diagnosis through the Observation-Knowledge-Analysis (OKA) paradigm. Specifically, at the observation level, we construct UniMed-5M, a dataset comprising over 5.6M samples that reformat diverse unimodal data into multimodal pairs for foundational observation.  At the knowledge level, we propose Progressive Curriculum Learning that systematically introduce medical multimodal knowledge.  At the analysis level, we introduce UniMedVL, the first medical unified multimodal model for the simultaneous analysis of image understanding and generation tasks within a single architecture. UniMedVL achieves superior performance on five medical image understanding benchmarks, while matching specialized models in generation quality across eight medical imaging modalities. Crucially, our unified architecture enables bidirectional knowledge sharing generation tasks enhance visual understanding features, demonstrating that integrating traditionally separate capabilities within a single medical framework unlocks  improvements across diverse clinical scenarios. Code is available at https://anonymous.4open.science/r/Uni-MedVL-65F2/README.md.

TL;DR: UniMedVL is the first unified framework that both understands and generates multimodal medical content (images and text) within a single model.