Unimedvl: Unifying Medical Multimodal Understanding And Generation
  Through Observation-Knowledge-Analysis
Authors: name: Junzhi Ning, name: Wei Li, name: Cheng Tang, name: Jiashi Lin, name: Chenglong Ma, name: Chaoyang Zhang, name: Jiyao Liu, name: Ying Chen, name: Shujian Gao, name: Lihao Liu, name: Yuandong Pu, name: Huihui Xu, name: Chenhui Gou, name: Ziyan Huang, name: Yi Xin, name: Qi Qin, name: Zhongying Deng, name: Diping Song, name: Bin Fu, name: Guang Yang, name: Yuanfeng Ji, name: Tianbin Li, name: Yanzhou Su, name: Jin Ye, name: Shixiang Tang, name: Ming Hu, name: Junjun He
Organization: General Medical AI

Medical diagnostic applications require models that can process multimodal
medical inputs (images, patient histories, lab results) and generate diverse
outputs including both textual reports and visual content (annotations,
segmentation masks, and images). Despite this need, existing medical AI systems
disrupt this unified process: medical image understanding models interpret
images but cannot generate visual outputs, while medical image generation
models synthesize images but cannot provide textual explanations. This leads to
gaps in data representation, feature integration, and task-level multimodal
capabilities. To this end, we propose a multi-level framework that draws
inspiration from diagnostic workflows through the
Observation-Knowledge-Analysis (OKA) paradigm. Specifically, at the observation
level, we construct UniMed-5M, a dataset comprising over 5.6M samples that
reformat diverse unimodal data into multimodal pairs for foundational
observation. At the knowledge level, we propose Progressive Curriculum Learning
that systematically introduces medical multimodal knowledge. At the analysis
level, we introduce UniMedVL, the first medical unified multimodal model for
the simultaneous analysis of image understanding and generation tasks within a
single architecture. UniMedVL achieves superior performance on five medical
image understanding benchmarks, while matching specialized models in generation
quality across eight medical imaging modalities. Crucially, our unified
architecture enables bidirectional knowledge sharing: generation tasks enhance
visual understanding features, demonstrating that integrating traditionally
separate capabilities within a single medical framework unlocks improvements
across diverse medical vision-language tasks. Code is available at
https://github.com/uni-medical/UniMedVL.