HealthGPT: A Medical Large Vision-Language Model for Unifying
  Comprehension and Generation via Heterogeneous Knowledge Adaptation
Authors: name: Tianwei Lin; name: Wenqiao Zhang; name: Sijing Li; name: Yuqian Yuan; name: Binhe Yu; name: Haoyuan Li; name: Wanggui He; name: Hao Jiang; name: Mengze Li; name: Xiaohui Song; name: Siliang Tang; name: Jun Xiao; name: Hui Lin; name: Yueting Zhuang; name: Beng Chin Ooi
Published: 2025-02-14 00:42:36

Abstract: We present HealthGPT, a powerful Medical Large Vision-Language Model
(Med-LVLM) that integrates medical visual comprehension and generation
capabilities within a unified autoregressive paradigm. Our bootstrapping
philosophy is to progressively adapt heterogeneous comprehension and generation
knowledge to pre-trained large language models (LLMs). This is achieved through
a novel heterogeneous low-rank adaptation (H-LoRA) technique, which is
complemented by a tailored hierarchical visual perception approach and a
three-stage learning strategy. To effectively learn the HealthGPT, we devise a
comprehensive medical domain-specific comprehension and generation dataset
called VL-Health. Experimental results demonstrate exceptional performance and
scalability of HealthGPT in medical visual unified tasks. Our project can be
accessed at https://github.com/DCDmllm/HealthGPT.