HealthGPT: A Medical Large Vision-Language Model for Unifying
  Comprehension and Generation via Heterogeneous Knowledge Adaptation
Authors: name: Tianwei Lin, name: Wenqiao Zhang, name: Sijing Li, name: Yuqian Yuan, name: Binhe Yu, name: Haoyuan Li, name: Wanggui He, name: Hao Jiang, name: Mengze Li, name: Xiaohui Song, name: Siliang Tang, name: Jun Xiao, name: Hui Lin, name: Yueting Zhuang, name: Beng Chin Ooi

We present HealthGPT, a powerful Medical Large Vision-Language Model
(Med-LVLM) that integrates medical visual comprehension and generation
capabilities within a unified autoregressive paradigm. Our bootstrapping
philosophy is to progressively adapt heterogeneous comprehension and generation
knowledge to pre-trained large language models (LLMs). This is achieved through
a novel heterogeneous low-rank adaptation (H-LoRA) technique, which is
complemented by a tailored hierarchical visual perception approach and a
three-stage learning strategy. To effectively learn the HealthGPT, we devise a
comprehensive medical domain-specific comprehension and generation dataset
called VL-Health. Experimental results demonstrate exceptional performance and
scalability of HealthGPT in medical visual unified tasks. Our project can be
accessed at https://github.com/DCDmllm/HealthGPT.