HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation
Authors: Tianwei Lin
Wenqiao Zhang
Sijing Li
Yuqian Yuan
Binhe Yu
Haoyuan Li
Wanggui He
Hao Jiang
Mengze Li
Song xiaohui
Siliang Tang
Jun Xiao
Hui Lin
Yueting Zhuang
Beng Chin Ooi
Venue: ICML 2025 spotlightposter (2025)
Keywords: Medical Large Vision-Language Models; Multi-Modal Comprehension and Generation
Primary area: applications->health_medicine

Abstract: We present **HealthGPT**, a powerful Medical Large Vision-Language Model (Med-LVLM) that integrates medical visual comprehension and generation capabilities within a unified autoregressive paradigm. Our bootstrapping philosophy is to progressively adapt heterogeneous comprehension and generation knowledge to pre-trained Large Language Models (LLMs). This is achieved through a novel heterogeneous low-rank adaptation **(H-LoRA)** technique, which is complemented by a tailored hierarchical visual perception **(HVP)** approach and a three-stage learning strategy **(TLS)**. To effectively learn the HealthGPT, we devise a comprehensive medical domain-specific comprehension and generation dataset called **VL-Health**. Experimental results demonstrate exceptional performance and scalability 
of HealthGPT in medical visual unified tasks. Our project can be accessed at https://github.com/DCDmllm/HealthGPT.

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation
Tianwei Lin
Wenqiao Zhang' Sijing Li'
Yugian Yuan 12 Binhe Yu² 3 Haoyuan Li
Wanggui He Yucting Zhuang'
Hao Jiang?
Mengze Li
Xiaohui Song'
Siliang Tang' Jun Xiao Hui Lin Beng Chin Ooi
7 Medical Multi-Modal Comprehension Tasks
List all anatomical locations showing pulmonary edema, opacity, mediastinal displacement.
Microsopy Comprehension
What the of the different trealments this saudy?
Comp, Pert
SLACE Clew
Comprehension
Al
Left structures, lung. right hilar structures, right lung.
The purpose of the different membrane treatments used...
VELL BAD
Comprehension
What is the purpose of compering OCT
CT Comprahonsion
Which abdominal organ shows any indication of a lasion or abnormalty the CT image?
image
image with histology?
To confirm the histological position the obtained OCT brain images.
No abdominal organs show any clear indications of lesions ebnommallies.
Fundus Comprehension
What spesific findings pathological changes can be observed Uhis fundus image?
7. Ultrasound Comprehension
What type imaging technique image?
3. MRI Comprehension
Could you expisin in MRI means my healt?? Is it very serious?
image sagittal gray- scale uttrasonographii...
The fundus image appears with noticeable signs pathology...
Certainly, defined mass your left nasal cavity: Thare na sign of the...
MRI shows a
Report-to-CXt
Here X-ray image you.
HealthGPT
The Xray shows
1. CT2MR/ Ganeration
need a version this CT representation in MRI.
pneumothorax.
The image has transformed
Image Reconstruction
Super Resolution Improve quality this MRI image?
Gen. Perf
Reconetruct the following medical images.
2. MRI2CT Generation
Transform the display into image.
of the MRI image.
Here is the raconstructed medical Image you
Here the with improved resolution.
CTMMUMMENTT BROOL
BAR
5 Medical Multi-Modal Generation Tasks
Figure I: HealthGPT enables medical multimodal comprehension and generation, outperforming both state-of-the-art unified visual models and medical-specifer models across various tasks. This highlights its superior capability in tackling complex tasks in healthcare applications. Comp.Perf. and Ger.Perf. denote the results of comprehension and generation.
Abstract
integrates medical visual comprehension and gen- eration capabilities within a unified autoregres- sive paradigm. Our bootstrapping philosophy is progressively adapt heterogeneous com- prehension and generation knowledge to pre- trained Large Language Models (LLMs). This is achieved through a novel heterogeneous low- rank adaptation (H-LoRA) technique, which is complemented by a tailored hierarchical visual perception (HVP) approach and three-stage learning strategy (TLS). To effectively learn the HealthGPT, devise comprehensive medical
We present HealthGPT, a powerful Medical Large Vision-Language Model (Med-LVLM) that
'Zhejiang University, Hangzhou, China PAlibaba Group, Hangzhou, China "University of Electronic Science and Tech- nology of China. Chengdu, China 'The Hong Kong University of Science and Technology, HongKong, China National University of Singapore, Singapore. Correspondence to: Wenqiao Zhang <wenqiaozhang@rjjn @zju.cdu.c>>>
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprchension and Generation
domain-specific comprehension and generation dataset called VL.-Heatth. Experimental results demonstrate exceptional performance and scal- ability of HealthGPT in medical visual unified tasks. Our project can be accessed at https: / //github. Coom/CDDmn htm.//eelhhhaa
(a)
(b)
70
70
60
60
Performencet
50
50
40
40
30
1. Introduction
20-
20
Large LasguanLanguage Models (LVLMs) (Liu et al., 2023; OpenAI, 2023; Liu et al., 2024c; Chen et al., 2024b) have demonstrated outstanding open-world woolld visual comprehension and reasoning abilities through language-base based interactive dialogue over the past years, simultaneously opening up new opportunities for applications in specialized domains. Specifically, recent studies (Li ct al., 2024b: Tu et al.. 2024; Li et al., 2025) have utilized pre-trained Large Language Models (LLMs) and visual instruction data to build interac- tive diagnostic tools and treatment planning systems, reveal- ing the immense potential of LVLMs in medical scenarios. However, these studies primarily concentrate visual com- prehension tasks that produce text-based outputs, such as medical visual question answering (Li et al., 2024b) or re- port generation (Nath et al., 2024), and deficient the "draw- ing" capability needed for medical visual generation. In practice, integrating visual comprehension and generation can significantly enhance the multifunctionality of medical LVLMs (Med-LVLMs).
10
10
25
50
200
Gen. Data Ratio (96)
Comp. Data Ratio (%)
Comp Perf
Gem Peef
Ours Comp, Perf
Ours Gem. Perf.
Figure 2: With a fixed amount of comprehension or genera- tion data, increasing the proportion of the other type leads to significant performance degradation.
develop a unified medical model from scratch. (ii) Con- flicts between Comprehension and Generation. Compre- hension tasks often strip away visual details to focus on abstraction, while generation tasks require detailed preser- vation, making tokens sensitive to all visual alterations. As shown in Figure 2, which features experiments conducted on medical images, the performance in comprehension (or generation) tasks steadily decreases as the proportion of gen- eration (or comprehension) data increases, and vice versa. This highlights a dilemma in autoregressive multimodal training, stemming from the need to maintain consistency of features between pre-LVLMs and post-LVLMMs Although some methods have explored mutual enhancement between comprehension and generation tasks (Pan et al., 2024; Tong et al., 2024), joint training in medical scenarios still faces challenges such as task conflict, data bias, and optimiza- tion saturation, leading to limited performance gains and persistent degradation issues.
Recent studies have increasingly focused on developing unified LVLMs capable of comprehending and generating content across diverse visual modalities. Earlier approaches predominantly utilized continuous visual tokens fed into LLMs, using the LLMs themselves as conditional gener- ators for external generative models (Ge et al., 2024; Wu et al., 2024b; Dong et al., 2024). More recent research has explored the use of discrete visual tokens for image representation and generation within a fully autoregressive framework (Team, 2024a; Wang et al., 2024a; Xie et al., 2024). These methods not only enhance controllability but also demonstrate early success in open-world, any-to-any tasks, highlighting the preliminary potential of a unified autoregressive learning paradigm in multimodal tasks.
To tackle the aforementioned challenges, we propose HealthGPT (see Figure 1) which progressively adapts a pre-trained LLM as an unified medical multimodal model with a small amount of visual instruction data. We de- vise innovative Parameter- approach (Ding al., 2023), called Heterogeneous Low- Rank Adaptation (H-LoRA), which decouples the learning process of LVLMs for comprehension and generation tasks. Inspired by the plug-and-play nature of LoRA (Hu et al., 2022), H-LoRA enables the model to store heterogeneous comprehension and generation knowledge in independent "plugins", thus avoiding joint optimization issues caused by conflicts between comprehension and generation tasks. In addition, we also consider the variety of sub-tasks among comprehension generation tasks. Qualitative research highlights the limitations of a single LoRA in handling mul- tidimensional task scenarios, mainly due to catastrophic forgetting and interference (Liu et al., 2024d; Lin et al., 2024). To address this, we draw on the concept of Mixture
Fine-Tuning (PEFT)
While unified LVLMs have achieved initial success in gen- eral scenarios, such a unified framework remains under- explored in the medical domain. Adapting the aforemen- tioned general unified model paradigm to the medical do- main presents two major challenges: (i) High-scale and -quality Data Limitations. Open-world models necessi- tate extensive pre-training on billions even more diverse, multimodal data samples for comprehension and generation tasks (Lu et al., 2024: Team, 2024a). However, the acces- sible medical data significantly lacks in scale and quality compared to natural multimodal datasets. Its specialized and domain sppeiffi characteristics make it challenging to
2

---

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprchension and Generation
of Experts (MoE) (Masoudnia & Ebrahimpour, 2014) and introduce LoRA experts. The aim is to dynamically trans- fer task-shared knowledge to adapt to downstream tasks. Unlike MoELoRA (Luo et al., 2024a), H-LoRA employs reversible matrix block multiplication to combine LoRA experts, significantly reducing the overhead of multiple ma- trix multiplications. Notably, when using four experts, it requires only 67% % of the MoELoRA training time.
comprehensive dataset designed for both comprehen- sion and generation tasks.
Superior Downstream Improvements: Extensive ex- periments are conducted and the tiveness of our approach in medical vision-language comprehension and generation.
results confirm effec-
2. Related Work
To effectively leverage H-LoRA in HealthGPT, we further introduce a Hierarchical Visual Perception (HVP) and devise a corresponding Three-stage Learning Strategy (TLS). HVP: we separate visual details learning from Vi- sion transformer (ViT) for comprehension and generation. As is widely recognized, the ViT encodes visual concepts with increasing abstraction, generally, becoming finer as we progress over levels (Vig, 2019). Thus, we maintain the visual features of the anterior and posterior layers to accom- modate the differing requirements for visual granularity in comprehension and generation tasks while preventing po- tential task interference. TLS: In the first and second stages, given the heterogeneity between comprehension and gener- ation tasks, we first train H-LoRA plugins for HealthGPT to incorporate both medical comprehension and generation knowledge, thus endowing the LLMs with capabilities for vision-language alignment and vision-to- toviiiioo reconstruc- tion. Additionally, through minimal mixed-task training, we built fusion embedding layers and output heads that merge text and visual tokens, establishing a unified Med-LVLM foundation for visual instruction tin-ttnnig. In the third stage, by only training the H-LoRA plugins, HealthGPT is able to rapidly adapt to wide range of downstream medical tasks, covering various types of medical comprehension and generation tasks.
Medical Vision Large Language Models. In recent years, the development of deep learning brings prosperity to the field in medical applications (Zhang et al., 2022; Ong et al., 2022). Most recently, medical vision large language models (Med-VLLMs) demonstrate excellent performance in under- standing medical images and responding to human queries based on these images (Zhou et al., 2023; Tian et al., 2023). XrayGPT (Thawakar et al., 2023) combines a medical visual encoder (MedClip) (Wang et al., 2022) with a fine-tuned LLM, using simple linear transformation layer to achieve alignment between visual and textual information, signifi- cantly enhancing the understanding of medical images. On this basis, some methods (Wu et al., 2023; Li et al., 2024a; Bansal et al., 2025) further enhance visual-text alignment in medical contexts by selecting high-quality image-text pairs from PubMed papers and synthesized VQA datasets. BiomedGPT (Luo et al., 2024b) employs BERT-style en- coder and GPT-style decoder architecture, pre-trained on interdisciplinary datasets. Compared to commercial mod- els like Med-Pal.M (Singhal et al., 2023), BiomedGPT significantly reduces model size while maintaining supe- rior performance. However, issues of language adaptabil- ity and dataset specificity still remain. To address these, HuatuoGPT Vision (Chen et al., 2024a) introduces the Pub- MedVision dataset, which contains 1.3 million high-quality medical samples, significantly improving the model's adapt- ability across diverse medical applications. However, cur- rent MLL-VLLMs mainly focus on medical comprehension and lack the capability for the medical vision-language gen- cration.
To effectively implement our approach, we have curated a dataset for training unified Med-LVLMs, called VL-HHalth, including seven comprehension tasks and five generation tasks (Figure 1). Through quantitative analysis and vali- dation multimodal tasks, the results demonstrate that HealthGPT is capable of unifying medical multimodal abil- ities in data-constrained scenarios, achieving performance comparable to or better than existing state-of- of-e--of-the-at (SOTA) models across multiple metrics. Overall, the main contribu- tions of this paper are summarized as follows:
Unified Visual Comprehension and Generation Mod- els. Recent research has increasingly concentrated cre- ating unified LVLMs that are adept at understanding and producing content across various visual modalities. NExT- GPT (Wu et al., 2024b) achieves perception and genera- tion for arbitrary combinations of multimodal inputs and outputs by aligning LLMs. Similarly, SEED (Ge et al., 2023), SEED-X (Ge et al., 2024), and DreamLLM (Dong et al., 2024) employ learnable queries and leverage next- token prediction to generate visual tokens, providing condi- tional inputs to external generation modules. Unlike these methods, which function as external conditioners, Unified- 10 (Lu et al., 2022), Unified-IO 2 (Lu et al., 2024). and Chameleon (Team, 2024a) internalize multimodal genera- tion tasks within a unified Transformer architecture by ex-
Unified Med-LVLM. We introduce HealthGPT, which, to the best of our knowledge, is the first unified frame- work for multimodal comprehension and generation in complex medical scenarios.
Effective Learning Paradigm. We present H-LoRA, an optimized multi-LoRA PEFT architecture based on task-gated decoupling, is designed to effectively mitigate data conflict issues.
Holistic Training Dataset. We curated VL--Health, a
3

---

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprchension and Generation
Text Decoder
VQ Decoder
Comp specific Module
Fusion Head
Bn-1
Bin
Gen-specific Module
Router
H-LoRA
Router
A2
Ar-1
An
A1
A2
An-1
An
Frozen Parameters
Decoder Block
Comp. Input
Gen. Task Input
Visual Embedding
Text Embedding
VQ Embedding
Trainable Parameters
<END_TEXTS
<END_MC
Comp. Visual Adapter
Gen.
Fusion Embedding Layer
Visual Adapter
Instruction Embedding
Vision Encoder
Text Tokenizer
VQ-Tokenizer
Response Embedding
Medical Reference Image
Medical Instruction Answer
Medical Generation Image
The HealthGPT architecture integrates hierarchical visual perception and H-LoRA, employing task-specific hard router to select visual features and H-LoRA plugins, ultimately generating outputs with an autoregressive manner.
Figure 3:
tending multimodal vocabularies, enabling direct generation based on next-token prediction. Building on this concept, Lumina-mGPTT (Liu et al., 2024a) and ANOLE (Chern et al., 2024) further enhance the generation capabilities of unified models using high-quality data, particularly improving the quality and flexibility of image generation.
is quantized using codebook Z = { { generating a discrete index sequence I [ where im E Z represents the quantized code index:
I = Quantize(z = arg min 2
|| #x|12-
(2)
In approach, the discrete index sequence I serves supervisory signal for the generation task, enabling the model to predict the index sequence from input conditions such as text or other modality signals. Finally, the predicted index sequence Ż is upsampled by the VQGAN decoder G, generating the high-quality image fimng = G(
3. Preliminaries
Large HisounLanguuag Models. The input to LVLM typically consists of an image img and a discrete text se- quence "M The visual encoder gimg converts the input image ing into a sequence of visual tokens V= = [e.] while the text sequence ax is mapped into sequence of text tokens T [t,,"= using embedding function to The LVLM M-VIM(-110) models the joint probability of the token sequence U (V, T), which is expressed as:
Low Rank Adaptation. LoRA(Hu al., 2022) effec- tively captures the characteristies of downstream tasks by introducing low-rank adapters. The core idea is to decom- pose the bypass weight matrix AW x into two low-rank matrices (A E Rd B E Rrxx"") }. where < min(dd ddm), significantly reducing learnable pa- rameters. The output with the LoRA adapter for the input is then given by:
N. Po(R()) = II Po(md((U
(1)
where - [r.jf is the text response sequence. The LVLM iteratively generates the next token based on r<s- The optimization objective is to minimize the cross-entropy loss of the response R. It is worth noting that most LVLMs adopt a design paradigm based on ViT, alignment adapters, and pre-trained LLMs(Liu al., 2023; 2024b), enabling quick adaptation to downstream tasks.
h Wo + ar /r W. oxAB/r,
(3)
where matrix A initialized with Gaussian distribution, while the matrix B is initialized scaling factor a/r controls the impact of AW the model.
matrix. The
VQGAN. VQGAN (Esser et al., 2021) employs latent space compression and indexing mechanisms to effectively learn a complete discrete representation of images. VQGAN first maps the input image imng to a latent representation z = (xx) through a encoder E. Then, the latent representation
4. HealthGPT
4.1. Unified Autoregressive Generation.
Our approach (Figure 3) utilizes a discrete token representa- tion that covers both text and visual outputs, unifying visual

---

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprchension and Generation
comprehension and generation as an autoregressive task. For comprehension, pre-trained L.M MLLM(-10) receives the imput joint sequence U and outputs series of text token R = [ri, r........ TN, where r; E Via. and Via represents the vocabulary of L.M:
(a)
(K)
(b)
900
Compated
800
765K
783K
Magarrts
lavelig
600
500
400
PR u)) II
OCT
Micemopp
Heromerarthe
300
200
For generation, MLLM first receives a special start token ASTARTJIMG)) then generates a series of tokens corre- sponding to the VQGAN indices I = [6 -2,...; ., & where 8 E Veg. and Vig represents the index range of VQGAN. Upon completion of generation, the LLM outputs an end token (JENDJIMG)):
Dightl Prangraphy
100
Otbe
Demmacy
0
Eadesogy
Comp.
Gen.
Figure 4: Data statisties of VL-Health.
modules and dynamically routes to extract task-relevant knowledge from these modules. At the task level, for each task type T, we dynamically assign dedicated H-LoRA submodule 0", which is expressed as:
N. PO 4) = II Poli, ((, j=1
(5)
Finally, the generated index sequence I is fed into the de-
coder G, which reconstructs the target image gimng GT)).
BT, R MMLM(210,00 O²), O 1 Ruteer ,
(7)
At the feature level for a single task, H-LoRA integrates the idea of Mixture of Experts (MoE) (Masoudnia & Ebrahim- pour, 2014) and designs an efficient matrix merging and routing weight allocation mechanism, thus avoiding the sig- nificant computational delay introduced by matrix splitting in existing MoFLoRA (Luo et al., 2024a). Specifically, we first merge the low-rank matrices (rank r) of LoRA experts into unified matrix:
4.2. Hierarchical Visual Perception
Given the differences visual perception between com- prehension and generation tasks where the former fo- cuses on abstract semantics and the latter emphasizes com- plete semantics we employ ViT to compress the image into discrete visual tokens at multiple hierarchical levels. Specifically, the image is converted into a series of features {f1, f2. } as it passes through L ViT blocks.
A merged B merged
Concat(((A) ((A.))) Concat( Concat((B,)f)) (B.)²), (8)
To address the needs of various tasks, the hidden states are divided into two types: (i) concrete-graaing features Frim = (fi, fuk for k < L,, derived from the shal- lower layers of ViT, containing sufficient global features, suitable for generation tasks; (ii) Abstract-grained features JAbs- = {fi+1. fx+2,..., fr), derived from the deeper layers of ViT, which contain abstract semantic information closer to the text space, suitable for comprehension tasks.
where A merged E Rd and meegged E B The k-dimension routing layer generates expert weights W e ploken.numx based on the input hidden state æ, and these are expanded to ii glloke.nmmrr space as follows:
Weexpaded = akW/r @ 1,,
(9)
where 8 denotes the replication operation. The overall output of H-LoRA is computed as:
The task type T (comprehension or generation) determines which set of features selected as the input for the down- stream large language model:
Oll-LORA_ = (x merged
Weexpanled)
Binerged
(10)
Zing
Con
if Generation Task
(6)
where C represents element-wise multiplication. Finally, the output of H-LoRA is added to the output of frozen pre- trained weights to produce the final output:
JoAbs
if = Comprehension Task
We integrate the image features F- and text features T into a joint sequence through simple concatenation, which is then fed into the LLM MLLM for autoregressive generation.
O = x Wo + H-LORA
(11)
4.4. Training Pipeline
4.3. Heterogeneous Knowledge Adaptation
1st Stage:
multimodal Alignment. In the first stage, we design separate visual adapters and H-LoRA submodules for medical unified tasks, For the medical comprehension
We devise H-LoRA, which stores heterogeneous knowl- edge from comprehension and generation tasks in separate
5

---

HcalthGPT: A Medical Large Vision-Language Model for Unifying Comprchension and Generation
Table 1: Comparison of HealthGPT with other LVLMs and unified multimodal models on medical visual comprehension tasks. Bold and underlined text indicates the best performance and second-best performance, respectively.
Medical L.L.M
VQA-RAD f
SLAKE
Vath close
MMMU -Med
Type
Model
# Params
OMVQAT
Avg.
close
all
close
Med-Flamingo
8,3B
58.6
43.0
47.0
25.5
61.9
31.3
28.7
34,9
41.4
LLaVA-Med
35.7
HuatuoGPT-Vision
7B
66.9
53.0
59.8
49.1
52.9
32.0
42.0
50.0
50.7
BLIP-2
6.7B
X
43.4
36.8
35.3
26.9
36.1
Comp. Only
LLaVA-v1.5
7B
X
51.8
42.8
37.1
37.7
53.5
31.4
32.7
44.7
41.5
InstrucIBLIP
7B
X
61.0
44.8
66.8
43.3
56.0
32.3
25.3
29.0
44.8
Yi-VL
X
52.6
42.1
52.4
38.4
54.9
30.9
38.0
50.2
44.9
InterVV22
8B
X
64.9
49.0
66.6
50.1
60.0
31.9
43.3
54.5
52.5
Llama-322
X
68.9
45.5
33.6
39.3
63.2
54.7
Show-o
1.3B
X
50.6
33.9
31.5
17.9
52.9
28.2
22.7
45.7
42,6
Unified-IO
X
32.6
35.9
52.5
33.0
33.8
Janus
1.3B
X
70.9
52.8
34.7
26.9
51.9
27.9
30.0
26.8
33.5
X
Comp. & Gen.
Janus-Pro
7B
62.9
49.2
51.3
38.7
48.9
27.2
32.7
54,1
45.6
Emu3
8B
X
72.1
55.2
30.6
28.7
39.5
47.2
HealthGPT-M3
3.8B
73.7
55.9
74.6
56.4
78.7
39.7
43.3
68.5
61.3
HcalthGFT-L144
14B
77.7
58.3
76.4
64.5
85.9
44.4
49.2
74.4
66.4
HcathhGPT-XL33
32B
78.1
60.5
83.7
68.2
92.4
50.9
58.0
77.2
71.1
task, we train Dostract-grainedd visual adapters using high- quality image-texi pairs to align visual embeddings with tex- tual embeddings, thereby enabling the model to accurately describe medical visual content. During this process, the pre-trained LLM and its corresponding H-LoRA submod- ules remain frozen. In contrast, the medical generation task requires training mencrete-grainee adapters and H- H-LoRA submodules while keeping the LLM frozen. Meanwhile, we extend the textual vocabulary to include multimodal to- kens, enabling the support of additional VQGAN vector quantization indices. The model trains image-VQ pairs, endowing the pre-trained LLM with the capability for image reconstruction. This design ensures pixel-level consistency of pre- and L...- The processes establish the initial alignment between the LLM's outputs and the visual inputs.
ther optimize the model and enhance its adaptability to downstream tasks such as medical visual comprehension (e.g., medical QA, medical dialogues, and report generation) generation tasks (e.g., super-resolution, denoising, and modality conversion). Notably, by this stage, the word em- bedding layer and output head have been fine-tuned, only the H-LoRA modules and adapter modules need to be trained. This strategy significantly improves the model's 's adaptability and flexibility across different tasks.
5. Experiments
5.1. Data and Experimental Setup
Data Details. We curate VL-Healtt dataset (see Fig- ure 4). For medical visual comprehension, we leverage multiple medical-specific datasets, including PubMedVi- sion (Chen et al., 2024a), LLaVA-Med (Li et al., 2024a), Path VQA (He et al., 2020), MIMIC-CXR- CXR-VQA (Bae et al., 2024), SLAKE (Liu al., 2021), and VQA-RAD (Lau et al., 2018). Additionally, we incorporate high-quality open-world data from LLaVA-1.5 (Liu et al., 2024b) to preserve the model's general knowledge and instruction- following capabilities. For generation tasks, we construct reconstruction dataset based on LLaVA-558K (Liu et al., 2024b). and also explore two key tasks in personalized medical image
2nd Stage: Heterogeneous H-LoRA Plugin Adaptation. The submodules of H-LoRA share the word embedding layer and output head but may encounter issues such as bias and scale inconsistencies during training across different tasks. To ensure that the multiple H-LoRA plugins seam- lessly interface with the LLMs and form unified base, we fine-tune the word embedding layer and output head using small amount of mixed data to maintain consistency in the model weights. Specifically, during this stage, all H-LoRA submodules for different tasks are kept frozen, with only the word embedding layer and output head being optimized. Through this stage, the model accumulates foundational knowledge for unified tasks by adapting H-LoRA plugins.
wtwprmeennttsuue---rrutttttoo and modal- ity conversion-using the IXI (Davies et al., 2014) and SynthRAD2023 (Thummerer al., 2023) datasets. In ad- dition to the above datasets, prehension task on the OmniMedVQA (Hu et al., 2024) and MMMU (Yue et al., 2024) benchmarks. Detailed data
further evaluate the com-
3rd Stage: Visual Instruction Fine- Tuning, In the third stage, we introduce additional task-specific data to fur-
6

---

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprchension and Generation
Table 2: The experimental results for the four modality conversion tasks.
CT MRI (Brain)
CT
MRI (Pelvis)
MRI CT (Brain)
MRI CT (Pelvis)
Model
SSIM f
PSNR f
MSE
SSIM f
PSNR f
MSE
SSIM f
PSNR
MSE
SSIM f
PSNR f
MSE
pix2pix
71.09
32.65
36.85
59.17
31.02
51.91
78.79
33.85
28.33
72.31
32.98
36.19
CycleGAN
54.76
32.23
40.56
54.54
30.77
55.00
63.75
31.02
52.78
50.54
29.89
67.78
BBDM
71.69
32.91
34.44
57.37
31.37
48.06
86.40
34.12
26.61
79.26
33.15
33.60
Vmanba
69.54
32.67
36.42
63.01
31.47
46,99
79.63
34.12
26.49
77.45
33,53
31.85
DifMMa
71.47
32.74
35.77
62.56
31.43
47.38
79.00
34.13
26.45
78.53
33.68
30.51
HealthGPT-MS
79.38
33.03
33.48
71.81
31.83
43.45
85.06
34.40
25.49
84.23
34.29
27.99
HcalthGPT-LL1
79.73
33.10
32.96
71.92
31.87
43.09
85.31
34.29
26.20
84.96
34.14
28.13
Table 3: Comparison results of super-resolution task.
et al., 2025), and Emu3 (Wang et al., 2024a). The experi- mental results are shown Table I, with the following key observations: SOTA Results Compared with LVLMs: In medical visual comprehension tasks, HealthGPT demon- strates superior performance, significantly outperforming both medical-specific models (e.g.. HluatuoGPT-Vision and general-purpost purpose models (e.g.. Llama-3.2), (ii) Surpass- ing Current Unified LVLMs: Despite being trained on billions of data points, unified models still exhibit poor gen- eralization performance in medical visual comprehension. For instance, Unified-IO 2 scored only 33.8. In contrast, HealthGPT-M33, with only 3.8B parameters, scored 61.3 the medical multimodal unified task, significantly out- performing existing unified models in medical downstream scenarios. (iii) Stable Improvement with Large Base Model: Our method demonstrates excellent scalability, with HealthGPT-L.I and HealtkGPT-XI.33 achieving a score of 66.4 and 71.1 in the larger model configuration. This result significantly outperforms all other models, highlighting the effecctiveness of scaling up the base model for enhanced performance in medical tasks.
Model
SSIM
PSNR
MSE
LPIPS
SRGAN
71.34
32.01
41.27
24.50
DASR
71.57
32.34
38.25
19.17
Real-ESRGAN
67.30
31.87
42.57
20.64
LIIF
73.27
32.13
40.14
22.93
BSRGAN
69.97
31.97
41.52
28.72
HealthGPT-M3
78.19
32.76
34.47
12.02
HealthGPT--114
77.94
32.71
35.19
12.43
selection and instruction templates are in the Appendix A.3.
Model Details. We select CLIP-LL/14 (Radford et al., 2021) as the visual encoder and used the hidden states of its second and penultimate layers as concrete- gancrete-graieed and abstract- grained features for dynamic hierarchical visual perception of models. Drawing on the successful experiences of LLLVA architectures, we employ MLP to align the multimodal feature embeddings. We choose the parameter-efficient phi- 3-mini (Abdin et al., 2024) and phi-4 (Abdin et al., 2024) as the base models. Additionally, we employ the larger-scale Qwen2.5 (Team, 2024b) to explore the scalability of the proposed method. For visual comprehension and generation tasks, we set the rank of H-LoRA to 16/8 and /4/32 for different base models, with four experts. Additionally, we use the 18-8192 version of VQGAN as the image indexing and upsampling module. The model parameters and training configurations are provided in Appendix A. and A.2.
It is noteworthy that some Med-LVLMs (Xie et al., 2025) perform continual fine-tuning and evaluation on the training sets of specific benchmarks, which may lead to structural degradation in model outputs and negatively impact gener- alization. To ensure fair comparison and reliable evaluation of generalization ability, we adopt unified experimental setting and only compare with models trained on diverse and comprehensive datasets.
5.2. Main Experiments
Generation. We study three key tasks in medical imag- ing. (i) Modality Conversion: In this task, we focus the conversion between CT and MRT modalities for the brain and pelvic regions, designing four specific sub- tasks. All comparative models (Pix2Pix (Isola et al., 2017), CycleGAN (Zhu et al., 2017), BBDM (Li et al., 2023a), Vmamba (Liu et al., 2024e), and DiffMa (Wang et al., 2024b)) trained a separate model for each sub-task, while HealthGPT unify all tasks into a single training process. The experimental results, shown in Table 2, demonstrate that our approach outperforms other methods across multi- ple evaluation metrics. For instance, in the CT2MRI-Brain
Comprehension. We compare HealthGPT with several existing models, including medical-specific LVLMs (e.g.. Med-Flamingo (Moor et al., 2023), LLaVA-Med (Li et al., 2024a), HuatuoGPT-Visinn (Chen et al., 2024a)) as well as recent open-world LVLMs (e.g., BLIP-2 (Li et al., 2023b), LLaVA-v1.5 (Liu et al., 2024b), InstructBLIP (Dai al., 2023), Yi-VI. (Young et al., 2024), V1.ernVI.2 (Chen et al., 2024b), Llama-3.2 (Dubey et al., 2024)). Additionally, we test several SOTA unified visual comprehension and genera- tion models, including Show-o (Xie et al., 2024), Unified-IO 2 (Lu et al., 2024), Janus al., 2024a), Janus-Pro (Chen
7

---

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprchension and Generation
Table 4: We present the performance and speed differences of LoRA, MoBLoRA (n=4), and H-LoRA (n=4) on medical visual comprehension and generation tasks.
Comp. Path
Gen.
Training Time
Model
VQA-RAD
SLAKE
MMMU -Med
OMVQA
RECOM
MTRANS
SR
close
all
close
all
close
all
+LoRA
71.3
57.2
70.0
53.4
76.4
38.6
41.30
65.10
62.67
59.99
65.88
1.00x
HealthGPT w/
+MoELoRA
72.5
57.2
66.4
52.4
73.2
36.0
39.30
64.90
67.31
59.76
65.91
1.49
+H-LoRA
73.7
55.9
74.6
56.4
78.7
39.7
43.30
68.50
67.69
60.30
66.14
1.00x
Table 5: Comparison between the H-LoRA-basce Three-Stage Learning Strategy and the mixed -exxedtrainnng approach.
Comp. VathVQA
Gen.
Training Strategy
VQA-RAD
SLAKE
MMMU Med
CT
MRI
OMVQA
close
all
close
all
close
all
Brain
Pelvis
Brain
Pelvis
Mixed-Traiee Mexxnd-r
56.6
37.9
45.0
32.9
65.7
33.6
44.0
48.9
65.64
62.75
56.61
50.77
HcalthGPT w/
Three-Stage Stain-Taiining
72.5
55.2
77.9
59.6
79.7
49.0
42.7
68.5
70.84
72.99
65.26
61.33
(a) Comp.
[b) Gen.
(a)
(b)
Comp Loss
54
64
3.3
3.0
60
Performance
52-
Performance
62
50
Reffrmancc
0.7
20
40
60
80
100.50
Gen. Tosk Loss
-0
20-
46
56
10-
32
64
128
128
256
512
LoRA
McELoRA
H-LoRA
Perception w/ Faustrcc
Perception. w/ Feoncoto
Figure 5: Performance comparison of LoRA, MoELoRA, and H-LoRA under different rank settings.
Figure 6: The loss visualization (a) and performance
parison (b) with respect to different visual perceptions.
task HealthGPT-M3 achieves an SSIM of 79.38, signifi- cantly surpassing traditional methods like Pix2Pix (71.09) and the recent DiffMa (71.47). (ii) Saper-Resolution. We conduct 4x super-resolution experiments on the IXI dataset, with the results presented in Table 3. Notably, most exist- ing methods fail to fully leverage the prior knowledge of key structures in medical images, resulting in significant shorteomings in detail recovery. In contrast, our method significantly mitigates this issue. Specifically, HealthGPT- M3 excels in key metrics such as SSIM, PSNR, and MSE, achieving scores of 18,19, 32.76, and 34.47, respectively. Additionally, HealthGPT-MS achieves the lowest perception distance of 12.34, further validating its exceptional perfor- mance in human visual perception. (iii) Reconstruction: We compare HealtkGPT-M3 with unified models with re- construction capabilities, such as Unified-IO 2 and SEED-X. The results show that approach performs better controlla- bility for visual reconstruction. Details are in the Appendix C.5.
the similar performance between the two models in Figure 2 and 3 meets our expectations. For efficiency considerations, most open-world unified models are limited more than 13B. In alignment with this practical constraint, we did not train the HealthGPT-XLL32 on generation tasks.
size of no
5.3. In-Depth Study
Effect of Heterogeneous Low-Rank Adaptation. H-LoRA provides optimized multi-LoRA architecture for multi- task learning. We conduct extensive validation of this struc- ture, with results presented in Table 4, comparing the per- formance of LoRA, MoELoRA, and H-LoRA in medical unified comprehension and generation tasks. In the majority of comprehension tasks and all generation tasks, H-LoRA demonstrates superior performance, particularly in the Om- niMedVQA QAA benchmark, where improved from 64.90 to 68.50. Notably, despite some applications of MoELoRA in certain scenarios, it do not show advantages in this task and had a training time approximately 50% longer than LoRA. Figure illlsstraees the performance of the three PEFT meth- ods medical visual comprehension and generation tasks
To explore the scalability of the proposed method gen- eration tasks, we also train HeaLLkG with similar number of trainable parameters to the M. version. Hence,
8

---

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprchension and Generation
(a) Normal
(b) Different Severity
(cI Different Areas
lungs, and puirennary
Purmonary adema decressed efuuton
Severe Dalm memorr adama plauro
gPt rehocardar atelectunis evertodd unchangod manitorinng dovioss.
Sabrl Increased light plouzal Incraasecbll plaural affuatin sightly porsiraat overiedd
pleunt eftaten
inffection unzi supportive devicos
Figure 7: Case study of report-to-CXR under different instructions. (a) shows a normal CXR image for comparison. (b) and (c) illustrate generated cases with varying severity and affected regions. The graffiti areas indicate abnormal conditions.
across different ranks, with H-LoRA consistently outper- forming the other methods in all scenarios, demonstrating significant advantages in handling diverse tasks.
6. Conclusion
In this paper, we introduce HealthGPT, a Med-LVLM that unifies medical vision-language understanding and genera- tion through novel heterogeneous knowledge adaptation approach. Experimental results demonstrate that HealthGPT achieves sigmificant performance improvements across mul- tiple medical understanding and generation tasks, showcas- ing its potential for healthcare applications.
Different Learning Strategy. We propose a three-stage learning strategy for H-LoRA that decouples comprehension and generation tasks. Unlike methods that train both tasks simultaneously, our approach reduces performance degrada- tion from task confticts (see Table 5). In the medical visual comprehension task, mixed training causes catastrophic for- getting and degrades visual reconstruction, whereas our strategy effectively uses the medical embedding knowledge in pre-trained LLMs to mitigate these confficts. Meanwhile, we examine how fusing heterogeneous H-LoRA plugins in the second training stage results in minimal performance degradation. Detailed results are in the Appendix C.3.
Acknowledgements
This work has been supported in part by the NSFC (No. 62402426. 62272411), the Key Research and Development Projects in Zhejiang Province (No. 2025C01 128, 2024C01 106, 2025C01030, 2025C02156). Ningbo Yongjiang Talent Introduction Programme (2023A- 400-G), Zhejiang University Education Foundation Qizhen Scholar Foundation.
Hierarchical Visual Perception Analysis. We conduct ablation analysis on visual perceptual inputs suitable for comprehension and generation tasks. Figure illustrates that the convergence efficiency for comprehension tasks is significantly higher with abstract-grained visual inputs com- pared concrete-grained inputs, whereas generation tasks perform better with concrete- grained inputs. This result further underscores the necessity of the hicrarchical visual perception we propose, which suggests that customizing visual inputs at different hierarchies for specific tasks can substantially enhance efficiency.
Impact Statement
The broader impact of HealthGPT, designed as a multimodal assistant for healthcare, holds the potential to significantly benefit a wide range of multimodal healthcare applications. Our research investigates the innovative, parameter-efficien fine-tuning approach, H-LoRA, which is combined with hierarchical visual perception and three-stage learning strategy. This method effectively integrates diverse com- prehension and generation knowledge into pre-trained large language models. Besides, the proposed VL-Health serves as fundamental basis for investigating medical vision- language comprehension and generation. We are confident in its superiority as state-of-the-art approach for medical multimodal comprehension and generation, and we antici- pate its substantial influence on future research directions in the field.
Report-to-CXh CXR Task. Text-to-image generation is com- mon component in unified models (Lee et al., 2023; Huang al., 2023; 2024); therefore, further extend our investi- gation to this task within the medical domain. We explore the Report-to-CXX CXR task without reference images, using small amount of CXR data (Johnson et al., 2019) for instruc- tion fine-tuning. Figure 7 annotates images with varying injury degrees and locations, comparing them to healthy CXR images. We observe that HealthGPT effectively gener- ates CXR images based the instructions, showcasing its potential in healthcare education and auxiliary diagnosis.

---

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprchension and Generation
References
Ding, N., Qin, Y., Yang. G., Wei, E, Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., et al. Parameter- efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-22 2023.
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A.. Bao, J., Behl, H., et al. Phi-3 technical report: A highly capable lan- guage model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
Dong, R., Han, C., Peng, Y., Qi, Z., Ge, Z., Yang, J., Zhao, L., Sun, J., Zhou, H., Wei. H., Kong, X., Zhang, X., Ma, K., and Yi, L. Dreamllm: Synergistic multimodal comprehension and creation. In The Twelfth Interna- tional Conference
Bae, S., Kyung. D., Ryu, J., Cho, E., Lee, G., Kweon, S., Oh, J., JI. L., Chang, E., Kim, T., et al. Mimic-ext-mimicce vqa: A complex, diverse, and acage-scale visual question answering dataset for chest x-ray images, 2024.
Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: http::///ppeeeeviieeee net forum?
Bansal, H., Israel, D., Zhao, S., Li, S., Nguyen, T., and Grover, A. Medmax: Mixed-modal instruction tuning for training biomedical assistants, 2025. URL https: //arxiv. org/ abs/ /2lss//422220 12661.
1d=y01KGvd9Bw: KGvd9Bw.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint 2007:2440.22/885 2024.
Chen, J., Gui, C., Ouyang, R., Gao, A., Chen, S., Chen, G. H., Wang, X., Cai, Z., Ji, K., Wan, X., and Wang, B. Towards injecting medical visual knowledge into multimodal Ilms scale. In Conference on Empirical Methods in Natural Language Processing, 2024a.
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision recognition, pp. 12873-12883, 2021.
vision and pattern
Ge, Y., Ge, Y., Zeng, Z., Wang. X., and Shan, Y. Planting a seed of vision in large language model. arXiv preprint arXiv:2307.0800
Chen, X., Wu, Z., Liu, X., Pan, Z., Liu, W., Xie, Z. Yu, X., and Ruan, C. Janus-pro: Unified multimodal understand- ing and generation with data and model scaling. arXiv preprint 2xXv:2501.17811 2025.
2023.
Ge, Y., Zhao, S., Zhu, J., Ge, Y., Yi, K., Song. L., Li, C., Ding, X., and Shan, Y. Seed-x: Multimodal models with unified multi granularity comprehension and generation. arXiv preprint arXiv:2404. 14396, 2024.
Chen, Z., Wang, W. Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al. How far are we to gpt-4v? closing the gap to commercial multimodal mod- els with open-source suites. Science China Information Sciences, 6111):220,, 2024b.
He, X.. Zhang, Y., Mou, L., Xing, E., and Xie, P. Pathvqu: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003. 10286, 2020.
Chern, E., Su, J., Ma, Y., and Liu, P. Anole: An open, autore- gressive, native large multimodal models for interleaved image-text generation. arXiv preprint arXiv:2407.06115, 2024.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zu,, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Hu, Y., Li, T. Lu, Q., Shao, W., He, J., Qiao, Y., and Luo, P. Omnimedvga: A new large-scale comprehensive evaluation benchmark for medical Ivlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22170 72-12283, 2024.
Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang. W., Li, B., Fung, P., and Hoi, S. C. H. Instructblip: Towards general-purposs vision-language models with instruction tuning. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M.. and Levine, S. in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, IA, USA, December 10 16, 2023, 2023.
Huang, K., Sun, K., Xic, E., Li, Z., and Liu, X. T2i- compbench: comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neu- ral Information Processing Systems, 36:78723-78747, 2023.
Davies, R. L., Royston, P. A., Leung, M. S., Haider, M. E. A. M. J., Barkhof, S. G. A. L., and B., P. E. T. M. The ixi dataset, 2014. URL https:/ deeeeoomeetttoogee /ix/iix-iaa data-ddttss/./... Ac- cessed: 2025-01-30.
Huang, Z., He, W., Long, Q. Wang, Y., Li, H., Yu, Z., Shu, F., Chan, L., Jiang, H., Gan, L., et al. T2i- factualbench: Benchmarking the factuality of text-to- image models with wwwwllldge-nntnssiie concepts. arXiv preprint arXiv:2412.043000, 2024.
10

---

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprchension and Generation
Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. A. Image-to- image translation with conditional adversarial networks. In Proceedings of the IEEE conference computer vi- sion and pattern recognition, pp. 1125-11333 2017.
IEEE 18th International Symposium on Biomedical Imag- ing (ISBI), pp. 1650-1654, IEEE, 2021.
Liu, D., Zhao, S., Zhuo, L., Lin, W., Qiao, Y., Li, H., and Gao, Lumina-mgpt: Illuminate flexible photorealis- tic text-to-image generation with multimodal generative pretraining. arXiv preprint arXiv:2408.02633 2024a.
Johnson, A. E., Pollard, T. J.. Greenbaum, N. R., Lun- gren, M. P., Deng, C.-y., Peng. Y., Lu, Z., Mark, R. G., Berkowitz, S. J., and Horng, S. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv: 1901.0704, 2019.
Liu, H., Li, C., Wu, Q. and Lee, Y. J. Visual instruction tuning. NeurIPS, 2023.
Lau, J. J., Gayen, S.. Ben Abacha, A., and Demner- Fushman, D. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018.
Liu, H., Li, C., Li, Y., and Lee, Y. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16296-263066 2024b.
Lee, T., Yasunaga, M., Meng, C., Mai, Y., Park, S., Gupta, A., Zhang, Y., Narayanan, D., Teufel, H., Bellagente, M.. et al. Holistic evaluation of text-to-image models. Advances in Neural Information Processing Systems, 36: 69981-70011, 2023.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. Llava-next: Improved reasoning, ocr, and world knowledge. https://
gituub. 2024c.
2024.
Liu, Q. Wu, X., Zhao, X., Zhu, Y., Xu, D., Tian, F,, and Zheng, Y. When moe meets 1lms: Parameter efficient fine- tuning for multi-task medical applications. In Proceed- ings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1104-1114, 2024d.
Li, B., Xue, K., Liu, B., and Lai, Y.-K. Bbdm: Image-to- image translation with brownian bridge diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern Recognition, pp. 1952-1961, 2023a.
Li, C., Wong, C., Zhang. S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., and Gao, Mava-med: Training large uunguage-andvvisin assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024a.
Liu, Y.. Tian, Y., Zhao, Y.. Yu, H., Xie, L.,, Wang, Y.. Ye, Q., Jiao, J., and Liu, Y. Vmamba: Visual state space model. Advances in neural information processing systems, 37: 103031-103063, 2024c.
Li C., Wong, C., Zhang. S., Usuyama, N., Liu, H., Yang. J., Naumann, T., Poon, H., and Gao, J. Llava mew: Training large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024b.
Lu, J., Clark, C., Zellers, R., Mottaghi, R., and Kembhavi, A. Unified-io: A unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference on Learning Representations, 2022.
Lu, J., Clark, C., Lee, S., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., and Kembhavi, A. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 2024.
Li, Li, D., Savarese, S., and Hoi, S. 2: Bootstrapping language-imann image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19744 19742. PMLR, 2023b.
Li, S., Lin, T., Lin, L., Zhang, W., Liu, J., Yang, X., Li, J., He, Y., Song, X., Xiao, J., et al. Eyecaregpt: Boost- ing comprehensive ophthalmology understanding with tailored dataset, benchmark and model. arXiv preprint arXiv:2504. 13650, 2025.
Luo, T., Lei, J., Lei, F., Liu, W., He, S., Zhao, J., and Liu, K. Moelora: Contrastive learning guided mixture of ex- perts on parameter -aaumtee--fffcient fine-tuning for large language models. arXiv preprini arXiv:2402.1 2851 2024a.
Lin, T., Liu, J., Zhang, W., Li, Z., Dai, Y., Li, H., Yu, Z., He, W., Li, J., Jiang, H., et al. Teamlora: Boosting low- rank adaptation with expert collaboration and competition. arXiv preprint arXiv:2408. 200860000080 2024.
Luo, Y., Zhang, J., Fan, S., Yang, K., Hong, M., Wu, Y., Qiao, M., and Nie, Biomedgpt: An open multimodal large language model for biomedicine. IEEE Journal of Biomedical and Health Informatics, 2024b.
Liu, B., Zhan, L.M Xu, L., Ma, L., Yang, Y., and Wu, X.- M. Slake: A semantically-Iabelle knowledge-enhance dataset for medical visual question answering. In 202/
Masoudnia, S. and Ebrahimpour, R. Mixture of experts: a literature survey. Artificial Intelligence Review, 42: 275-293, 2014.

---

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprchension and Generation
Moor, M., Huang, Q., Wu, S., Yasunaga, M., Dalmia, Y., Leskovec, J., Zakka, C.,, Reis, E. P., and Rajpurkar, P. Med-flamingo: multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pp. 353-367. PMLR, 2023.
narrative review. Quantitative Imaging in Medicine and Surgery, 14(1): 1088, 2023.
Tong, S., Fan, D., Zhu, J., Xiong, Y., Chen, X., Sinha, K., Rabbat, M., LeCun, Y., Xie, S., and Liu, Z. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint arXiv:2412. 14164, 2024.
Nath, V., Li, W. Yang, D., Myronenko, A., Zheng, M., Lu, Y., Liu, Z., Yin, H., Law, Y. M., Tang, Y., et al. Vila-m3: Enhancing vision-language models with medical expert knowledge, arXiv preprint 2..i::2411.199/11 2024.
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-.., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. Towards generalist biomedical ai. NEJM AI, 1(3): Aloa2300138, 2024.
Ong, W., Zhu, L., Zhang, W., Kuah, T., Lim, D. S. W., Low, X. Z., Thian, Y. L., Teo, E. C.. Tan, J. H., Kumar, N., et al. Application of artificial intelligence methods for imaging of spinal metastasis. Cancers, 14(16):4025, 2022.
Vig, J. A multiscale visualization of attention in the trans- former model. arXiv preprint arXiv: 1exiv:/990... 0eteii906007777 05.7 257/4, 2019.
Wang. X., Zhang, X., Luo. Z., Sun, Q. Cui, Y., Wang, J., Zhang, F., Wang, Y., Li, Z., Yu, Q. et al. Emu3: Next-token prediction is all you need. arXiv preprint MKKV:2409.18866 2024a.
OpenAl.
Gpt-4v(ision) system card. https: //edn. wwwnai. wwtpp/ppes///Pt. Syste__ssemmm_ fard.pdf. 202.
Pan, K., Tang, S., Li, J., Fan, Z.., Chow, W., Yan, S., Chua, T.-S., Zhuang, Y., and Zhang, H. Auto-encoding morph- tokens for multimodal Ilm. In Proceedings of the 41st In- ternational Conference on Machine Leurning, 1CML:24. JMgR.org, 2024.
Wang. Z. Wu, Z., Agarwal, D., and Sun, J. Medelip: Con- trastive learning from unpaired medical images and text. Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods Natural Language Processing, 2022:3876 3887, 2022.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learming, pp. 8748-8763, PMLR, 2021.
Wang. Z., Zhang, L., Wang, L., and Zhang, Z. Soft masked mamba diffusion model for ct to mri conversion. arXiv preprint arXiv:2406. 15910, 2024b.
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. Large language models encode clinical knowledge. Nature, 620(7972):122-180, 2023.
Wu, C., Zhang, X., Zhang, Y., Wang, Y., and Xie, W. Towards generalist foundation model for radiology by leveraging web-scale 2d3d medical data, 2023. URL https:// wwwp:iiwwww./aa///e//e/e/oesse/t 2308. 02463.
Wu, C., Chen, X., Wu, Z., Ma, Y., Liu, X., Pan, 7., Liu, W., Xie, Z., Yu, X., Ruan, C., and Luo, P. Janus: Decoupling visual encoding for unified multimodal understanding and generation, 2024a. URL https: htapps://aaxiv..oo abs/2410.
Team, C. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.0988/. 2024a.
Team, Q. Qwen2.5: A party of foundation models, Septem- ber 2024b. URL attps://qwenlm.gg github...ttt........m.. /o// blog/
qwen/.wen22.. blog/qwen2. 5/.
Wu, S., Fei, H., Qu, L., W., and Chua, T.SS. Next-gpt: Any-to-any multimodal 11m. In Forty-first International Conference on Machine Learning, 2024b.
Thawakar, O., Shaker, A. M., Mullappilly, S. S., Cholakkal, H., Anwer, R. M., Khan, S. S., Laaksonen, J., and Khan, S. Xraygpt: Chest radiographs summarization using large medical vision-language models. In Workshop on Biomedical Natural Language Processing, 2023.
Xie, J., Mao, W., Bai, Z., Zhang, J., Wang, W., K. Q., Gu, Y., Chen, Z., Yang, 7., and Shou, M. Z. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.
Thummerer, A., van der Bijl, E., Galapon Jr, A., Verhoeff, J., Langendijk, A., Both, S., van den Berg, C. N. A., and Maspero, M. Synthrad2023 grand challenge dataset: Generating synthetic et for radiotherapy. Medical physics, 50(7):4664-466 2023.
2024.
Xie, Y., Zhou, C., Gao, L., Wu, J., Li, X., Zhou, H., Liu, S., Xing, L.,, Zou, J., Xie, C., and Zhou, Y. Medtrinity- 25m: A large-scale multimodal dataset with multigran- ular annotations for medicine. In The Thirteenth Inter- national Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net,
Tian, D., Jiang, S.. Zhang, L., Lu, X., and Xu, Y. The role of large language models in medical image processing: a
12

---

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprchension and Generation
2025. URL http:s///ppereeveee..etetttttt id=IwgmgidYPS.
Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G. Li, H., Zhu, J., Chen, J., Chang, J., et Yi: Open foundation models by 0l. ai. arXiv preprint AXiv:24433046652 2024.
Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
Zhang, W., Zhu, L., Hallinan, J.. Zhang, S., Makmur, A., Cai, Q., and Ooi, B. C. Boostmis: Boosting medical image semi-supervised learning with adaptive pseudo labeling and informative active annotation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
2022.
Zhang, W., Lin, T., Liu, J., Shu, F., Li, H., Zhang, I.., Wang- gui, H., Zhou, H., Lv, Z., Jiang, H., et al. Hyperllava: Dy- namic visual and language expert tuning for multimodal large language models. arXiv preprint arXiv:2403.1344, 2024a.
Zhang, W., Lv, Z., Zhou, H., Liu, J.-W., Li, J., Li, M., Li, Y., Zhang, D., Zhuang, Y., and Tang, S. Revisiting the domain shift and sample uncertainty in multi-source active domain transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 117516116766 2024b.
Zhou, H., Liu, F., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y. Chen, S., Zhou, P., Liu, J., et al. A survey large language models in medieine: Progress, application, and challenge. arXiv preprint arXiv:2311.05/ 2023.
Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. Unpaired image-to-image translation using cycle-consistent adver- sarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 22322, 2017.
13

---

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprchension and Generation
Appendix
This is the Appendix for "HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation". This Appendix is organized as follows:
Section A presents the experimental implementation details, the training process of HealthGPT, and the specifics of
VL.-Heltth.
Section B systematically provides an analysis of Heterogeneous Low-Rank Adaptation.
Section C shows supplementary experimental results validate the effectiveness of HealthGPT.
A. Implementation Details
A.1. Model Details
We employ CLIP-L/14 (Radford et al., 2021) the visual feature extractor, extracting both shallow and deep features to serve visual tokens. The model uses alignment adapters, implemented with two-layer MLPs, align shallow features, representing concrete visual granularity, and deep features, representing abstract visual granularity. These visual tokens are concatenated with text tokens and input into the Large Language Models (LLMs).
The development of MLLMs has undergone several paradigm shifts (Zhang et al., 2024a: OpenAl, 2023; Liu et al., 2024c; Chen et al., 2024b); however, the LLaVA architecture (Liu al., 2023) has gained widespread attention due to its simplicity and portability. Therefore, we adopt this architecture in work. HealthGPT offers three versions: HealthGPT-M3., HealthGPT-LI4, and HealthGPT-XL33,, which are based on Phi-3-mini (Abdin et al., 2024), Phi-4 (Abdin et al., 2024), and Qwen2.5-32B (Team, 2024b) as the pre-trained base models, respectively. In addition, expand the LLM vocabulary with 8192 VQ indices derived from VQGAN-f8-4 18-88888222 (Esser et al., 2021), serving as multi-modal tokens to further augment the capacity of models for understanding both visual and textual input and output. Figure 6 shows the details.
Table 6: Overview of the Components of HealthGPT.
Model
ViT
Adapter
MLP Dims
Model Dims
Base Model
Params
Vocab Size
H-LoRA Rank
HealthGPT-Mi
CLIP-L/14
2-layer MLP
1024
3072
Phi-3-mini
3.8B
40206
16(Cop..). 64(Genn.
HealthGPT-LI4
CLIP-L/14
2-layer MLP
1024
5120
Phi-4
14B
108547
8(Cop..) 32(Gen.)
HealthGPT-XI.2
CLIP-L/14
2-layer MI.P
1024
5120
Qwen2.5-32BB
32B
152064
((Copp.) 32(Gen.)
A.2. Training Details
In this study, propose three-stage learning strategy that is compatible with innovative heterogeneous low-rank adaptation (H-LoRA). We provide detailed hyperparameter configuration for the model's three-stage training process. The specific hyperparameter settings used are listed in Table 7. These hyperparameters are crucial for ensuring the model's learning efficacy and final performance.
Table 7: Overview of Hyperparameter Configurations.
HealthGPT-M3
HeatthGPT-LI4/XL33
Hyperparameter
Stage-1
Stage-2
Stage-3
Stage-1
Stage-2
Stage-3
Comp.
Gen.
Comp.
Gen.
Comp.
Gen.
Comp.
Gen.
Comp.
Gen.
Comp.
Gen.
Optimizer
AdamW W
AdamW
AdamW
AdamW
Adam
AdamW
Adapter LR
le-3
2e-5
2e-5
2e-5
le-3
2c-5
2e-5
20-5
Learning Rate
/
2c-4
2e-4
2c-4
le-4
2e-4
2e-4
Global Batch Size
256
64
32
128
64
256
64
32
128
64
Weight Decay
0
0
Dropout Rate
0
0.05
0.05
0.05
0.05
0.05
0.05
LR Scheduler
Warm Up
Constant 2048
Warm Up 2048
Warm Up
Constant
Warm Up 2048
Max Sequence Length
2048
2048
2048
14

---

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprchension and Generation
aa)
(b)
Number of datusets for comprehension tasks
Number of datasets for generation tasks
8ow
Namber of QA Pains
Image Pairs for Super Resolution Tisks
Text- Image Pairs
of Images
CT MRL Pairs for Modality Conversion Tasks
60w
60w
Image Pairs for Reconstruction Tasks
40W
40W
20W
20W
VQA-RAD
SLAKE
Patl- Patl-YQAA VOA
MIMIC-CXR-vRR
LlaVi-MMd
PubMeeVisiin
IXI
MIMIC-CHEST-XRAY 1lava-558k SymthRAD2023
Figure 8: VL-Health dataset collection distribution.
It is worth noting that we sometimes observe instances of loss spikes during the training of medical visual comprehension and generation tasks. Through repeated validation, we discovered that larger model parameters and learning rates tend to lead to this issue, which is the reason for the slight differences in hyperparameters between HealthGPT-M3 and HealthGPT. LI4/XL32.
A.3. VL-Health
The construction of the VL-Health dataset involves two key steps: (i) data collection, (ii) data processing, as detailed below:
Data Collection: During the collection phase, we carefully considered the diversity of medical images and the complexity of the tasks, selecting appropriate subsets for comprehension and generation tasks. For comprehension tasks, we selected datasets such VQA-RAD (Lau et al., 2018), SLAKE (Liu et al., 2021), PathVQA (He al., 2020), and MIMIC- CXR-VQA (Bae et al., 2024), which cover various medical imaging modalities like radiology and pathology, and include professional annotations to assist the model in learning tasks such as lesion detection and disease diagnosis. Additionally, large-scale multi-modal datasets like VMeed (Li et al., 2024a) and PubMedVisson (Chen et al., 2024a) were included to provide broader medical knowledge support and facilitate the training of complex reasoning tasks. For generation tasks, we focused on four mainstream task categories: super-resolution image generation, modality conversion, text- to-image generation, and image reconstruction. The IXI (Davies al., 2014) dataset, containing a large number of healthy brain MRI images, is suitable for training super-resolution models; the MIMIC-CHEST-XRAM (Bae al., 2024) dataset, with X-ray images and their corresponding textual reports, is appropriate for text-to- text-to-image generation tasks; the SynthR AD2023 (Thummerer et al., 2023) dataset provides large number of paired CT and MRI images, supporting modality conversion model training; for image reconstruction tasks, we rewrote and adjusted the LLaVA-558k (Liu et al., 2024b) dataset.
Data Processing: After data collection, we performed filtering and processing of the raw data. For VisualQA tasks, we standardized the data entries into two forms: open-ended questions and single-choice questions, enabling flexible training and evaluation. Additionally, considering that multi-image data has minimal impact performance but introduces extra padding and training time, excluded multi-image data. For the scanned image data in generation tasks, applied slicing extraction, image registration, data augmentation, and normalization to treat 2D images as visual inputs for model training or used VQGAN-generatte indices to supervise the generation tasks.
1. DATA STATISTICS
This section provides detailed statistical information about the VL-Heellth dataset to offer a more comprehensive understand- ing.
Data Overview; To ensure balanced development of the model's comprehension and generation capabilities, in addition to the L.LVVA-558k and PubMedVision-PPT datasets used for alignment, the VL-Hea 1 th dataset ultimately selected 765,802 additional visual question-answertt (VQA) training samples (to endow the model with visual comprehension and inslluction-following capabilities) and 783,045 generation training samples (to provide the model with reconstruction and visual generation instruction-folloot following abilities). This contributes to the transfer of knowledge between comprehension and generation tasks, enhancing the model's overall performance. For medical image comprehension tasks, images were
15

---

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprchension and Generation
selected from VQA-RAD (approximately 450 images), SLAKE (approximately 630 images), Path VQA (approximately 2,600 images), MIMIC-CXR-VOL VQA (approximately 52,000 images), LLa (approximately 61,000 images), and PubMedVision (approximately 500,000 images), Multiple question-answer pairs were retained for each image to enhance the model's understanding and generalization of the image content. Table 8 shows the data distribution of VL-Health for three-stage learning strategy, where mixed-47k is based on the sampling of all data in stage-1.
Diversity and Quality Assessment: VL-Heellth covers 11 modalities, including CT, MRI, X-ray, microscopy, OCT, ultrasound, and fundus photography, which aids the model in learning features from various modalities. The dataset also encompasses a wide range of discases, from common to rare, and from localized lesions systemic discases, including pulmonary diseases, skeletal abnormalities, brain lesions, tumors, cardiovascular diseases, and cellular abnormalities. This provides comprehensive training support to the model, enabling it to learn the characteristics and diagnosis of various diseases.
Table 8: Data distribution of VI.-Hea 1 th in stree-stage learning strategy.
Medical Task
Stage-1
Stage-2
Comp.
LLaVA-558k, PubMedVision-PPe
Mixed-47k
Gen.
I.L.VA-558
Medical Task
Stage-3
Comp.
LLaVA Med, MIMIC CXR VQA, PubMedVisionnFT, LLaVA-665k, PathVQA, SLAKE, VQA-RAD IXI, SynthRAD2023, MIMIC-CHEST-XRAY
Gen.
A.3.2. DATA FORMAT.
All data samples are converted into a unified instruction-response format for training and evaluation. Specifically, the VL-Health dataset consists of the following components:
Task Type: Specifies the granularity of visual features output by the visual encoder and selects the corresponding H-LoRA submodule. For generation tasks, the response also includes multi-modal tokens corresponding to vQ indices.
Task Instruction: Guides the model interpret the image and generate response, covering various aspects of the image and specifying the output format.
Response: The textual output generated based on the task instruction and input image, ensuring it meets the question and formatting requirements.
Input Image: Provides the visual signal for the model process.
Target Image Index: In generation tasks, this is added as a multi-modal token to the response for autoregressive generation.
B. Analysis of Heterogeneous Low-Rank Adaptation
We propose H-LoRA, which utilizes hard routing selection to allocate plugins for knowledge learning and representation across tasks, thereby preventing conflicts arising from heterogeneous knowledge. This is especially beneficial for domain adaptation, where task and distribution shifts often cause interference between domains (Zhang et al., 2024b). Furthermore, within each task, we optimized based on MoELoRA, enhancing performance while reducing computational overhead. The pseudocode detailed Algorithm 1.
We further analyzed the computational overhead differences between MoELoRA and H-LoRA. Assuming that both methods use the same number of LoRA experts k. we can compare their time complexity from the perspective of the operational steps involved.
Computational Overhead of MoELoRA. In MoELoRA, the operations involving the expert matrix mainly include the following steps: (i) Expert Multiplication: MoELoRA requires 2k multiplications with the LoRA experts. (ii) Router Multiplication: One multiplication with the Router is required. (iii) Router Output Expansion: MoELoRA needs to
16

---

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprchension and Generation
Algorithm H--LORA Algorithm
Input: concrete-grainet visual features F.Con, abstract-grained visual features JFAbs, comprehension--asel H-LoRA
modules (14; ([AA,opp)) 1/k
1 Ji=1: R (enee)) ). generation-based H-LoRA modules ((4,)m Ji=1: Roiter), task type (comprehension
or generation), number LoRA experts k, origin linear layer weights Wo. text features T, hidden state
Output: final output (
// Select task-specific image features
if T generation task then
Fimg &- FC
else T = comprehension task then
Fimg f- J
end
U concat(Fiimg T) // Concatenate image features and text features
{A.) ): Roster [ATT) (B,T
y. R // Assign task-specific H-LoRA submodule
// Merge LoRA experts' matrices
A merged + concat( [A
B merged - concat( {B, )
W + R(h // Generate routing weights based on input hidden state x
Wexppanded +- a x W/r so 1, Expand routing weights to match merged matrices
OlL-LORA t (x A merged Wespanded). Buergad Compute H-LoRA output using element-wise multiplication
O Wo + OH-ROAA // Add H-LoRA output to pre-trained weights to get final output
Return O
perform & expansion operations on the Router's output weights to generate the appropriate shapes that match the dimensions of the input and LoRA experts while iterating through the experts. (iv) Dot Product: For each expanded Router weight, dot product with the intermediate state of the expert is required, resulting in k multiplications. (v) Addition: Finally, A: addition operations are required to accumulate the results from each LoRA expert into the final output. Assuming the time complexity of each operation is the same, the additional time complexity introduced when equipping a fully connected layer with MoELoRA is: O(2k + 1 + + k) = (5k + 1). Thus, MoELoRA introduces an additional time overhead of (5k: + 1) during computation.
In contrast to MoELoRA, H-LoRA reduces the computational overhead by concatenating the LoRA expert matrices. Specifically: (i) Expert Multiplication: H-LoRA merges all LoRA experts by directly creating a larger A and B matrix, instead of performing independent operations for each expert. This process can be implemented through matrix initialization without additional concatenation operations. Therefore, only 2 multiplications with the LoRA experts are required. (ii) Router Multiplication: H-LoRA still requires one multiplication with the Router. (iii) Router Output Expansion: H-LoRA only requires one expansion operation on the Router's output weights. (iv) Dot Product: H-LoRA only requires one dot product between the Router's output and the expert's intermediate state. (v) Addition: Finally, H-LoRA only requires one addition operation to accumulate the LoRA expert results into the intermediate state. Therefore, +) (66).
H-LoRA.
the additional time complexity introduced by H-LoRA is: O + 1+
Comparing the two, we see that MoELoRA introduces a linear increase in additional time complexity with respect to the number of experts k; resulting in a complexity of (5kk: + 1), while H-LoRA's additional time complexity is fixed at (6)), independent k. We observe that when l: is small, the time complexity differences between MoELoRA and H-LoRA are negligible. However, as & increases, MoELoRA's computational overhead grows linearly, while H-LoRA's remains constant. This makes H-LoRA significantly more computationally efficient than MoELoRA, particularly large-scale tasks. We will further demonstrate the significant advantage of H-LoRA in training time in subsequent experiments, validating its efficiency in practical applications.
C. Supplemental Experimental Results
In this section, we include additional experiments to demonstrate the superiority of HealthGPT and articulate our design philosophy.
17

---

HcalthGPT: A Medical Large Vision-Language Model for Unifying Comprchension and Generation
C.1. OmniMedVQA Benchmark
OmniMedVQA (Hu al., 2024) novel, large-scale medical visual question answering (VQA) benchmark designed encompass various modalities and anatomical regions by collecting diverse images from multiple medical datasets. Our experimental results are presented in Table 9.
Table 9: Performance comparison of OmniMedVQA Benchmark.
Medical LVLM
OmniMedVOA
Type
Model
Params
CT
X-ray
FDM
MiS
OCT
MRI
USS
Avg.
Med-Flamingo
8.3B
30.1
33.9
25.5
37.0
60.0
27.6
30.4
34,9
LLaVA-Med
7B
28.4
32.8
42.7
31.6
55.3
45.0
53.6
41.3
HuatuoGPT-Viste
35.3
41.5
51.4
62.3
59.3
40.4
60.1
50.0
BLIP-2
6.7B
X
26.6
29.1
22.3
36.9
29.1
22.7
21.4
26.9
Comp. Only
LLaVA-vI.5
7B
28.0
55.7
35.5
42.1
49.2
52.9
49.7
44.7
InstructBLIP
7B
20.1
22.2
34.1
30.6
38.6
31.9
25.5
29.0
Yi-VL
6B
x
51.2
47.1
27.7
62.6
67.6
55.0
40.3
50.2
Intern
40.2
57.9
53.2
64.0
59.1
49.1
54.5
Llama-3.2
11B
37.6
55.2
71.4
82.1
62.5
65.2
68.6
63.2
Show-o
1.3B
29.0
50.4
30,9
22.0
30.8
34.2
33.8
33.0
Unified-IO
10.8
37.7
12.3
25.3
32.6
30.9
37.7
26.8
Janus
1.3B
24.9
54.8
35.9
62.7
54.2
50.7
36.8
45.7
Janus-Pro
7B
31.2
61.2
51.4
71.7
60.2
57.2
45.8
54.1
Comp. & Gen.
Emu3
8B
34.0
31.2
34.5
34.8
78.3
51.7
12.3
39.5
HcalthGPT-M3
3.8B
35.3
81.9
54.6
88.2
89.3
78.5
51.4
68.5
HealthGPT-L114
39.0
86.6
88.6
99.7
80.9
62.2
74.4
HcalthGPT-XL33
32B
66.4
81.3
67.7
85.6
98.7
76.3
64.5
77.2
Through our analysis, we make the following observations: (i) HealthGPT-M3 outperforms other models in out of 7 sub-tasks, achieving an average score that exceeds cutting-edge medical Large Tision-Languee Models (Med-LVLMs) as well as general LV; (ii) the unified model demonstrates relatively weak performance on OmniMedVQA: however, our approach effectively mitigates performance degradation caused by generation tasks, serving as unified model; (iii) HealthGPT-LI4 and [lealthGPT-XL32 excel across all sub-tasks, achieving optimal or near-optimal results with average score of 74.4 and 77.2, significantly surpassing other models.
C.2. Stability Analysis of Number of Experts
We investigated the impact of the number of LoRA experts on model performance within multi-LoRA architecture, conducting extensive experiments are presented in Table 10. As the number of experts increases, the training time for MoEL oRR is significantly prolonged. When n 8, the training time for MoELoRA is twice that of LoRA, whereas H-LoRA incurs no additional training delay and performs better. is estimated that at n = 32, the training time for MoELoRA could reach eight times that of LoRA, preventing it from completing training and inference. This result aligns with the analysis in Appendix B, indicating that H-LoRA not only avoids introducing additional training delays compared to LoRA but also outperforms MoELoRA.
MoELoRA and H-LoRA with varying numbers of experts. The experimental results
Table 10: explored the performance of MoELoRA and H-LoRA with different numbers LoRA experts. At n = 32, MoELoRA was unable to complete training.
n=2 Gen.
n=4 Gen.
n= Gen.
n=32 Gen.
Model
Comp.
Time
Comp.
Time
Comp.
Time
Comp.
Time
MMoBLoRA
50.3
62.98
1.22x
50.0
64.33
1.49
50.8
63.71
2.09 x
5.81
HealthGPPT w/
+H-LoRA
51.5
63.48
0.99
52.8
64.71
1.00
53.6
64.98
0.99
53.5
64.74
1.01x
18

---

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprchension and Generation
C.3. Impact of Heterogeneous Knowledge Fusion on Performance
Traditional unified models often utilize mixed training methods, which may result in performance degradation due variations in task modes. To address this, propose three-phase learning strategy to support H-LoRA, effeetively mitigating inter-task conflicts. Specifically, the second phase (Heterogeneous H-LoRA Plugin Adaptation) integrates LLMs with different H-LoRA plugins into new unified foundation by mixing the training of the embedding layers and output heads for two tasks. Figure 9 illustrates the impact of this phase on the performance of medical comprehension and generation tasks. We observe that the second phase effectively unifies the model with minimal impact on overall performance, significantly alleviating the conflict issues arising from mixed training medical scenarios.
C.4. Human Evaluation.
We further conduct human evaluation on the VQA-RAD, SLAKE, and PathVQA benchmarks, which contain 1,000 open- ended questions. Specifically, we recruit clinicians rank the randomly shuffled responses from HealthGPT-III4 LLaVA-Med, SitttooPT-Visioon Llama-3.2, InterVL--2 and Show-o. During the evaluation, questions were randomly selected, and the model-generated responses were anonymized and ranked. The results, shown in Figure 10, indicate that HealthGPT was frequently selected as the best answer. This suggests that HealthGPT has further application potential in medical care scenarios.
C.5. Reconstruction Performance
Currently, unified models that align visual features based recon- struction tasks include Pre-I.VIMMs post-LVI.Ms, well Unified- 1 (Lu et al., 2024) and SEED-X (Ge et al., 2024). To investigate the controllability of visual generation in rigorous settings such as medical contexts, we evaluated the performance of these models in medical image reconstruction in Table 1. Experimental results demonstrate that HealthGPT exhibits the most stable reconstruction performance with small amount of data.
70
65.7
67.7
60
Performancer
40
30
C.6. Case Study
Figures 1I and 12 illustrate examples of modality transformation and super-resolution reconstruction. In Figure 11, the results generated by our method in the CT (MRI) to MRI (CT) transformation task are highly close to the ground truth, effectively guiding the model in the transformation across different regions. For the MRI super- resolution reconstruction task, Figure 12 demonstrates the accuracy of our method in restoring scan image details, accurately reconstructing the essential details of the image.
10-
Comp. pre-stage-2
Gen. post-stage-2
Figure 9: Performance changes before and after the stage-2.
Table 11: The experimental results for the four reconstruction tasks.
CT(Brann) PSNR
CT(Pelvis) PSNR
MRI (Brain)
MRI(Pelvis)
Model
SSIM
MSE
SSIM
MSE
SSIM t
PSNR
MSE
SSIM 1
PSNR
MSE
SEED-X
20.18
27.
112.11
21.53
28.02
102.87
27.62
112.86
27.89
106.21
Unified-IO
83,93
36.09
17.95
85.36
35.10
25.46
87.50
34,25
25.47
86.31
33,53
29.80
HealthGPTT-M3
91.73
36.42
15.46
94.26
37.30
12.53
88.76
33.97
27.05
84.40
33.11
32.62
19

---

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprchension and Generation
(a)
(b)
5.62
HealthGPT
9.16
SLACE
34.08
LLaVA-Med
VQA-RAD
HuatuoGPT-Vision
Human Evaluation
Llama-3.2
InternVL-2
21.94
15.87
Show-o
Figure 10: (a) Proportion of model responses selected the best in human evaluation. (b) Human Evaluation Dataset.
MRI
Pred-CT
GT-CT
MRI
Pred-CT GT-CT
CT
Pred-MRI GT-MRI
CT
Pred-MRI GT-MRI
Figure Case of modality transfer.
20

---

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprchension and Generation
LR
LIIF
DASR
Ours
GT
Figure 12: Case of MRI image super-resolution.
21


Reviews:
[{"review_id": "OM3XfJS2yh", "reviewer": "Reviewer_wUUr", "summary": "This paper presents HealthGPT, a medical large vision-language model (Med-LVLM) that unifies both comprehension and generation of medical images. The key contributions include: \n1. Heterogeneous Low-Rank Adaptation (H-LoRA);  \n2. Hierarchical Visual Perception ; \n3. Three-Stage Learning Strategy; \n4. A curated multi-modal dataset covering 7 comprehension tasks (VQA, medical reasoning, pathology) and 5 generation tasks (modality transformation, super-resolution, image reconstruction); \n5. The model outperforms existing Med-LVLMs (e.g., LLaVA-Med, Med-Flamingo, HuatuoGPT-Vision) and achieves state-of-the-art (SOTA) results across multiple medical imaging benchmarks.\n\nWhile the work is technically solid and introduces meaningful innovations, its improvements in VQA performance are marginal, and the evaluation of image generation lacks perceptual metrics.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the detailed and thoughtful feedback, which greatly helped us refine our experiments and methodology, and better highlight HealthGPT’s innovation and practical value.  Below are our point-by-point responses.\n## 1. VQA Analysis\n**1.1 Comparison with LLaVA-Med++**\n\nWe wish to clarify that **LLaVA-Med++ is not a chat LVLM and is fine-tuned on individual downstream tasks**, which compromises its generalization ability. For fairness, we use LLaVA-Med++'s training setting and the results are below:\n\nMethod|VQA-RAD|SLAKE|PathVQA\n-|-|-|-\nLLaVA-Med++|86.0/77.1|85.3/80.8|98.9/58.7\nHealthGPT-M3|85.6/71.9|85.9/81.7|99.1/67.2\n\n**1.2 Comparison experiments**\n\nNotably, we further release the **HealthGPT-XL** with a stronger LLM (Qwen2.5-32b). **We believe this model is the new SOTA Med-LVLM compared with baselines**:\n\nMethod|VQA-RAD|SLAKE|PathVQA|MMMU|OMVQA\n-|-|-|-|-|-\nchameleon|45.4|54.3|54.0|25.7|18.9\nJanus-Pro|62.9|51.3|48.9|54.1|32.7\nRadFM|58.3|34.4|58.4|31.3|36.2\nLLaVA-Med++ (SFT)|64.7|87.1|55.1|x|x\nMed-MoE|66.9|52.6|69.1|32.7|55.8\nHealthGPT-M3|73.7|74.6|78.7|43.3|68.5\nHealthGPT-XL|79.1|85.7|92.4|58.0|77.2\n\nThe aforementioned results further demonstrate HealthGPT's effectiveness.\n\n**1.3 Failure case**\n\nAfter analysis, the failure cases stem from two sources:\n1. Broad questions (e.g., 'What is present?')\n2. Same question, different answers (Q → A1 vs. Q → A2)\n\nAddressing (1) and (2) typically requires fine-tuning on the corresponding training set. However, our model still achieves the best performance through strong generalization and instruction-following capabilities.\n\n**1.4 VQA qualitative**\n\nWe visualize heatmaps for different questions and masked key regions to assess answer quality: https://anonymous.4open.science/r/HealthGPT-9533/visualization.png. Results confirm our model relies on critical regions for reasoning.\n\n## 2. H-LoRA\n**2.1 Performance of H-LoRA**\n\nWe believe the reviewer may have misunderstood—we did compare performance with MoELoRA. Fig.5, Tab.4, and Tab.10 in our paper systematically demonstrate the **performance and efficiency advantages of H-LoRA**.\n\n**2.2 H-LoRA better than LoRA**\n\n- **Efficiency.** Existing research confirms that LoRA with MoE and router mechanisms improves performance for medical tasks [1]. To enhance efficiency, we propose internalizing LoRA experts as matrix space separation: $\\sum ka w_i A_i B_i / r \\rightarrow ka A \\odot \\mathcal{W} B / r$.\n\n- **Effectiveness.** We find the conventional router mechanism reduces the expected scaling factor from $a/r$ to $a/(rk)$, leading to performance degradation [2]. We correct the scaling factor and experimentally validate its effectiveness in our paper. Additionally, we included an ablation study on the router module:\n\nMethod|VQA-RAD|SLAKE|PathVQA|MMMU|OMVQA\n-|-|-|-|-|-\nHealthGPT-M3 w/o router|72.2|70.7|76.2|38.0|66.4\nHealthGPT-M3|73.7|74.6|78.7|43.3|68.5\n\nWe hope the above analyses and experiments further demonstrate the potential of H-LoRA in the medical domain.\n\n## 3. Image Metrics\n**3.1 LPIPS & FID**\n\nWe appreciate the reviewer’s concern about perceptual consistency in images. As requested, we’ve added LPIPS and FID metrics (https://anonymous.4open.science/r/HealthGPT-9533/lpips_fid.png), showing HealthGPT’s strength in both pixel-level and perceptual quality.\n\n**3.2 Human evaluation**\n\nWe also conduct human evaluation of the modality conversion task across five dimensions, comparing against the best-performing BBDM method. H-LoRA achieves **higher average scores** (4.30/4.52 vs. BBDM's 3.54/4.16):https://anonymous.4open.science/r/HealthGPT-9533/human_eval.png.\n\n## 4. Other Issues\n**4.1 Mutual Gains between Comprehension and Generation**\n\nWe would like to respectfully clarify that our work aims to **mitigate the significant conflicts between comprehension and generation tasks**, rather than assuming inherent complementarity. Given current modeling paradigms and data limitations in the medical domain, achieving mutual benefit is particularly challenging (see Fig. 2). To this end, we design dedicated mechanisms to reduce such conflicts and enable unified modeling.\n\n**4.2 Clinical Potential**\n\nWe highly value clinical evaluation and are already collaborating with two public hospitals, which provide ocular disease reports and case data for validation. These efforts represent a concrete step toward assessing the clinical applicability of our approach, and preliminary evaluations on the provided clinical data show **promising advantages of our model in ocular disease understanding**.\n\n**4.3 Dataset Bias**\n\nDataset bias is an important concern. Due to space limits, please see our response to `Reviewer ugJa (Section 4: Dataset Bias)`.\n\nWe hope our response clarifies the model’s rationale and its potential in medical applications.\n\n**Reference**\n\n[1] When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications  \n[2] A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA", "claims_and_evidence": "1. HealthGPT is the first unified Med-LVLM integrating comprehension and generation: Supported, as existing works (e.g., LLaVA-Med) focus on comprehension only.\n2. H-LoRA effectively separates comprehension and generation tasks: Partially supported. Ablation studies demonstrate benefits, but there is no clear comparison with other PEFT methods (e.g., MoELoRA) beyond training efficiency.\n3. HealthGPT achieves SOTA performance on medical VQA and generation tasks: Not supported. The improvements in VQA over HuatuoGPT-Vision are minor, and some of the published work has a higher VQA effect than this study.\n4. Three-stage training improves performance: Well-supported, as the structured training prevents catastrophic forgetting.\n\nSuggested Improvements:\n1. Provide qualitative visualizations for VQA (e.g., heatmaps, model reasoning breakdown).\n2. Add perceptual metrics (e.g., FID, LPIPS) for evaluating generation tasks.\n3. For QA/VQA, please compare with more advanced methods such as LLAVA-MED+ +, etc.", "methods_and_evaluation_criteria": "The methodology is well-designed:\n1. H-LoRA: Decouples learning across tasks using a Mixture-of-Experts (MoE)-like mechanism.\n2. HVP: Dynamically selects abstract vs. detailed image features.\n3. TLS: Ensures a structured alignment between vision-language modalities.\n\nHowever, the evaluation metrics for generation tasks are too simplistic (SSIM, PSNR, MSE) and VQA performance shows only limited improvement over previous works.\n\nSuggested Improvements:\n1. Introduce qualitative evaluations (e.g., radiologist evaluations of generated images).\n2. Add error analysis for VQA failures.\n\nAnd the other question is that although this approach takes into account both VQA and image generation, which is also a feature of this study, it seems that the two tasks cannot improve the quality of each other.", "theoretical_claims": "While the paper does not present new theoretical derivations, H-LoRA’s mathematical formulation could be further elaborated:\n1. There is no direct theoretical comparison to standard LoRA and MoELoRA.\n2. The routing mechanism in H-LoRA is not well justified.\n\nSuggested Improvement: Provide a theoretical explanation for why H-LoRA is expected to generalize better than standard LoRA in medical contexts.", "experimental_designs_or_analyses": "The experiments are well-structured but have notable limitations: \n1. VQA evaluation lacks detailed error analysis and no comparison with the SOTA methods, like  LLAVA-MED+ +.\n2. No human evaluation for image generation quality.\n\nSuggested Improvements:\n1. Include failure case analysis for VQA errors.\n2. Conduct blind radiologist assessments of generated medical images.\n3. Please compare with more advanced methods such as LLAVA-MED+ +, etc.", "relation_to_broader_scientific_literature": "The paper builds on prior Med-LVLMs (LLaVA-Med, Med-Flamingo) but fails to compare against general vision-language models (e.g., SEED, Chameleon) and the other models like LLaVA-Med++.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths:\n1. Strong multi-modal integration of comprehension and generation.\n2. Innovative H-LoRA adaptation mechanism.\n3. New VL-Health dataset, covering diverse medical tasks.\n\nWeaknesses:\n1. Limited VQA improvement over prior Med-LVLMs, and not compared with LLaVA-Med++.\n2. Image generation evaluation lacks perceptual metrics (e.g., FID, LPIPS).\n3. Comparison to MoELoRA only focuses on training time, not task performance.\n4. Although this method takes into account both text and image output, it seems that the two tasks cannot improve the quality of each other.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. Has HealthGPT been tested in real-world clinical settings?\n2. Could you provide qualitative examples where VQA fails?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y5ZLjjcUAC", "reviewer": "Reviewer_ugJa", "summary": "The paper introduces HealthGPT, a unified Medical Large Vision-Language Model (Med-LVLM) designed to integrate medical visual comprehension and generation. The proposed model employs innovative methods including Heterogeneous Low-Rank Adaptation (H-LoRA), hierarchical visual perception (HVP), and a three-stage training strategy (TLS). HealthGPT leverages a specially curated VL-Health dataset comprising multiple comprehension and generation tasks in medical imaging. Experimental results clearly demonstrate superior performance compared to state-of-the-art methods across diverse medical tasks such as modality conversion, super-resolution, and medical visual question answering (VQA).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer’s recognition of our work. Your recognition of the model’s originality, innovation, and effectiveness greatly encourages us and has helped improve HealthGPT.  Below are our point-by-point responses.\n\n## 1. Unified LVLM Comparison\n\nThank you for the suggestion. Below, we provide further comparisons with SoTA unified LVLMs:\n\nMethod|VQA-RAD|SLAKE|PathVQA|MMMU|OMVQA\n-|-|-|-|-|-\nchameleon|45.4|54.3|54.0|25.7|18.9\nJanus-Pro|62.9|51.3|48.9|54.1|32.7\nEmu3|72.1|57.3|54.1|28.7|39.5\nHealthGPT-M3|73.7|74.6|78.7|43.3|68.5\nHealthGPT-XL|79.1|85.7|92.4|58.0|77.2\n\nMeanwhile, we present an improved **HealthGPT-XL with enhanced performance**. We believe this additional experiment demonstrates significant advantages of our method.\n\n## 2. Generalization Capabilities\n\n**2.1 Generalization ability**\n\nTo demonstrate HealthGPT’s generalization and practical value, we highlight two aspects:\n1. **Benchmark generalization**: The model performs well on unseen tasks and modalities (e.g., OmniMedVQA, MMMU-Med), showing robustness beyond the training distribution.\n2. **Clinical collaboration**: Ongoing partnerships with public hospitals on ocular disease diagnosis have yielded promising early results, with further evaluation in progress.\n\nThese results underscore HealthGPT’s potential for both benchmark-level generalization and practical clinical impact.\n\n**2.2 Complex medical tasks**\n\nThank you for your thoughtful concerns. We recognize their importance and provide the following clarifications:\n\n1. **Unseen medical conditions**: HealthGPT demonstrates strong generalization and medical knowledge coverage, enabling it to handle most previously unseen conditions.\n2. **Unseen modalities**: In cases lacking prior modality-specific data, relying solely on a pre-trained LLM may introduce bias. However, our plugin-based learning approach allows for rapid adaptation using high-quality modality-specific data.\n3. **Simultaneous multimodal inputs**: This engineering challenge can be effectively addressed through targeted data collection and training. We are actively collaborating with hospitals to tackle this issue in real-world clinical settings.\n\nWe will incorporate the clarifications in the next version.\n\n## 3. Differences with MoE Mechanism\n\nWe appreciate the reviewer’s detailed attention to the model's structure, allowing us to clarify the **differences between H-LoRA and MoE architectures** and highlight the **necessity and advantages** of our design.\n\nExisting work explore LoRA and MoE combinations, including symmetric[1] and asymmetric structures[2]. While performance benefits are evident, we observe that introducing LoRA experts significantly increases resource consumption, compromising the efficiency of PEFT.\n\nTo validate this, we compare the training efficiency of several LoRA+MoE structures with the same LoRA setting (r=64, k=4):\n\nMethod|LoRA|MoELoRA|HydraLoRA|DS-LoRA|H-LoRA\n-|-|-|-|-|-\nTFLOP|311.6|204.9|238.3|224.1|313.7\n\nDS-LoRA is a LoRA mechanism based on the powerful MoE model Deepseek-V3. We find that MoELoRA, HydraLoRA, and DS-LoRA significantly reduce training speed, while H-LoRA avoids this limitation.\n\nUnlike traditional MoE scheduling, H-LoRA uses low-rank matrix subspace dynamic routing, preventing performance degradation and instability from issues like routing jitter and load imbalance. It also adaptively activates parameter subspaces, creating soft isolation and weak coupling to enhance adaptability to task heterogeneity. \n\nAdditionally, we emphasize that our approach is not a dense-to-MoE alternative for expanding FFN. While traditional MoE increases capacity by expanding parameters, our method focuses on **balancing performance and computational resources** in efficient fine-tuning.\n\n## 4. Other Issues\n\n**4.1 Potential Limitations**\n\nWe appreciate the reviewer’s attention to potential limitations. Our work explores the feasibility of efficiently handling unified comprehension and generation tasks in medical scenarios. As shown in Figure 2, joint training faces a significant bottleneck in data-scarce medical tasks. However, task cooperation enhancement is a long-term focus, and we aim to explore the complementary potential between the two tasks for synergistic effects.\n\n**4.2 Dataset Bias**\n\nWe understand that bias in sample distribution and task labeling can impact model generalization. To improve chat-LVLM practicality, we balanced general instruction data with curated medical datasets during training, enhancing generalization and medical knowledge. A section on dataset bias has been added to the appendix.\n\nThank you again for your positive and encouraging feedback and insightful comments. We hope this response further clarifies our design, contributions, and the model's potential.\n\n**Reference**\n\n[1] When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications\n\n[2] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning", "claims_and_evidence": "The claims presented are robustly supported by experimental evidence, including thorough comparisons to state-of-the-art models across multiple benchmark tasks (Tables 1, 2, and 3). Performance metrics such as SSIM, PSNR, and MSE clearly substantiate the advantages of HealthGPT in both comprehension and generation scenarios.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria, including specific benchmarks such as VQA-RAD, SLAKE, PathVQA, and IXI datasets, are appropriate and well-suited to the addressed problems. The experimental methodology is carefully designed and rigorously validated.", "theoretical_claims": "The paper primarily addresses methodological advancements and does not involve explicit theoretical proofs.", "experimental_designs_or_analyses": "The validity and soundness of the experimental designs are well-established, particularly for critical tasks such as modality conversion and super-resolution, which are backed by extensive benchmarks and standardized metrics.", "relation_to_broader_scientific_literature": "The paper clearly contextualizes its contributions within the broader literature, showing significant advancement over previous models like Med-Flamingo, LLaVA-Med, and Unified-IO 2.", "essential_references_not_discussed": "While the paper adequately covers related works, including some additional emerging unified LVLM approaches or advanced MoE methods would further contextualize its contributions.", "other_strengths_and_weaknesses": "Strengths:\n\nOriginality in combining comprehension and generation capabilities in a single Med-LVLM.\n\nSignificant methodological innovations (H-LoRA, hierarchical visual perception).\n\nStrong empirical validation against existing baselines.\n\nWeaknesses:\n\nLimited exploration of generalization capabilities across diverse medical scenarios not directly addressed in benchmarks.", "other_comments_or_suggestions": "The paper could improve clarity by explicitly discussing potential limitations and dataset biases.", "questions_for_authors": "Can HealthGPT generalize effectively to medical conditions or modalities that were not explicitly included in the training datasets?\n\nHow well does HealthGPT handle complex multi-modal medical tasks involving simultaneous inputs from multiple modalities?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yi8drOn4Oe", "reviewer": "Reviewer_8ufe", "summary": "This paper introduces a medical vision and generation framework based on LVLM. This pipeline consists of autoregressive generation, hierarchical visual perception, and heterogenous knowledge adaption. The authors provide experiment results on different datasets and compare with other baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review. Your insights have helped us clarify HealthGPT's contributions in unified architecture design, task modeling efficiency, and real-world application potential. Below are our point-by-point responses.\n\n# 1. Contribution Statement\n\n**1.1 Core Contribution**\n\nWe appreciate the reviewer’s detailed focus on our method and contributions. It is worth noting that `Reviewers wUUr, ugJa, and B8gX` have recognized the novelty of our work, describing it as  `innovative` and `meaningful`. Building on this recognition, we would like to respectfully clarify a potential misunderstanding regarding the core contribution of our paper.\n\n- **Paradigm Innovation.**  Our primary innovation lies in **proposing the first unified Med-LVLM paradigm**, which goes beyond alignment or fine-tuning strategies. Specifically:\n  - At the **global-task level**, we enable the model to learn and handle heterogeneous characteristics and formats across both comprehension and generation.\n  - At the **local-task level**, we design mechanisms to efficiently learn and transfer shared knowledge across diverse sub-tasks.\n- **Method Innovation.** We carefully designe H-LoRA to align with our proposed paradigm, and further optimize TLS and HVP to support its integration. Thus, we emphasize that **alignment and fine-tuning are techniques we adopt to support this unified paradigm, not the main novelty itself**. \n\nTo further clarify this point, we briefly summarize the key contributions:\n1. Lines 139–142 (left): We propose the first unified Med-LVLM to solve the challenge of integrating diverse medical tasks under a single LVLM framework.\n2. Lines 74–100(right): We introduce H-LoRA to solve the significant conflict between comprehension and generation tasks, greatly improving both the performance and efficiency over MoE+LoRA-based methods.\n3. Lines 104(right)–126(left): We propose HVP and TLS to solve the challenge of heterogeneous knowledge fusion and hierarchical visual feature requirements.\n4. Lines 127–136(left): we introduce VL-Health to solve the lack of high-quality datasets that jointly support unified tasks in the medical domain.\n\nThus, this work goes beyond benchmarking by establishing a unified Med-LVLM framework that expands the capabilities of current medical models, which is crucial for advancing practical medical solutions.\n\n**1.2 H-LoRA batter than MoE+LoRA**\n\nOur results show that while MoE+LoRA improves performance, it violates the PEFT principle of efficiency. In contrast, H-LoRA significantly outperforms MoE+LoRA in both performance and training efficiency, as supported by theoretical analysis (`Reviewer wUUr (Section 2: H-LoRA)`) and experiments (Fig.5, Tab.5, Tab.10). H-LoRA achieves higher average scores (+3.6/+0.38 on comprehension/generation) with only 67% of the training time. We kindly hope these results clarify the necessity and advantages of H-LoRA.\n\n**1.3 Med-LVLM Significance**\n\nWe appreciate the reviewer’s thoughtful concern regarding the application of Med-LLMs. As highlighted in prior studies [1,2], developing Med-LLMs is both a **timely scientific challenge and a transformative opportunity for healthcare**. These models can **bridge multimodal medical data, support informed decision-making, and ultimately improve clinical outcomes**.\n\n# 2. Efficiency Analysis\n\nWe appreciate the reviewer’s focus on efficiency, vital in resource-limited medical scenarios. To validate efficiency, we compared H-LoRA with MoELoRA under identical settings. H-LoRA outperformed MoELoRA across tasks and reduced training time to 67% of MoELoRA’s:\n\nPEFT Method|LoRA|MoELoRA|HydraLoRA|H-LoRA\n-|-|-|-|-|\nTFLOP|311.6|204.9|238.3|313.7\n\nBesides, our three-stage learning strategy eliminates unnecessary padding caused by task sequence length mismatches, reducing training FLOPs by 22% (from 6.4e19 to 5.0e19) and lowering memory usage per GPU from 36GB to 30GB, saving 17%.\n\nFinally, HealthGPT-M3 completed training on 8 A100 GPUs in 35 hours, significantly fewer GPU hours than most general LVLMs, demonstrating its potential for rapid deployment in medical scenarios.\n\n# 3. Content Refinement\n\nWe thank the reviewer for their helpful suggestions. We will simplify Figure 1 by streamlining labels, standardizing colors, and improving the layout to focus on structural logic.\n\nRegarding the formulas, we appreciate the reviewer pointing out redundancy. We have identified and will simplify or merge unnecessary functions and symbols (e.g., in Equations 4 and 5), ensuring the focus remains on our key innovations.\n\n\nWe sincerely thank you for your valuable suggestions.  We believe this work addresses critical gaps in medical multimodal LVLMs and supports their deployment in real medical scenarios.\n\n**Reference**\n\n[1] A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics  \n[2] A Survey of Large Language Models in Medicine: Progress, Application, and Challenge", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "There are no proofs or theoretical claims in this paper.", "experimental_designs_or_analyses": "The experiment and analysis are fine.", "relation_to_broader_scientific_literature": "They have the impact in the medical domain.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n1, This paper provides a comprehensive study from the method to the results. \n\n2, The writing and presentation help readers to understand and follow.\n\n3, They discuss the main challenges in the medical domain from Line 88 to Line 97. \n\n\nWeakness:\n\n1, I am concerned about the novelty, especially in the training process. They are using a very common and typical approach including feature alignment, LoRA-based plugin, and fine-tuning.\n\n2, For the key part, which is the heterogenous knowledge adaption, the MOE+LoRA is also a well-used approach. \n\n3, I understand that the application in medical domain with LLM might not be a traditional approach. With an extremely extensive experiments in a new domain plus a limited adjusted approach, I am concerned about the significance of contribution and if it is more like a benchmark work.", "other_comments_or_suggestions": "1, Figure 1 is too much and redundant, which is more distracted than being helpful.\n\n2, Many commonly used formulation and equations do not need to re-phrased to be presented in the paper.", "questions_for_authors": "1, Could you please provide some insights of the efficiency of the proposed approach?\n\n2, Could you please emphasize which KEY part of your work is ORIGINALLY proposed by you? I appreciate if you could answer it in a simply and straightforward way. For example, \"from Line xxx to Line xxx, I use xxx to solve the challenge of xxx\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mzjQE0uCa4", "reviewer": "Reviewer_B8gX", "summary": "The paper introduces HealthGPT, a medical vision-language model that unifies visual comprehension and generation through heterogeneous knowledge adaptation. Key contributions include H-LoRA, a parameter-efficient fine-tuning method that decouples task-specific knowledge via independent low-rank plugins, a hierarchical visual perception strategy to handle abstract and concrete visual features, and the VL-Health dataset for medical multi-modal tasks. Experiments demonstrate HealthGPT outperforms state-of-the-art models in tasks like medical visual QA, image reconstruction, and modality conversion.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and insightful review, which greatly helps to refine the manuscript's quality and clarity. Below are our point-by-point responses.\n\n## 1. Additional Details\n\nWe appreciate the reviewer’s insightful questions and welcome the opportunity to further elaborate on the relevant concepts. Below, we provide detailed clarifications.\n\n**1.1 Fig.2 setting**\n\nRegarding the experimental setup in Figure 2, its purpose is to illustrate conflicts that arise in traditional joint modeling methods when comprehension and generation tasks are trained on heterogeneous data. The experiment consists of two stages:\n1. In the first stage, lightweight adapters align visual and textual features to build the visual-language pretraining base.\n2. In the second stage, while preserving single-task data integrity, we gradually introduce heterogeneous data from the other task (25%/50%/100% generation or comprehension data). \n\nWe observe that increasing heterogeneous task data significantly degraded performance due to task interference. In contrast, our method introduces only 2% additional repeated data (used in the second phase of TLS), **greatly improving the performance of both tasks**.\n\n\n**1.2 Diminishing returns clarified**\n\nWe would like to clarify a possible misunderstanding of the “diminishing returns” description. Our intent was not to undervalue methods like MetaMorph, but to highlight the difficulty of transferring joint training paradigms across tasks with distinct structures and imbalanced data scales. \n\nIn the medical domain, comprehension and generation differ significantly in objectives and data availability, making mutual gains through joint training challenging and sometimes prone to negative transfer (see Fig. 2). We will revise the phrasing in the next version to avoid broad descriptions:\n> Joint training of comprehension and generation may face challenges such as task interference, data bias, or optimization saturation.\n\n**1.3 Fusion embedding layer**\n\nAdditionally, we appreciate the reviewer’s attention to the fusion embedding layer. Our design introduces multimodal tokens into the original vocabulary to integrate discrete indices from the visual encoder (VQGAN). This includes 8,192 discrete tokens (from the VQ codebook) and two special tokens, `<START_IMG>` and `<END_IMG>`, forming the fusion embedding layer to seamlessly integrate visual information into the language model input. We will clarify and define this term in the method section.\n\n\n## 2. Comparative Experiments\n\nWe understand the reviewer’s request to compare our method with more powerful LVLMs for better evaluation. To address this, we add experiments with the enhanced **HealthGPT-XL** version and additional comparisons:\n\nMethod|VQA-RAD|SLAKE|PathVQA|MMMU|OMVQA\n-|-|-|-|-|-\nJanus-Pro|62.9|51.3|48.9|54.1|32.7\nEmu3|72.1|57.3|54.1|28.7|39.5\nQwen2-VL|73.3|61.9|62.6|46.0|59.5\nGPT-4o|57.4|54.3|66.5|48.7|36.7\nMedMax|74.9|86.8|91.6|27.3|95.1\nHealthGPT-M3|73.7|74.6|78.7|43.3|68.5\nHealthGPT-XL|79.1|85.7|92.4|58.0|77.2\n\nWe carefully evaluate the concurrent work MedMax, which performs well, especially in OmniMedVQA. We have updated the experimental comparisons content in the manuscript.\n\nIt is important to emphasize that HealthGPT aims to provide a unified, scalable framework adaptable to diverse tasks. It can seamlessly incorporate high-quality datasets like MedMax to boost downstream performance, and its design is **highly complementary to data-driven approaches** such as HuatuoGPT-Vision and MedMax.\n\n## 3. Model Details\n\n**3.1 VQGAN**\n\nWe appreciate the reviewer’s focus on the model's resource consumption and component compatibility, which are essential for the method's reproducibility and practicality. Below are our detailed responses:\n\nThe VQGAN model, pretrained on OpenImages, is validate on three medical modalities: X-ray, CT, and MRI. The results demonstrate its strong performance in preserving key structural and lesion region information, with good fidelity and consistency:\n\nModality|SSIM|PSNR|MSE\n-|-|-|-\nCT|93.95|36.80|14.17\nMRI|87.10|34.45|26.54\nX-Ray|86.80|33.97|29.41\n\n**3.2 Training details**\n\nOur method demonstrates strong resource efficiency and reproducibility. In the default configuration, the **M3 model** completes a full training cycle in about **35 hours** using **8 A100 GPUs**, which we consider reasonable given its performance across multiple tasks and the modular design.\n\n**3.3 Code Release Plan**\n\nWe recognize the importance of open sourcing for the community and have nearly completed organizing the code, data processing scripts, model weights (M3, L14, and XL versions), and inference workflows. We will release the full codebase publicly to ensure reproducibility and ease of extension.\n\nThank you again for your recognition and suggestions. We believe the response better highlights HealthGPT’s technical value and practical potential, and we hope it contributes to the development of medical multimodal models.", "claims_and_evidence": "- In the Introduction, the authors claim there are conflicts between comprehension and generation and refer to Figure 2. What is the detailed experimental setting of this figure? Also, as mentioned by the authors, e.g. MetaMorph (Tong et al.) finds these two tasks are mutually beneficial. What do authors mean by \"improvements still exhibit diminishing returns, with performance degradation remaining a significant issue\" - line 71/72 (right col.)? I did not find corresponding evidence from this paper.\n- Other claims are supported by experiments.", "methods_and_evaluation_criteria": "- There is no explanation of the fusion embedding layer in Figure 3, except a brief mention in line 120. What exactly is the fusion embedding layer? From Figure 3, it is after the text tokenizer and VQ tokenizer, so is it just the regular embedding layer for the conversion of the tokens?\n- The evaluations are extensive and demonstrated the effectiveness of the proposed method.", "theoretical_claims": "No theoretical claim.", "experimental_designs_or_analyses": "- The authors have compared a series of models for medical visual comprehension. Can authors also compare with some stronger, state-of-the-art models such as GPT-4o and Qwen2-VL?\n- There are analysis and ablation studies in section 5.3, which are helpful for understanding the proposed method.", "relation_to_broader_scientific_literature": "It is related to the general unified vision-language models (e.g. unified-IO, Janus) but specifically focuses on medical applications.", "essential_references_not_discussed": "To my knowledge, most essential references are discussed. I found this paper relevant, can authors discuss and compare the performance with it: MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistants. https://arxiv.org/abs/2412.12661", "other_strengths_and_weaknesses": "###Strengths:\n\n- The proposed method and dataset are meaningful explorations towards unified medical vision-language models. The proposed H-LoRA can efficiently enable the training.\n- Experimental results in both comprehension and generation benchmarks indicate the effectiveness of the proposed model.\n\n### Weaknesses:\nPlease address comments in other sections.", "other_comments_or_suggestions": "NA", "questions_for_authors": "- A pre-trained VQGAN is used for generation. Does the natural image pre-trained VQGAN work well for medical images?\n- What is the total training time and GPUs required?\n- Will the model weights be open-sourced? It's \"coming soon\" in the repo.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submission proposes HealthGPT, a unified medical large vision-language model (Med-LVLM), integrating comprehension and generation tasks in medical imaging through a novel heterogeneous low-rank adaptation (H-LoRA), hierarchical visual perception (HVP), and a three-stage learning strategy. It contributes the comprehensive VL-Health dataset covering diverse comprehension and generation tasks. Overall, the reviewers found the methodological innovations significant, particularly the H-LoRA approach, and recognized the strong empirical validation against existing baselines. Initial concerns about incremental improvements in VQA performance, the lack of perceptual evaluation metrics for generation, and limited interpretability were effectively addressed by the authors in their detailed rebuttal and supplementary analyses. Reviewers noted the work’s substantial practical potential and the thoroughness of its experimental evaluation. While one reviewer raised concerns about novelty and whether the contribution aligns fully with ICML's typical methodological focus, the extensive experiments and the practical significance for medical applications justified acceptance. Thus, the committee decided to accept this paper, highlighting its robust innovation, strong empirical validation, and considerable potential for practical medical applications.
Decision: Accept (spotlight poster)