Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE
Authors: Xun Zhu
Ying Hu
Fanbin Mo
Miao Li
Ji Wu
Venue: NeurIPS 2024 poster (2024)
Keywords: medical generalist foundation model
multi-task learning
mixture-of-experts
Primary area: machine_learning_for_healthcare

Abstract: Multi-modal large language models (MLLMs) have shown impressive capabilities as a general-purpose interface for various visual and linguistic tasks. However, building a unified MLLM for multi-task learning in the medical field remains a thorny challenge. To mitigate the tug-of-war problem of multi-modal multi-task optimization in MLLMs, recent advances primarily focus on improving the LLM components, while neglecting the connector that bridges the gap between modalities. In this paper, we introduce Uni-Med, a novel medical generalist foundation model which consists of a universal visual feature extraction module, a connector mixture-of-experts (CMoE) module, and an LLM. Benefiting from the proposed CMoE that leverages a well-designed router with a mixture of projection experts at the connector, Uni-Med achieves efficient solution to the tug-of-war problem and can perform six different medical tasks including question answering, visual question answering, report generation, referring expression comprehension, referring expression generation and image classification. To the best of our knowledge, Uni-Med is the first effort to tackle multi-task interference at the connector in MLLMs. Extensive ablation experiments validate the effectiveness of introducing CMoE under any configuration, with up to an average 8% performance gains. We further provide interpretation analysis of the tug-of-war problem from the perspective of gradient optimization and parameter statistics. Compared to previous state-of-the-art medical MLLMs, Uni-Med achieves competitive or superior evaluation metrics on diverse tasks. Code and resources are available at https://github.com/MSIIP/Uni-Med.

Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE
Xun Zhu! Ying Hu' Fanbin Mo² Miao Li Ji Wu
Department of Electronic Engineering, Tsinghua University
2 Beijing University of Posts and Telccommunications
College of AI, Tsinghua University 4 Beijing National Research Center for Information Science and Technology
{zhu-x24, yinghu Brighuu_hh@maall @mlimm........ tsinghua. edu. mofanbin@bupt. edu. (miao-li, wuji_ee) @tsinghua. edu. cn
(e) corresponding author
Abstract
Multi-modal large language models (MLLMS) have shown impressive capabilities general-purpose interface for various visual and linguistic tasks. However, building a unified MILM for multi-task learning in the medical field remains thorny challenge. To mitigate the tug-of-war problem of multi-modal multi- task optimization in MLLMs, recent advances primarily focus on improving the LLM components, while neglecting the connector that bridges the gap between modalities. In this paper, we introduce Uni-Med, novel medical generalist foun- dation model which consists of universal visual feature extraction module, a connector mixture-ofe ceeee-off-eeeete (CMoE) module, and an LLM. Benefiting from the proposed CMoE that leverages well-designed router with mixture of projection experts at the connector, Uni-Med achieves efficient solution to the tug-of-war -aar problem and can perform six different medical tasks including question answering, visual question answering, report generation, referring expression comprehension, referring expression generation and image classification. To the best of our knowl- edge, Uni-Med is the first effort to tackle multi-task interference at the connector in MLLMs. Extensive ablation experiments validate the effectiveness of intro- ducing CMoE under any configuration, with up gains. We further provide interpretation analysis of the tug-of- of- problem from the perspective of gradient optimization and parameter statistics. Compared previous state-of-the-arr medical MILMs, Uni-Med achieves competitive or su- perior evaluation metries diverse tasks. Code and resources available htttp::/gitten Camm/MSIIP/Uni---.
average 8% performance
Introduction
Driven by the growth of datasets, the increase in model size, and advances in generative language foundation models [Achiam et al. [2023] Touvron et al. 22023], multi-modal large language models (MLLMs) now offer unprecedented abilities as general-puue purppss interfaces. These advancements
spurring innovation across various visual and linguistic tasks [Chen et al.) 2023a Lyu et al. 2023]
Su et al. 2023]. While significant strides have been made in building uniñied foundation model
for natural scenery [Chen et at.) 2022 Lu et al. 2022 2023). the development of generalist medical artificial intelligence is still its early stages [Moor al. 2023a
The goal of a unified and generalist medical foundation model is to enable joint training on massive medical datasets. This model aims to handle multiple tasks and modalities within single architecture with shared parameters [Zhang et al. 2023 Li et al.) [024]. It seeks to eliminate the need for task-specific modules and further fine-tuning, thereby revolutionizing the traditional task-specific
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

LLM
LLM Transörier laee
LLM
Tnnndumer lpper
LLM
Connecto
Connector
Mac IT
Comnora
Projctive Expon
Prdectae Expan
Drpen
Prielce
Three hypotheses and corresponding architectural implementations for multi-task learning in MLLMs. (a) Synergy hypothesis. (b)-(c) Conflict hypothesis in LLM and connector, respectively. (d)-(e) Conffict-synergy coexist hypothesis in LLM and connector, respectively.
Figure 1:
approach to model development [W et al. 2023b Tu et al. 2024 However, existing open-source efforts have not yet fully achieved these ambitious goals.
key challenge in creating a unified medical foundation model is the complexity of modll, multi-task learning, often exacerbated by the tug-of-war problem (Hadsell el al.) 2020). Inherent task conflicts and data imbalances can cause interference during the simultaneous learning of different tasks. This problem is particularly acute in the medical field, where tasks and modalities are highly specialized and diverse. As a result, the performance of each task may degrade compared to task- specialized models (Yu et al. 2020; Zhu al. [022].
To mitigate the tug-of-war problem in mullii-task learning, recent advances introduce the well-known Mfxture-of-FEi -ofturr-of-Experts (MoE) [lacobs er al.) 1991] into MLLMs. Figure [ illustrates three distinct hypotheses and their corresponding architectural implementations for multi-task learning in MLLMs. The first "synergy hypothesis" suggests that all tasks benefit from a fully shared backbone comprising a visual encoder, connector, and language model, which is the standard architecture for MLLMs. The second "conflict hypothesis", proposes that each task requires its own specific adaptations, thereby preventing knowledge sharing among tasks. The third "conffict-synergy coexistence hypothesis", posits that all tasks share multi-task adaptations, which reduces interference and promotes more efficient knowledge sharing. However, current research [Zadouri et al.) 2023) Gou et al. 2023 Liu et
al. 2023b overlooking the potential benefits of exploring and enhancing the connector in MLLMs. Furthermore, the optimization of the tug-of war problem lacks detailed, explainable analysis.
Lin et al. 2024] mainly tailors the MoE approach to the language model components,
In this study, we first identify tug-of-war war problem in multi-task learning at the connector level within standard MLLM architectures. This issue indicates that different tasks may emphasize different types of features in multi-modal, multi-task scenarios. Consequently, a fully shared connector may fall short as it cannot accommodate the diverse modal features required by each task. Drawing inspiration from the successful application of MoE in LLMs, we introduce Connector-Mod (CMoE), a novel approach that employs mixture of projection experts to align visual and language embedding spaces effectively. thus mitigating the tug-of-war problem. As a pioneering effort in constructing a unified generalist foundation model in the medical field, we present Uni-Med. This model comprises universal visual feature extraction module, a CMoE module, and an LMM. Uni-Med demonstrates impressive performance across six distinct medical tasks, with minimal training computational overhead. It achieves joint training on 12 datasets on a single A800 in under 10 hours. The effectiveness and generalization of CMoE are underscored through ablation experiments. Additionally, an interpretable analysis reveals that Uni-Med provides a superior solution to the tug-of-war problem at the connector level. Overall, Uni-Med delivers competitive even superior performance compared open-source, sfate-of-the-ar medical MLLMS on all test sets. Our contributions can be summarized as:
We present Uni-Med, an open-source medical generalist foundation model with a unified interface and shared parameters, which can perform six different medical tasks including question answering, visual question answering, report generation, referring expression comprehension, referring expression generation and image classification.
We propose CMoE, a well-designed connector component for MLLMs, which significantly outperforms baselines under any configuration, with up to an average 8% performance gains. To our knowledge, Uni-Med is the first attempt to focus on the connector MILMs to mitigate the tfg-of-war problem, which is critical but has always been overlooked.

---

Focusing on the question of how the tug-of-war problem is optimized, which has never been quantitatively discussed, we provide detailed interpretability analysis and instructive findings from the perspective of gradient optimization and parameter statisties.
Uni-Med achieves competitive or superior performance compared to the open-source, state- of-the-art medical MI.LMs on test set of diverse tasks and datasets, which demonstrates the huge potential of medical generalist foundation models.
2
Related work
Medical foundation models The increasing availability of medical data, as well as advances in multi-modal LLM technologies, have paved the way for the emergence of medical foundational models. Med-Flamingo [Moor et al.) 2023b) continues pre-training on paired and interleaved medical
image-text data based on OpenFlamingo [Awadalla et al., [203]] LLaVA-Med [Li et al.) 2024
curates a medical multi-modal instruction following dataset and fine-tunes LLaVA [Liu et al. 2024a] with it. XrayGPT (Thawkar et al., [202]] can analyze and answer open-ended questions about chest X-rays. BiomedGPT [Zhang el al. [023] is a multi-task foundation model pretrained on diverse source of medical images, literature, and clinical notes. However, most of these efforts require further fine-tuning on task-specific data to support downstream applications. One step further, the generalist foundation model uses the same weight to excel at various tasks without fine-tuning. RadFM [Wu er al. 2023b] is dedicated to build a generalist foundation model for radiology. Med-PaLM M [Tu et al.) 2024] is directly trained in a unified framework to jointly handle many tasks, which is perhaps most similar to our effort, but does not provide access for usage. In addition, recent studies (Wu et al.
2023a Yan et al. 20244 Xia et al. 2024] have suggested the the necessity for more comprehensive and detailed evaluation of the capabilities of medical MLLMs.
MoE in multi-task learning MoE is originally considered to increase the model capacity [Riquelme et al. 2021 Fedus et al. [202)] and gains popularity in mitigating multi-task interference [Chen er | 2024]. It achieves this by utilizing a router determine the token handled by each expert, thus reducing interference between different types of samples. Recent studies have focused on combining MoE with LLM, such as MoE-LLaVA [Lin al.) 2024] and Mixtral 8x7B (Jiang et al. 2024), or combining MoE with one of the representative parameter-efficien tuning techniques, i.e., LoRA [Hu el al. 202T], such Octavius [Chen al. 2023d]. MoCLE [Gou el al., 2023). MTLoRA Agiza et al. 2024) and MOELoRA [[R et al. [2024b]. However, neither of them introduces MoE into the connector component for MLLMs. Furthermore, there is lack of clear and explicit interpretable analysis on how the task interference is mitigated through the use of MoE.
Cross-modality connector in MILM The connector between the multi-modal encoder and the LLM is critical in aligning modti features [Song er al.] 2023]. One of the most popular paradigms map multi-modal features into a feature space that aligns with language, such as linear projection [Liu et al. 2024a] and MLP projection (Liu et al.) 2023a; Chen er al. [023cc. Another paradigm to transform multi-modal features into multi-modal tokens that are consistent with the embedded representation space of LLM,, such as cross-attention [Li el al. 2022. Ye el al. 2023b 2024), perceiver resampler (Alayrac et al.) 2022] Peng et af. 2023) and Q-Former [Li el al. [2023] Zhu et al. 2023). However, existing paradigms use the same connector when processing the same modal data for different tasks, ignoring the imperative to acquire distinct alignment patterns tailored
to the demands of each task.
3 Methodology
3.1 Preliminaries
3.1.1
Multi-task interference
To quantify the intricate tug-of-wa war problem in unified foundation model, we provide interpretabilitty from the perspective of gradient optimization and parameter statistics.
Perspective of gradient optimization When optimizing the shared parameters 0 according to task the change in the update direction of loss L, for task can be defined as (Zhu et al. 2022
3

---

LANE
G GM)
CA0A
COM
4041
FJM
C200
ACH
1000
13084
CINE
LAGE
2310
CARD
1251
MA17
002
18:1
CCTH
MATT
NAM
AMH
6:10
1550
KAM
KIN
ATM
RATI
JAH
600
ACTA
AIT
MMX
Oryanmeat
(A)
Figure 2: Dataset-level multi-task interference of the synergy hypothesis model at the connector in MLLMs. (a) Perspective of gradient direction GD. (b) Perspective of gradient magnitude GM.
AjLi (2i) = Bx, (L. (1:;; Li 22;; 0-
Vol. (13) Vol3(13))) VoLs (2:)(()))122)) (22)]la)
VoL, (1z)² |VoL (zj)ll2
ALEE,
VoLi (2i)
(1)
where 24 and by the sampled training batches of task i and j. respectively. The interference of task j on task in the update direction can be quantified as:
La (2.)
GD = Bar
(2)
A.L. (z:))
The gradient magnitude similarity between task i and task j can be defined as: 2Ez, (IIVoL. (xx) Ba, (IVV (z)))l2
G.M., = GM = (Ez, (llV,L, (2.) (E., (LVoL, (2,))la))²
(3)
GMi,, goes to zero when the difference in gradient magnitudes large, indicating that some task dominant Yu et al. 2020). For all T tasks, we can get GD, GM E R Then, we define the tug-of-war indexes for each task in multi-task learning through the function G as follows:
warrof-war indexes = G(GD, = Σ GD$, GM..
(4)
Perspective of parameter statistics Inspired by the Gradient Positive Sign Purity proposed by Chen et al. [2020), we define the statistics score a single parameter in multi-task learning:
Σ.
VoLa
statistics score
(5)
Σ VoL.
where Vo is the gradient for task i. The range of the statistics score is [0, I]. and value close to 1 indicates that this parameter suffers less gradient conflict during multi-task training. Upon collecting the statistics scores of all parameters, we can intuitively demonstrate and analyze the phenomenon of multi-task interference.
To be specific, we sample 100 batches for each datasets and record the gradients to calculate all of the above metrics. Figure shows the dataset-level (more granular than task-level) multi-task interference of the synergy hypothesis model at the connector in the standard MLLM architecture.
3.1.2 cexture-offExxertt
Mixture-oo-Exppreee (MoE) contains a set of expert networks E₁, Ez,... Ey along with a routing network R. For each token x; in the input sequence X (e the output of MoE is the weighted sum of outputs from each expert, where the weight is calculated by the router:
Ræsss. Ex(xs)
(6)

---

Multi-task
Multi-modal
Vivual Feature
Extraction Module
Seaee-Lord Lord
Answering
Question
Agpeoaiiee
LoRA
I
Routs
Vieual Answering
Xiop
Projection Pepert
Large I Aaniiine Mndd
Visual Encodder
Roport Generstio
Finks
Projection
Reforring Expression
S.
Compreheension
Uirawed
Refering Expression
Poojection Expert
Connector-Moß Module
Trasaane Sporal
(ngil
Image Clasefication
Response
Overall architecture of Uni-Med, which consists of universal vision feature extraction module, a connector-Mof module and an LLM. Uni-Med can perform six different medical tasks including question answering, visual question answering, report generation, referring expression comprchension, referring expression generation and image classification.
Figure 3:
The types of R can mainly be divided into: 1) Constant router, which assigns equal weight to each expert. 2) Hard router, which enforces one-to-one mapping between tasks and experts. 3) Sparse router, which selects Top-K experts with the maximum routing weight. 4) Soft router, which calculates the routing weights for each expert. For more details on the routing networks, see Appendix A.I
3.2 Model Architecture
With the primary goal of achieving unified medical generalist foundation model and mitigating the tug-of-war problem of multi-task learning in mind, we design the overall architecture of Uni-Med as illustrated in Figure which contains three components: universal vision feature extraction module, a connector-Me Moo module and LLM. Detailed descriptions are presented in the following sections.
3.2.1 Visual feature extraction module
Taking one of the multi-modal medical images I E #x as input, the visual encoder Ven extracts the image tokens fo E N. for image perception, where N. HW/P" /22 is the number of image patches and D, is the hidden size of visual embeddings.
To alleviate the efficiency issues caused by prolonged visual input tokens during the training and inference, scheme resampler with a compression rate for visual feature aggregation. Concretely, adjacent visual tokens concatenated and projected into one single embedding. Thus we obtain aggregated image tokens foo R N./axD, D.a follows:
resampler (Ven (I),cc)
3.2.2 Connector-MoE module
Aligning the visual space with the language embedding space of the large language model a critical process, especially in the complex and diverse input of multi-task multi-modal medical image text pairs. Based on the conffict-yyerry cocxist hypothesis, we propose the Connector-Mo module, which aims to adaptively minimize task conflict and maximize task synergy at the connector. CMoE module has N projection experts Et, E2, .... EN, where each expert is two-layer MLP, and soft router Ragg to control the contribution of each expert.
(CMoE)
According to Figure 2 find that: (1) Gradient optimization conflict is common and consistent at the task level. (2) Even for the same task, there are significant differences conflict and synergy at letel.t-level. To alleviate the above problems, we randomly initialize vision-level special task tokens ( JEET, where f: R.a and T' is the of tasks. Regh is lightweight MLP designed to receive the concatenated inputs of fae (token level) and feP (task level), and calculate the routing weights Waeh E D of each expert for each image token, which can be formulated as:
5

---

Table 1: Text-level special task identifiers for different tasks.
Visual
Referring Expression Compreheension
Referring Expression Generation
Task
Question Answering
Question
Report Generation
Inage Clossification
Answering
Identifier
Iqal
Ivqal
[caption]
[refer]
lidentiy]
[cis]
Wish (J: = o. Rust (ff0 Repeaa (f(²))
(8)
where [.] denotes concatenation operation, is SoftMax function. Then we can obtain aligned visual tokens falign RN///xD, through a weighted sum all experts' output as follows:
palign fu
W (x()(²)
(9)
where D, is the hidden size of the language embedding space of the large language model and Wayf. denotes the routing weight of the k-thh projection expert. We discuss and analyze the effects of router type, router strategy, and number of experts in Section 4.2.1
3.2.3 Large language model
Similar the vision-level special task tokens, assign the text-level special task identifiers for question answering (QA), visual question answering (VQA), report generation (RG), referring expression comprehension (REC), referring expression generation (REG) and image classification (CLS) as shown in Table which can help reduce multi-task ambiguity (Chen et al.) [003bb] The text prompt designed "<Img> ImageFeeturre </Img> [Task Identifier] Instruction" " which merges the converted image features with the textual instructions. See details about our multi-task instruction template in Appendix C
After word embedding, we can obtain textual tokens ft E R Ne where N₂ denotes the number of textual tokens. LLM generates the response O { conditioned on the aligned visual tokens fo falign and textual tokens f. inputs an autoregressive manner, which can be formulated as:
(0.
f calion f:)
II (o. falion Oz.
(10)
where is the length of output tokens. We use low-rank adaption (LoRA) Hu et al. 2021 for efficient LLM fine-tuning, which is applied to all the linear layers.
4 Experiments
4.1 Experiment settings
Tasks and datasets Text-only data is collected from MedQA [Jin al. 2021 and PubMedQA (Jin et al.) [019] for the task of QA. Image-text pairs are collected from Path-VQA [He et al. 2020 and Slake-VOA VQA Liu al. 2021] for the task of VQA. MIMIC-CRR [Johnson al [019 and MPx-Single [w et al, 2023b) for the task of RG, MedMNIST v2 (Yang et al. 2023 for the task of CLS. For tasks such as REG and REC that require representation of spatial locations, we use the
bounding boxes of the format "<Xmin>>< > >>t,>>>>m>>>>> which denotes the coordinates of objects. Then, we respectively process datasets Slake- VQA [Liu al. [02T] and SA-Med2D-20M
Ye et al.) 2023a] to get datasets Slake-RFC. Slake-REG, SA-Med2D-REC and SA-Med2D-RFGG.
For detailed description, processing and splitting of all datasets, see Appendix B
Implementation details We adapt the open open-sourced ViT-G/14 from EVA-CLIP Fang et al. 2023 and LLaMA2-Chai (7B) [Touvron et al. 2023) as our visual backbone and LLM, respectively. During the training process, each task is assigned sample rate that is calculated in proportion to the respective task's data volume. The visual backbone remains frozen with an input image resolution of 224*224 and the LLM is fine-tuned through LoRA (Hu et al. 2021 with the rank of 8. The compression rate 4=4 and the number of projection experts N=5. Uni-Med only requires one-stage training a NVIDIA A800-SXIL SXM4-80GB GPU, with the first 10k iterations to warm-up and total

---

of 100k iterations with batch size of 4, which lasts roughly 10 hours. The peak learning rate is set le-6 and decays to le-7 following the cosine strategy. We use AdamW [Loshchilov and Hutter 2017] optimizer with 31=0.9, 32=0.95 and weight decay of 0.05.
Evaluation metries For ablation studies, we report BLEU-1 for the task of VQA, REG, and RG, IoU for the task of REC, Accuracy for the task of CLS. In addition, we use A &Ei=1 (M.. Mb.)) /Ma,i 100% to evaluate the performance gains, where Mn and M,.; are the metrics of our model and baseline model, 5 can be the number datasets or tasks. For the overall comparison between models, report more metrics such as F1, ROUGE, METEOR, RadGraph Fl and RadCliQ YYu et al. 2023). See details at Appendix D.I
4.2 Ablation study
4.2.1
Ablation module design
Connector design Taking the connector of two-layer MLP baseline setup, we first discuss the performance of different multi-task learning hypothesis. In (a), connectors based on conflict- synergy coexist hypothesis (CMoE with sparse / soft router) show more holistic improvement trend in multi-task learning compared to connectors based on the conflict hypothesis (CMoE with
Table 2: Experiments of ablation study. Metrics are reported on "Slake Mialee--QA/Pat--OOO "Slake- REC/SA-Med2D-REC" -E--REG/AA--MeDDROOOO- "MIMIC-CXRMMX--Snnee "DermaM- NIST/OrganSMNIIST for the task of VQA, REC, REG, RG, and CLS, respectively.
Connector
Router
VQA BLBU-1
A (t)
REC IoU
A (t)
RBG
(1)
RG
(t)
CLS Accuracy
(f)
Total & (f)
Type
Strutegy
BLEU-]
BLBU-1
(a) Connector design
Lincar
77.90 56.27
-1.4%
28.44/ 11.59
-23.%
74.91/5561
-2.1%
13,07
1585
-11.6%
72.47
(69.9)
-5.4%
-8.9%
ME.P
70.81 /56.48
35.187
16.36
74.54758.45
18.55/
15.50
76.26
73.64
Constant
73.58
1.7%
51.85
30.01
55804
-2.8%
22.76/
8.6%
CMsE
Spirs
Tokess
Tokes
50.68 /57.02
$1.79/57.69
2.3%
1.0%%
37.07/ 18.41
35.51
17.39
5.2%
9.3%
76.6//600 60.08
74.43/61.34
2.4%
3.0%
26.27/ 15.1
24.02/ 15.73
21.2%
15.5%
73.4774495
76.56/77.21
-1.0%
2.6%
6.7%
5.6%%
Task
$1.51/57.43
38.33
78.18
23.34/
77.56/76.55
2.9%
Token Tack
52.57757775
37.54
34.707 15:55
75 776.92
1.8%
(b) Resampler design
Compression
75.63/58.34
15:54
Compression Rase Projection
83.74/57.70
3,8%
37.02/
18.57
9.7%
71,29/000
60.32
0.2%
25,83/ 15.77
20.5%
74.56/76.70
76.70
1.0%
6.9%%
Compression Rase 4, Max Pooling
50.36/57.44
1.2%
27.16/
14.37
17..2%
68.301573
-4.9%
18.85/ 15.60
1.1%
75.71
73.08
-0.7%
-4.1%
Compression
Pooling
$1.96/57.88
73.39/00
74.54
Compression 4, Projection
81.52757.75
37.54/
00.42
24,707
76.92
(c) Number projection experts
90.45
35.98/
66.64/58
74.59
-4.3%
51.52/55.75
37.54/
77.47760
776.92
1.8%
8
S2.71 $2.71/57.8 57.88
3.0%
36.66
18.34
8.5%
75.15/5640
-2.5%
24.47/
15.74
16.7%
77.91
77.91/7.555
76.55
3.0%
5.8%
10
83.21 05.11/5.50 57.88
3.3%
38.70
1901
13.5%
76.0661144
2.9%
25,02
1505
16.0%
77.66
/7.99
77.99
3,9%
7.9%
76.3//6.44
2.3%
(d) Module generalization under LeRA rank setting
Commector&Rousee
MLP
22.70/
CMak.
$1.94/57.7
2.0%
29.30/
1098
-26
70.14/51.55
.0.4%
22.74/
15.76
0.8%
81.95
31.95/80.44
/80.45
11.5%
-2.4%
CMol.,oot
82.63/7.88
2.3%
32.807
1531 800%
68.12 60.84
6.3%
24.967
15.01
4.1%
76.1177384
3.0%
1.5%
MLP
79.81/ 56.45
35.18/
16.26
74.54/58.12
18.55/ 15.50
73.26/73.61
$1.35
30.01
12.3%
8.6
SMol..off
$1.52/57.55
2.2%
37.54/20.30 20.30
15.8%
77.55600442
3.7%
24,707 15:3
16.7%
75.61776.92
1.%
8.0%
MLP
59.10/55.45
32.73/ 14.51
157.89
24,127 16.09
69.1 69.18/75.43
CMol.Haad
71.56/56.42
12.0%
CMol.Soft
26.527
MLP
79.23/56.50
33.50/
16.13
72.04 58.78
18.67 15.42
71.67 72.69
32
Cark.Hard
82.25 0225//588 58.54
3.7%
30.56
11.97
-17.3%
75.16/59.20 1.6%
23.22/ 15.58
12.7%
/2.19/880.7
12.8%
1.7%
CMol.Soft
$1.39/57.18
35.957
70.4439.66
26.567 15:45
21.2%
77.61/77.18
7.2%
22.291
71.12/75.84
CMal.Haad
51.52/5855
2.5%
31.95/
12:24
-0.59%
66.82
46.29
-13.0%
23,88
15355
6.7%
79.97
10.4%
.2.8%
CMolE,Soti
81.53/58.00
58.04
2.1%
35.66/
18.31
3.0%
73.22/60.26
4.1%
25.897
16.77
14.3%
75.21/77.99
3.8%
5.5%
(e) Mudule gemeralization under LeRA- Mol setting (runk
1LM fine-tuning
Connector&&.o. Rouker
LoRA
MLP
50.7/ 56.42
36337 16.32
64.34/57.02
22,70/ 1554
71.62/74.86
L-RA-M-E
CMd..Hanl Soff.Soft
54 10/06/1.2
6.4%
31.56
-179%
/63.51
22,67/ 13.23 -7.5%
77989
10.2%
LORA-MoE
7.3%
39.33/
17.10 65%%
76.96/67.10
21.4%
18.757 1375
-14.5%
77.99 7.7%
5.7%

---

hard router) and synergy hypothesis (linear, MLP, CMoE with constant router). Though the hard router has a obvious lead on the CLS task, implying that the CLS task is better suited a separate connector to avoid conflicts with other tasks. The soft router achieves the best multi-task performance, indicating that not only alleviates conflicts between tasks, but also promotes collaboration between tasks. We then discuss three types of router strategy. The strategy of combining token-level with task-level information is superior to using each information separately, indicating the effectiveness for considering the tug-of-war problem from both token and task level.
Resampler design We explore whether aggregating visual features through resampler has unfavor- able effects in Table (b)) Despite an increase in compression rate from 1 to 4, the performance of models utilizing projection aggregation is improved. While the performance of average pooling and max pooling approaches is not satisfactory, especially the latter has severe performance degradation, which may be attributed to the excessive loss of feature information. This phenomenon shows that appropriate visual feature compression can bring efficiency to the training process without losing or even improving performance.
Number of projection experts The number of projection experts N is one of the most significant hyperparameters, which closely related to the number of tasks and modalities that the CMoE module can accommodate. It is challenging study as the complexity of the scenario can end up overfitting to simpler tasks and modalities or underfitting complex ones. As shown in Table (c), increasing the number of experts N, namely an augmentation in parameters, still brings performance gains on some datasets, but the average gain tends stabilize across all tasks and datasets. Therefore, CMoE with 5 projection experts is sufficient to handle the tug-of-war problem in the existing medical multi-task learning scenarios and training configuration. A higher value of N does not bring the desired further improvement in total A.
4.2.2 Ablation on module generalization
We demonstrate the generalization capability of the CMoE module in any configuration, especially when the key hyperparameters and strategies for LLM fine-tuning change. We first focus on the rank of LoRA, which directly determines the L.MM capacity, i.e., trainable parameters. Our observations in Vable@((c (d) reveal that CMoE with soft router can steadily improve multi-task performance when LoRA rank increases from 4 to 64. In (e), we introduce MoF to LoRA, namely L-RA- which is considered a favorable parameter-effiiiiett tuning solution for multi-task applications (Liu et at. 2023bb Chen et al. 20224). See details of LoRA-MoEE at Appendix AA.2. We find that separate LoRA-MoE results in significant performance improvement in 3 tasks while degradation in 2 tasks, indicating that it does not achieve the efficient solution to the tug-of-war -arr problem. After combining CMoE with soft router, we achieve balance of consistent performance gains, further demonstrating the necessity and effectiveness of mitigating the tug-of-war problem at the connector level in MLLMs.
4.3 Interpretation
We conduct interpretation analysis of the tug-of-war problem based on methods mentioned in Section 3.I.I Specifically, we focus on the changes in the connector using CMoE compared to MLP and show how the tug-of-wai warr problem is optimized: (1) From the perspective of gradient optimization, we use maximum normalization to make the tug-of war indexes comparable under different architectures. CMoE results in more consistent oug-of-war indexes, i.e. higher mean and smaller standard deviation, among different tasks or datasets, implying each individual gets a more balanced optimization, as shown in Figure 4 (a). (2) From the perspective of parameter statistics, we discrete the statistics scores into ten intervals and count the ratio of all parameters at connector by interval. CMoE results in an increase in the proportion of high- value intervals in Figure 4 (b). We show the routing weights of projection experts after the warm-up stage and the final model in Figure 4 (c). CMoE adaptively learns different patterns of routing weights for different tasks.
To better reflect the coexistence of conflict and synergy among tasks, as well as the critical role played by the connector, we visualize the distribution of visual features before and after passing through the connector using the t-SNE method [Van der Maaten and Hinton| 2008 From the perspective of multi-task learning, we randomly select 200 samples from each task. It can be observed that CMoE promotes the optimization of the tug of war problem when aligning the visual space with the textual space of the LLM in Figure 5 Specifically, visual features of the same task are more
8

---

tightly distributed. For fine- ine-grained REC and REG tasks, the distribution is highly overlapping, which facilitates synergy between tasks. For coarse-grained CLS task, the distribution is significantly different from other tasks, which is consistent with the conclusion in Section 4.2. We also provide visualization analysis of visual features on different medical image modalities Appendix D.4
rag-af-war
VQJ
Indenss
03023
0.555
03231
0.033
RG
03155
03733
11233
11964
14009
1.90
01900
E1
B2
15
E4
E5
133793
10333
VQA
RBC
ast-level
dstssttleve
REG
O801 C13D a20l 6364 9405 0.E6 #607 0765 6594 05.18
parameter statsties scores
rooting weights
Figure 4: Interpretation analysis of the tug-of war problem. (a) changes in tug-of-war indexes, (b) changes in the distribution of parameter statistics scores, (c) routing weights for different tasks.
15
3.5
Tack
06
64
VQS
RFC
00
M
00
KIG
8.5
0.6
06
84
04
14
1.0
(b)
(6)
Figure 5: Visual features distribution maps-3D. (a) fag distribution, (b) falign distribution obtained through MLP, (c) fatign distribution obtained through CMoE.
Table 3: Model capability comparison with open source medical MLLMs. The mean and standard deviation of performance Uni are obtained after several 300k iterations. Results with bold, underlines and gray background are the overall best, second, and zero-shot performance, respectively.
Task
Dataset
Metric
Med-Flamingo
RadFM
LLaVA-Med
XrayGIT
Uni-Med
BLEU-I
21.51
81.66
76.95
82.12=0038
Slake-VQA
Visual
FI
23.66
82.38
77.30
83.07=0.34
Question
BLEU-1
33,38
24.83
58.07=0.32
Answering
Path VQA
F1
34.01
25.20
47.08
58.74=0.33
BLEU-1
23.25
6.81
19.90
27.11
27.79=2.50
BLEU-4
1,92
1.52
0.59
3.56
6.46:10.20
ROUGE-1
18.73
16.81
24.35
28.81=1.22
ROUGE-2
9.62:10.99
MIMIC-CXR
ROUGE-L
12.25
12.67
10.29
16.29
22.58
METEOR
7.95
5.32
5.47
9.71
10.59=0.87
RadGraph-F1
7.15
7.19
2,86
9.00
13.98-244
RadCHO--OU
4.44
4.43
4,79
4.42
0.71
RadCliQ-v11
1.80
1.79
1.38+0.11
Report Generation
BLEU--
8.14
9.46
8,51
15.80 0.224
BLEU-4
0.45
0.59
0.23
2.47-10.08
ROUGE-1
11.37
11.31
8.00
14.32 0.03
ROUGE-2
MPx-Single
ROUGE-L
12.29=0.04
METEOR
4.31
5.51
3.60
5.02-10.07
RadGGrph Fl
1,85
2.63
1.32
1.91.1..11
RadCHiQ-vOL
4.00
3.88
4.17
3.59-
RadCtiQ-111
1.62
1.55
1.72
1.37:10.01
DemuMNIST
Accuracy
1.15
5.14
76.96 0.44
Image
Classification
OrganMNIST
Accuracy
8.90
18.90
78.07=1.63

---

4.4 Overall comparison
To demonstrate the model capabiliitiss of Uni-Med on multi-task learning, four open source and state-of-the-an medical MLLMs including Mlaminna [Moor er al. (023b), RadFM (Wu er al.
20236]. LLaVA-Med [L et al. 2024]. and XrayGPT (Thawkar et al. al.] 2023] are used for performance
comparison in Table Any method of fine-tuning will inevitably lead to changes in the initial capability of the model. Therefore, we use readily available model checkpoints for testing, following the prompt template requirements of different models. Under this comparison strategy, if the training datasets of model and Uni-Med intersect and strietly follow the official partition, it is fair and comparable to Uni-Med on these datasets. Specifically, LLaVA-MED provides dataset-specific fine- tuning checkpoints on Slake-VQA and Path-VQA separately. XrayGPT focuses on the task of report generation and utilizes MIMIC-CXR as training dataset. RadFM provides a model checkpoint for joint fine-tuning on Slake- VQA. MIMIC-CXR and MPP--Sngle. However, we do not list performance of RadFM on MPx MPx-Single
have identified the issue of data leakage, see Appendix [D2]
The results in Table show that our Uni-Med achieves leading and competitive evaluation metrics across all tasks, which has the following prominent advantages: (1) Uni-Med is able to handle greater variety of medical tasks, which attributed to multi-task learning during training process. Due to the fact that the above MILMs do not support input and output in coordinate form, we report the performance of Uni-Med on REC and REG tasks at Appendix and output forms supported by each model, we have also listed the zero-shot results in Table for reference only. (2) Uni-Med achieves better results through joint training fine-tuning rather than dataset-speciffcc fine-tuning like LLaVA-Med, which benefits from efficient optimization of the tug-of- war problem. In addition to directly compare the capability of existing models, we take LA-MMMMe as an example to compare the capability of model architectures in Appendix D.6
Based on the different input
5 Conclusion
In this paper, we present a novel open-source medical generalist foundation model Uni-Med, which can handle six different medical tasks including question answering, visual question answering. report generation, referring expression comprehension, referring expression generation and image classification. Benefiting from the proposed CMoE, which combines MoE with the connector, Uni- Med achieves efficient solution to the tug-of-war problem in multi-task learning. Uni-Med not only achieves competitive or superior performance compared to the open-source MLLMs, but also provides interpretability analysis from the perspective of gradient optimization and parameter statistics on how the tfg-of-war problem is optimized. We hope promote the development of medical generalist foundation models and inspire more research toward generalist medical artificial intelligence.
tftte-oftth-arr medical
can greatly
Limitations
While Uni-Med has demonstrated strong potential as a unified and generalist medical foundation model, it still exhibits several limitations: (1) Limitations in handling genuine 3D medical image inputs. Most commonly used medical image in 3D.. Same most medical MILMs,, we process 3D images into 2D slices as input, resulting in significant information loss. (2) The potential of performance gains in more complex multi-modal and multi-task learning scenarios has not yet been explored. Uni-Med use 12 datasets of 6 medical tasks, with a total data volume of 140k. (3) The potential of performance gains in different LLM backbones has not yet been explored. Uni- Med utilizes LLaMA2-7B. (4) Deeper theoretical analysis of tug of war problem remains to be explored. We attempt to combine the existing methods analyze it from the perspective gradient optimization and parameter statistics. (5) Potential negative socictal impacts. We cannot prevent potential malicious or unintended uses, such as generating fake profiles or wrong medical diagnoses. and provide necessary safeguards.
Acknowledgements
The work is supported by Noncommunicable Chronic Nastines-National Science and Technology Major Project (2023ZD0506501) and National Key R&D Program of China (202
10

---

References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida. Janko Altenshmmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Ahmed Agiza, Marina Neseem, and Sherief Reda. Mtlora: A low-rank adaptation approach for efficient multi-task learning. arXiv preprint Sttl::403.203200, 2024.
lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736, 2022.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: open-soure source framework for training large autoregressive vision-language models, arviv preprint arXiv:2308.011..0 2023.
Patrick Bilic, Patrick Christ, Hongwei Bran Li, Eugene Vorontsov, Avi Ben-Cohen, Georgios Kaissis, Adi Sze- skin, Colin Jacobs, Gabriel Efrain Humpire Mamani, Gabriel Chartrand, et al. The liver tumor segmentation benchmark (lits). Medical Image Analysis, 84:102680, 2023.
Zhao Chen, Jiquan Ngiam, Yanping Huang. Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick sign: Optimizing deep multitask models with gradient sign dropout. Advances in Neural Information Processing Systems, 33:2039- 2020.
Ting Chen, Saurabh Saxena, Lala Li, Tsung- Yi Lin, David Fleet, and Geoffrey E Hinton. A unified sequence interface for vision tasks. Advances in Neural Information Processing Systems, 35:31333-311 2022.
Feilong Chen. Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi. Shuang Xu, and Xu. X-IIm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages. arXiv 2023.
preprint
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478, 2023.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang. Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.1279; 2023.
Zeren Chen, Ziqin Wang. Zhen Wang. Huayang Liu, Zhenfei Yin, Liu, Lu Sheng, Wanli Ouyang. Yu Qiao, and Jing Shao. Octavius: Mitigating task interference in milms via moe. arXiv preprint arXiv:2311.026. 2023.
Zitian Chen, Yikang Shen, Mingyu Ding, Zhenfang Chen, Hengshuang Zhao, Erik G Searmed-Milles and Chuang Gan. Mod-squad: Designing mixtures of experts as modular multi-task learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 11828-11837, 2023.
Shaoxiang Chen, Zequn Jie, and Lin Ma. Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning milms. arXiv preprint urXiv:2401.16166 2024.
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv: 190.199022.3333 2019.
Yuxin Fang, Wen Wang. Binhui Xie. Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
Yunhao Gou, Zhili Liu, Kai Chen, Langing Hong, Hang Xu, Aoxue Li. Dit-Yan Yeung, James T Kwok, and Yu Zhang. Mixture of cluster-conditional lora experts for vision-language instruction tuning. arXiv preprint arXiv:2312.12379 2023.
Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in cognitive sciences, 24(12)1028-14) 1040, 2020.
11

---

Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint wrXiv:2003.1026, 2020.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Yuanzhi Li. Shean Wang. Lu Wang. and Weizhu
Chen. Lora: Low-rank adaptation of large language models, arXiv preprint arXiv:2106.0666: 2021.
Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong. Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew Lungren, Andrew Y Ng. et al. Radgraph: Extracting clinical entities and relations from radiology reports. arXiv preprint arXiv:2 14463, 2021.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint 1tf1:2401.04088. 2024.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu, Pubmedqa: A dataset for biomedical rescarch question answering. arXiv preprint 190.0669 2019.
Di Jin, Bileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang. and Peter Szolovits. What disease does this patient have? large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):621,, 2021.
Alistair EW Johnson, Tom J Pollard, Seth Berkowitz, Nathanicl Greenbaum, Matthew Lungren, Chih-ying Deng. Roger G Mark, and Steven Horng. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019.
Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language image pre- ge-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888- PMI.R, R., 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. International conference on machine learning. pages 19730 19742. PMLR, 2023.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang. Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vissg assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36. 2024.
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-Ilava: Mixture of experts for large vision-languag models. arXiv preprint arXiv:2401.15947 2024.
Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically-labelee knowledge- enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650-1654. IEEE, 2021.
Haotian Liu, Chunyuan Li. Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arX1:22310003744, 2023.
Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications. arXiv preprint arXiv:2310.78339) 2023.
Haotian I. Liu, Chunyuan I., Qingyang Wu, and Yong Jae Lec. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. When moe meets Ims:: Parameter efficient fine-tuning for multi-task medical applications. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1104 1114,112101 2024.
Ulya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arxiv: arXiv: 1711.05001 2017.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unitied-io: A unified model for vision, language, and multi-moda tasks. In The Eleventh International Conference on Learning Representations, 2022.
12

---

Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. arXiv preprint urXiv:3312.17172. 2023.
Chenyang Lyu, Minghao Wu, Longyue Wang. Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration. arXiv preprint WrXi:2330000909, 2023.
Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovee, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, 616(7956):259-265, 2023.
Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (MI4H), pages 353-367. PMLR, 2023.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583-8595, 2021.
Shezheng Song. Xiaopeng Li, and Shasha Li. How to bridge the gap between modalities: A comprehensive survey multimodal large language model. arXiv preprint arXiv:2311.07550 2023.
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model instruction-follow them all. arXiv preprint arXiv:2305. 16355, 2023.
Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Shahbaz Khan. Xraygpt: Chest radiographs summarization using medical vision-language models. arXiv preprint 2rXiv:2306.077 07977, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schackermann, Mohamed Amin, Pi- Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, al. Towards generalist biomedical ai. NEJM AI, 1(3):1oo2300138, 2024.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1 1), 2008
Chaoyi Wu, Jiayu Lei, Qiaoyu Zheng, Weike Zhao, Weixiong Lin, Xiaoman Zhang, Xiao Zhou, Ziheng Zhao, Ya Zhang, Yanfeng Wang, et al. Can gpt-4v (ision) serve medical applications? case studies on gpt-4v for multimodal medical diagnosis. arXiv preprint arXiv:2310.09909 2023.
Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology. arXiv preprint arXiv:2308.00 02463, 2023.
Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong. Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, et al. Cares: A comprehensive benchmark of trustworthiness in medical vision language models. arXiv preprint arXiv:2406.06000 2024.
Xuanang Xu, Fugen Zhou, Bo Liu, Dongshan Fu, and Xiangzhi Bai. Efficient multiple organ localization in image using 3d region proposal network. IEFE transactions medical imaging, 1888) 2019.
Qianqi Yan, Xuchai He, Xiang Yue, and Xin Eric Wang. Worse than random? an embarrassingly simple probing evaluation large multimodal models in medical vqa. arXiv preprint (XXi::4405.24221 2024.
Jiancheng Yang. Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a acale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):41, 2023.
13

---

Jin Ye, Junlong Cheng. Jianpin Chen, Zhongying Deng. Tianbin Li, Haoyu Wang, Yanzhou Su, Ziyan Huang. Jilong Chen, Lei Jiang, al. Sa-med2d-22m dataset: Segment anything 2d medical imaging with 20 million masks, arXiv preprint urXiv:2311.119990 2023.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye. Ming Yan, Yiyang Zhou, Junyang Wang. Anwen Hu, Pengcheng Shi. Yaya Shi, al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint wIXv:2304.14178, 2023.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Zhang, and Fei Huang. mplug-ow12: Revolutionizing multi-moda large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13040-13051, 2024.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824-5883 2020.
Feiyang Yu, Mark Endo, Rayan Krishnan, lan Pan, Andy Tsai, Eduardo Pontes Reis. Eduardo Kaiser Uru- rahy Nunes Fonseca, Henrique Min Ho Lee, Zahra Shakeri Hossein Abad, Andrew Y Ng, et al. Evaluating progress in automatic chest x-ray radiology report generation. Patterns, 4(9), 2023.
Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker. Pushing mixture of experts the limit: Extremely parameter efficient for instruction tuning. arXiv preprint arXiv:2309.0m 05444, 2023.
Kai Zhang, Jun Yu, Zhiling Yan, Yixin Liu, Eashan Adhikarla, Sunyang Fu, Xun Chen, Chen Chen, Yuyin Zhou, Xiang Li, al. Biomedgpt: A unified and generalist biomedical generative pre-trained transformer for vision, language, and multimodal tasks. arXiv preprint arXiv:2305. 17100, 2023.
Jinguo Zhu. Xizhou Zhu, Wenhai Wang, Xiaohua Wang, Hongsheng Li, Xiaogang Wang, and Jifeng Dai. Uni- perceiver-moc: .earning sparse generalist models with conditional moes. Advances in Neural Information Processing Systems, 35:2664-2678, 2022.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
14

---

A Component design
A.1 Type of the routing network
Constant router The simplest routing network is assign equal weights to the output of each expert, which can expressed as:
Reonstond (xt)
(L/N
(11)
Hard router Each token is assigned to specific expert based its type (task / modal), with the number of experts being equal the number token types. It can formulated as:
Rowd (xx)) = IsType km)
if x, belongs to type
(12)
IsType (2i k))
otherwise
Sparse router Using small network g. the sparse router computes with length equal to the number experts N. Subsequently, the Top- function retains top- K values the vector, while setting all other values to zero. Finally, the Sofimax function is applied to obtain the final routing vector. The whole process is shown as follows:
vector for each token,
Riporss (#:) = Soffmax (Top-K (g K))
Top-K K) = 1)=100 otherwise
v, if is in the top K
(13)
Soft Router Similar to the sparse router, the soft router computes score vector for each token through small network g. Subsequently, applies the Sigmoid function to the score vector and normalizes it, yielding the final routing vector. It can be formulated as:
Sigmoid g
Rug
(14)
Sum Sigmmiggo gzz:
A.2 LoRA-MoE
LoRA-MoE freezes the original parameters of the model to preserve world knowledge and introduces LoRA experts to learn new knowledge, thereby improving performance across multiple downstream tasks with few parameters.
Specifically, given frozen linear layer with weight matrix Wo E dhe LoRA-MoF creates N low-rank trainable matrix pairs Aş and Br. where A& Rd xr. B& E Rrxx and the rank & min (diee d.ut). As in the case of LoRA, A₁ is initialized with random Gaussian distribution, and Bg is initialized to zero. During training, the parameters of Wo frozen, and the parameters of Aş and B& are updated. The forward process of LoRA-MoF layer can be represented as:
h Wor. AWI; Wox; Σ R(::)A.Ba.,
(15)
where x; is the input token, the router in the LoRA-MoE layer, a is the learning rate scaling factor, and h is the output token. In ablation experiments, we transform each linear layer in the LLM into a LoRA-MoE layer with sparse router. The rank r 4, the learning rate scaling factor a the number of LoRA experts N = 5, and select the top 2 experts.
B Dataset
B.1
Data source
MedQA MedQA (Jin et al., [021] is open-domain multiple-choice question answering dataset for solving medical problems. These questions are sourced from professional medical board exams, which feature diverse content and typically demand comprehensive understanding of related medical concepts learned from medical textbooks in order to provide accurate answers. This dataset covers three languages: English, simplified Chinese, among which there are 12,723 QA pairs for English.
15

---

2019 is biomedical question answering dataset collected PubMedQA PubMedQA [Jin et al.
from PubMed abstracts. The task of PubMedQA to answer research questions with yes/no/maybe using the corresponding abstracts. It has IK expert-annotated, 61.2K unlabeled and 211.3K artificially generated QA instances. Each instance consists of: (1) a question which either an existing research article title or derived from one, (2) context which is the corresponding abstract without its conclusion. a long answer, which is the conclusion of the abstract and, presumably, answers the research question, and (4) yes/no/maybe answer which summarizes the conclusion.
Slake-VQA Slake-VQA [Liu el al. 2021 a semantically annotated, knowledge-enhaneed bilingual (English and Chinese) VQA dataset for radiology images. It contains 642 annotated images accompanied by 14,028 question-anse answer pairs, spanning 12 diseases, 39 organ systems, and 3 imaging modalities (CT, MRI, and X-ray). Questions are either open-ended (free-form) or closed-ended (balanced yes/no) related to various aspects of the image content such as plane, quality, position, organ, abnormality, size, color, shape, and knowledge graph.
PQt Path-VQA (He el al. 2020 is a pathology VQA dataset comprising 4,998 pathology images and 32,799 question-answer pairs. These pathology images are sourced from medical textbooks and online digital libraries. Each image is associated with multiple QA pairs pertaining to different aspects of the pathology including color, location, appearance, shape, etc. The dataset includes 16,465 open-ended questions, which make up 50.2% of the total and are categorized into six types: what, where, when, whose, how, and how much/how many. The remaining questions are close-ended "yes/no" questions, with a balanced distribution of 8,145 "yes" answers and 8,189 "no" answers. In the official dataset split, the training set, validation set and test set contain 19,755, 6,279 and 6,761 QA pairs, respectively.
SA-Med2D-20M SA-Med2D-20M. (Ye et al. 2023a is large-scale segmentation dataset of 2D medical images built upon numerous public and private datasets. It consists of 4.6 million 2D medical images and 19.7 million corresponding masks, covering almost the whole body and showing significant diversity. comprises 10 modalities, with CT and MR modalities being predominant in both the number of images and masks. Specifically, there are 2338,755 753 images and 12547,00 ,037 masks for CT and 2217,633 images and 7147,784 masks for MR. This is primarily attributed to their widespread presence in public medical image segmentation datasets and the 3D dimension of CT and MR scans, which yields a high volume of slices when segmented across three axes.
MIMIC-CXR MIMIC-CXR (Johnson et al. 2019 is large dataset of chest radiographs with free-text radiology reports. A total of 10 images are available in the dataset from 227,835 image studies collected for 65,379 patients. Each patient may have multiple studies and each study may contain one or more images associated with the same free-text report. Images in MIMIC-CRR are collected from multiple view positions: e.g., anterior anterior-posterion (AP), posterior- anterior, and lateral (LA). Protected health information (PHI) in radiology reports and images is removed, which results in missing information in some sentences of the reports.
The HIMIC-CX--IDE dataset is derived from MIMIC- MIMIC-CXR, providing JPG format files derived from the DICOM images and structured labels derived from the free-text reports. The aim of MIMIC- CXR-JPG is to provide a convenient processed version of MIMIC-CXR, as well as to provide a standard reference for data splits and image labels.
RadFM (Wu et al.) 2023b) processes radiology reports in MIMIC- by extracting the indication, findings, and impression sections, and removing redundant white spaces. Images without reports and reports where the findings section can not be extracted are discarded from both the training and test sets. Additionally, reports with findings sections exceeding 800 characters are filtered out. To enhance the model's 's capability to process images from different view positions, images of different orientations associated with the same report are treated as independent samples.
MPx MPx Wu et al. 2023b) is a report generation dataset collected from the MedPix website and organized by cases. Each case includes multiple radiologic scans, general clinical findings, discussions, and diagnostic results. Additionally, MPx provides scan-level annotations, such as image modality, shooting plane, and captions for each scan. The dataset is divided into MPx-Single and MPx-Multi, with annotations provided at the case level and scan level, respectively.
thtps://medpix.nill
16

---

MedMNIST v2 MedMNIST v2 Yang et al. 2023] is large-scale MNIST-like collection of standardized biomedical images, including 2D datasets with resolutions up 224x224 pixels and 3D datasets with resolutions up to 64x64x64 voxels. The 2D datasets include 12 subsets: PathM- NIST, ChestMNIST, DermaMNIST, OCTMNIST, PneumoniaMNNSST, RetinaMNIST, BreastMINIST, BloodMNIST, TiSSI, OrganAMMNIST, OrganCMNISST, and OrganSMNIST. The 3D datasets comprise 6 subsets: OrganMNIST3D. NoduleMNIST3DD FractureMNIST3D. AdrenalMNIST, Ves- seIMNIST3D, and SynapseMNIST33.. Covering primary data modalities in biomedical images, it is designed to perform classification on lightweight 2D and 3D images with various data scales (from 100 to 100,000) and diverse tasks (binary/multi-classs ordinal regression and multi-label). The comprehensive dataset, comprising approximately 708K 2D images and 10K 3D images, supports a wide range of research and educational purposes in biomedical image analysis, computer vision, and machine learning.
DermaMNIST, a 2D subset of MedMNIST v2, is based on HAM10000 (Tschandl et al. 2018 Codella et al. 2019]. a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Comprising 10,015 dermatoscopic images, the dataset is categorized into 7 distinct classes: actinic keratoses and intraepithelial carcinoma, basal cell carcinoma, benign keratosis-like lesions, dermatofibroma, melanoma, melanocytic nevi, and vascular lesions.
OrganSMNIST, another 2D subset of MedMNIST v2, is based on 3D computed tomography (CT) images from Liver Tumor Segmentation Benchmark (LiTS) [Bilic er al.) [023). Organ labels are obtained by using bounding-box annotations of 11 body organs from another study [Xu er all, 2019]. Hounsfield- Unit (HU) of the 3D images are transformed into grey scale with abdominal window. Subsequently, 2D images are cropped from the center slices of the 3D bounding boxes in sagittal views. Comprising 25,211 images, the dataset is categorized into 11 distinct classes: bladder, left femur, right femur, heart, left kidney, right kidney, liver, left lung, right lung, pancreas, and spleen.
Custom dataset splitting To prevent the model from encountering training images during testing, the official dataset split from Slake-VQA not utilized. Instead, we randomly divide all images into training and testing sets at a ratio of 6:1, along with their respective QA pairs and bounding boxes. Consequently, the training set comprises 550 images, 6018 English QA pairs, and 1421 bounding boxes, while the testing set includes 92 images, 1014 English QA pairs, and 201 bounding boxes.
For MIMIC-CES -IM,, JPG images provided in HIMIC-CXR-JPE and the corresponding reports from RadFM are used for the report generation task. The training set a subset of the original training set, containing 9,997 samples, while the test set remains the same the original test set, containing 3,858 samples.
B.2 Well-crafted datasets for REC and REG tasks
Slake-REC Slake-REG As a semantically-labeee knowledge-enhanced dataset for medical visual question answering. VQA provides bounding boxes for each object in the image. As shown in Figure (a), (a), the original format of each bounding box is [X, Y, W. M.. First, we convert to the [X 1Xmin, Ymin, Xmaz, Y Ymaz) format. Assuming the relative size of each image is 100x100, we then normalize each coordinate value in the bounding box to fall within the range of to 100.
As shown in Figure (c), in the REC task, an image and object name are given to find the object's bounding box. In the REG task, an image and object bounding box are provided to identify the object's 's name. The Slake-REs -EC and Slake-REG datasets are thus created.
SA-Med2D- SA- SAAED-RRE//SA-AMD--RERREE Each image in the SA-Med2D- 22MM dataset has one more masks, with each mask corresponding to an object. As shown in Figure 6 (b), we calculate the bounding box for each mask and normalize it to range of 0 to 100, resulting in bounding box for each object in the [Xmin: Y Ymin: Xmanx Ymaz] format.
The SA-Med2D-REES and SA-Med2D-REFE datasets are organized depicted in Figure (c). 10,000 samples each are selected from the CT and MR subsets as the training set, and 2,000 samples each selected as the test set.
17

---

Convert (XYWH to XYXY)
Normalization (0-100)
(a)
[X,YW, W,
[xmin min Xmax Y mmax] Processed Box
Original Box
Right Heart Ventricle
(b) SA-Med2D
Objoct Name
[mmi min Xmax max]
Object Box
Image
Object Mask
Bouding Box
Liver
[X Xmine Xmax Y Yaxx
Object Name
Object Box
[imins Vinnn X max Y max]
Liver
Object Box
Object Name
Image
REC REG
Image
Figure 6: Data production process for REC and REG tasks. (a) the process of transforming bounding boxes in Slake- VQA, (b) the process of obtaining bounding boxes from masks in SA-Med2D, (c) the input-output organization of REC and REG tasks.
B.3 Data availability
In the Table B.3 we list the links for each dataset, the number of samples in the training and sets, and their licenses.
Table 4: Data availability.
Dataset
Link
Train / Test Split
License
McdQA
http://giddutt.co///jdidiii 1/ModoAA
10178 1273
MIT Licease
PubMedQA
500 200/500 500
MIT License
Slake-VQA
6018 1014
Open Accoss
https://www.mmmlww.... www.w//wwwwwww...............
1421
Slake-REG
1421 / 201
Path-VQA
19755 6761
MIT License
SA Med-MeddD-D
Apache: 2.0 license
SA-Mcd2D-REC
20000 4000
Med2D-REG
20000
PhysioNet Credentialed
Health Data License 1.5.0
MIMIC-CXR
9997 3858
https://huggnngfac www.e/aeeeeseeeccaeaaaoa/aaaaaae
Apache-2.0 license
Open Access
MPX-Single
31416
https://uggingg face www.ataseess/chhaai..vooo w//Raud _dttu_dddddMMddd_cs
Apache
DemaMMNSS
http::/mnedmmiss.com
7007 2005
Apacho-2.0 License
OrganSMNIST
http://hneemmiiil.com
13932 / 8827
Agache 2.0 License
C Multi-task instruction template
We have designed different instruction templates for different datasets. During the training process, when sample from dataset is selected, an instruction template is also sampled from the corre- sponding dataset's template pool and used to format the sample. Examples of instruction templates for each dataset shown below.
18

---

MedQA
Example 1: [qa] A researcher evaluates healthy breast tissue from 100 women, 50 women that pregnant at time the study and age-matched non-prenee non-pregnant women. The breast tissue pregnant women contained increased number of acinar glands with epithelial proliferation compared to the non-pregnant women. Which process caused this change?
[qa] If you are doctor, please answer the following question briefly: a researcher evaluates healthy breast tissue from 100 women, 50 women that were pregnant the time of the study and 50 age-matched non-pregnant women. The breast tissue in pregnant women contained an increased number of acinar glands with epithelial proliferation compared the non-pregnant women. Which process caused this change?
Example 2:
PubMedQA
Example 1: [qa] Does the severity of obstructive sleep аpnса predict patients requiring high continuous positive airway pressure?
Example 2: [qa] If you are doctor, please answer following question using "yes", "no" or "maybe": does the severity of obstructive sleep apnea predict patients requiring high continuous positive airway pressure?
Slake-VQA VQt
Example <Img> <ImageFcature>> </Img> [vqa] What modality is to take image?
Example 2: <Img> <lmturFeature> </Img> [vqa] Based on image, respond to this question with short answer: what modality is used to take this image?
Slake-REC SA-Mcd2D-REC
Example <Img> <lmageFFcature> </Img> [refer] Liver.
Example 2: <lnggeeetneeeetttteeee> </In> [refer] Give me the location liver.
Example 3: <Img> <hwgg//aaaaaacuuuuuc///gg [refer] Where is liver?
Example 4:
[refer] From this image, tell me the location of liver.
Example 5: <inngegeuiaa [refer] The location of liver is
Example 6: claggeelceeee </Img> [refer) Could you tell the location for liver?
Example 7: <Img> <lmageFcatuur>> </Im>> [refer] Where
locate the liver?
Slake-REG SA-Med2DD-RFC
Example 1: <lmg> <lmageeFeattuuree <'Img> [identify] <16><36><<4><66>>
Example 2: <Img> <lmageFcature> </Img> [identify] What object is in this location
<16><3><<42<611>
19

---

Example 3: <lmg> <ImageFeeture> </Img> [identify] Identify the object present at this location
Example 4: </mng What is in <1>><6><422<61>>>>>>> 3
Example 5: </ (identify) Describe this in
Example 6: <Img> <lnng>><tnnggeelluuue>>>m/gg [identify] This <16><<66<44><<11>>>
Example <lmg> <lnageFeatture> </Img> [identify] The object in <162<36><42261>1
MIMIC-CXR
Example 1: <Img> <lmagcFeatture>>>eeeeeoee </Img [caption] Describe the given chest x-ray image detail.
Example 2: <Img> <ImageFeature>> </Img> [caption] Take a look at this chest x-ray and describe the findings and impression.
Example 3: <lmg> <lmageFeature> </Imgg </Img> [caption] Could you provide a detailed description of the given x-ray image?
Example 4: <lmg> <ImageFeature> </Img> [caption] Descrihe the given chest x-ray image detailed as possible.
Example 5: <Img> <lmageFeature>> </Img> [caption] What are the key findings in this chest x-ray image?
MPx-Single
Example 1: <Img> <lmageFeature>>> </Img> [caption] Describe this input image.
Example 2:
<Img> <lmageFeatuure> </Img> [caption] Help captioning the image.
Example 3: <Img> <lmageFcature> </Img> [caption] What be inflected from the scan?
Example <lmg> <lmageFcatuurs </Img> [caption] Can you give caption this image?
Example 5: <lmg> <Image </lmgg [caption] Can you provide brief summary the radiology image?
Example 6: <lmg> <lmageFecccctt </Img> [caption] Please write report about the image?
Example 7: <Img> <lmageFaatuuee </Ig>> [caption] Can you provide analysis of this image?
Example & <lmg> <lmageeeattte </Ig>> [caption] Can you explain what shown this image?
Example <Img> <ImageFcature>> </Img> [caption] What can be indicated from radiologic scans?
Example 10: <lmg> <lmageFcature> </Img> [caption] What can you infer from this photograph?
20

---

DermaMNIST
Example: <lmg> <Imagefeeture>> </lmg> [els] Which calegory does multi-source dermaloscopic image of common pigmented skin lesions belong to: actinic keratoses and intraepithelial carcinoma, basal cell carcinoma, benign keratosis-like lesions, dermatofibroma, melanoma, melanocytic nevi, or vascular lesions?
OrganSMNIST
Example: <Img> <ImageFeature>> </Img> [cls] Which category does CT image helong to: bladder, left femur, right femur, heart, left kidney, right kidney, liver, left lung, right lung, pancreas, spieen?
D Experiments
D.1 Evaluation metrics
F1 Score Assuming m is the number of common words in the candidate C and the reference R with the number of words of and the precision and recall for a candidate sentence can be calculated as:
precision
(16)
772
recall
(17)
Considering class imbalance, Fl score is used to evaluate the performance of the model on both the VQA and REG tasks, which means the harmonic mean of precision and recall. A higher average Fl score for the dataset indicates a higher performance of the model.
precision recall
Fl
(18)
precision + recall
BLEU-N We use BLEU-1 to assess the model's performance on both the VQA and REG tasks, while employing both BLEU-1 and BLEU-4 to evaluate its performance in the report generation task. Given the candidate C' and reference R, BLEU-N is defined as:
Egram EC Countetsp (gram))
BLEU-N
(19)
E Egrap EC Count(gramyy)
When N=1, the above formula calculates BLEU-1; when N=4, it calculates BLEU-4.
ROUGE-N We use ROUGE-1 and ROUGE-2 to evaluate the performance of the model on the RG task. Given the candidate C and reference R, ROUGE-N is defined as:
Egram ye Countmates (grama)
ROUGE-N
(20)
Egramy ER Count(gram
When N=1, the above formula calculates ROUGE-1; when N=2, it calculates ROUGE-2.
ROUGE-L ROUGE-L also used evlaute the quality of the generated text on the task of report generation, which stands for recall-oriented understudy for gisting evaluation with the longest common subsequence. Given the candidate C' and reference Rt, let LCS(C, R) be the length of the longest common subsequence, which is determined by using dynamic programming, it can be an defined
)
ROUGE-L RLCS 3 P2Pcs
(21)
where RLCS LCS(C,R) PLcs LCS(C.R) B PLCs Le and LR represent the length of the Ru.s'
Le candidate and reference. A higher ROUGE-L score means that the generated text shares more of the same sequences words as the reference text, which typically indicates better quality terms of capturing the salient points of the reference.
21

---

METEOR METEOR is also used to evlaute the quality of the generated text on the task of report generation, which stands for metric for evaluation of translation with explicit ordering. METEOR for sentence computed as:
precision recall
METEOR (1 p) precision + (1 (1 x recall
(22)
where p - ( is the penalty factor. ch is the number of chunks, which means a contiguous ordered block. at, and are hyperparameters determined according different datasets.
RadGraph F1 To assess the semantic accuracy in the task of report generation, RadGraph FI computes the overlap in clinical entities and relations between a machine-generated report and andiologist-generatee report. Specifically, following the criteria in RadGraph [Jain er al. 2021] two entities are matched if their tokens (words in the original report) and labels (entity type) match. Two relations are matched if their start and end entities match and the relation type matches. RadGraph F1 metric computes the overlap in entities and relations separately and reports their average.
RadCliQ RadCliQ (radiology report clinical quality) is also used to assess the semantic accuracy in the task of report generation. Two versions of the RadCliQ metric: RadCliQ-vó and RadCliQ-vl both use machine learning model to take in values from other metrics, such as BERTScore and CheXbert vector similarity, and then produce a composite score based on these input values, which predict the total number of errors in report.
IOU We use IoU (Intersection over Union) evaluate the performance of the model on the REC task. can be formulated as:
PnG
IoU
(23)
PUG
where P is the prediction of the model, G is the area of the ground truth.
R@0.5 We alse use @@.5 evaluate the performance of the model on the referring expression comprehension task. R stands for recall, and 0.5 denotes the IoU threshold. When the loU between the prediction and the ground truth is greater than or equal to 0.5, it is considered true positive (TP). When the IoU is less than 0.5, it is considered false negative (FN). Therefore, for sample with only bounding box, R @05 can be formalized
TP
I, loU 2 0.5
R@0.5
(24)
TP+FN
0,
loU 0.5
D.2 Data leakage issue of RadFM on MPx-Single
When we directly use the model checkpoint provided by RadFM open-source repository for model inference, we find that the model outputs for many samples were completely consistent with ground truth. This issue only occurs on the MPx-Single dataset, while we strictly follow the official test set split. This appears to be unreasonable, raising suspicions of potential data leakage. Here are some examples:
Example
MetF117
Instruction: Describe this input image.
Ground truth: Neuroradiology Quiz
RadFM: Neuroradiology Quiz
Uni-Med: axial noncontrast ct image shows large, well-cireumseribed.. non-aggressive appearing mass in the right frontal lobe.
22

---

Example
Instruction: Describe this input image.
Ground truth: Focal necrosis of splenium of corpus callosum secondary to subacute diffuse axonal injury.
RadFM: Focal necrosis of splenium of corpus callo- sum secondary to subacute diffuse axonal injury.
Uni-Med: there axial mass in the right frontal region. the mass is predominantly cystic with a few foci solid tissue. there is evidence of hemorrhage or enhancement on the mr sequences.
large, well-cireumseribed, extra-
Example
Instruction: Describe this input image.
MedPix R
Ground truth: MRI day 2- hydrocephalus Acute bi- lateral cerebellar infarcts Acute bilateral thalamic infarcts Acute right occipital lobe infarct MRA no flow in distal basilar artery or posterior cerebal arteries.
RadFM: MRI day hydrocephalus Acute bilateral cerebellar infarcts Acute bilateral thalamic infarcts Acute right occipital lobe infarct MRA no flow in distal basilar artery posterior cerebal arteries.
11fd-ddb9bbe85555 Uploa Steven Goldstein,12-00
Uni-Med: acute right mca infarct. acute infaretion of the right cerebellar hemisphere. acute infarction of the right brainstem. acute cerebral edema.
D.3 Ablation on special token and identifier
We have designed vision-level special task tokens and text-level special task identifiers for visual features and text prompt, respectively. Through ablation experiment, we verify whether they have a positive effect on model performance. As shown in Table 5 we observe that text-level special task identifiers bring limited improvement. In contrast, vision-level special task tokens significantly improve the model's overall performance on all datasets, further illustrating the effectiveness of mitigating the -g-oof-war problem at the connector.
Table 5: Ablation Experiments on special token and identifier.
Special Token/ Identifter
VQA
REC loU
REG BLEU-1
RG BLEU-1
CLS Accuracy
Total
Text-level Vision-level
BLEU-1
Avg.
(t)
(1)
(1)
MLP
79.81 56.48
35.18
16.25
74,54
58.12
18.55
15.50
76.26
73.64
$1.59 57.35
1.9%
36.76
18.74 9.9%
76.07
5831
1.4%
2471
15:42
10.4%
74,46
70.07
0.5%
6.0%
81.33 57.29
1.7%
37.85
20.14
15.7%
77.23
62.72
5.5%
23.29
15.74
13.6%
76.76
76.55
2.3%
CMoB
233
248
1561
76.56
$1.52 57.75
2.2%
37.54
20.30
15.8%
77.45
60.2
3.7%
24.70
15.55
16.7%
75.61
7692
1.8%
8.8%
D.4
Visualization analysis of visual features on image modalities
We use t-SNE to visualize the distribution of visual features by modalities in Figure We first observe the visual feature distribution of different modalities under the same task in Figure 7 (a-c). The feature of CT and MRI modalities in the REG task already have good discriminability after
23

---

Silhoactic score: 0.3049
Silboudiec score: 0.3232
Silhoucte soore: 0.3335
1.0
6.8
Matally
0.
50
05
1.
0.
0.8
3.
09
04
06
3.0
06
1.4
28
08
08
24
0.5
Xeo
Posta
62
PCT
1.
1.8
1.4
1.
3.0
10 0.6
(d)
(f)
Figure 7: Visual features distribution image modalities. (a)-(c) The feature distribution of CT and MRI modalities in the REG task. (a) passing through the frozen visual encoder. passing through the MLP connector. (c) passing through the CMoE. (f)-(f) The feature distribution of 8 modalities after passing through the frozen visual encoder. (d) EVA-CLIP ViT-G/14. (e) CLIP ViT-L/14. (f) BiomedCLIP VIT-B/16.
passing through the frozen visual encoder. After passing through the connector, the improvement in Silhouette score (from 0.3049 to 0.3335) relatively limited. addition, we select 100 samples from each of the 8 modalities and observe their visual feature distributions after passing through different visual encoders Figure (d-f). It can still be observed that visual features of different modalities already have specific patterns in the feature space, whether using EVA-CLIP, CLIP or BiomedCLIP.
The above findings also provide an explanation for why we attempt to explicitly introduce task information instead of modality information in CMoE. When aligning visual and language embedding spaces through the connector Uni-Med's multi-modal and multi- scenario, task information more difficult to distinguish than modality information.
D.5 Performance of Uni-Med on REC and REG tasks
We report the metrics of Uni-Med on the tasks of referring expression comprehension and referring exression generation in Table 6 The mean and standard deviation of performance of Uni-Med are obtained after several 300k iterations.
Table 6: Performance of Uni-Med on REC and REG tasks.
Task
Dataset
Metric
Uni-Med
ToU
37.7110.52
Slake-RBC
R@0.5
39.30=10.76
Referring Expression Comprehension
21.60.12.19
SA-Med2D-REC
14.42:+:3220
BLEU-1
75.78:42.05
Slake-REG
F1
1..3
Accuracy
1..1
Referring Expression Generation
BLEU-1
6..47:1:1.76
SA Med2D. REG
F1
12.17=+1.90
Accuracy
57.69-+1.07
24

---

D.6 Comparison of architecture capability between Uni-Med and LLa LLaVA-Med
In addition to directly compare the capability of existing models, we take LLaVA-Med as an example to compare the capability of model architectures.
the checkpoints of the second stage (medical instruction tuning) perform two strategies of L.M full parameter fine-tuning: (1) Dataset-specific fine-tuning; (2) Joint training fine-tuning. The data split and the prompt format completely consistent with Uni-Med and LLaVA-Med, respectively. Both strategies last for epochs (the same as Uni-Med). The results are shown in Table
Specifically,
Following the model architecture of LLaVA-Med, there a serious tug-of-war problem when we implement joint fine-tuning strategy multiple tasks and datasets. While the strategy of dataset- specific fine-tuning has significantly improved the evaluation metrics each dataset.
It is worth noting that Uni-Med has achieved competitive and leading results through joint training, without dataset-specific fine-tuning. It can be concluded that the model architecture of Uni-Med, especially the design of CMoE, has achieved superior solution to the tug-of-war problem, which reduces interference and promotes more efficient knowledge sharing.
Table 7: Comparison of architecture capability between Uni-Med and LLaVA-Med. We utilize dataset-specific fine-tuning and joint training fine-tuning on LLaVA-Med, respectively.
LLaVA-Med
Uni-Med Joint Training
Task
Dataset
Metric
Joint Training
Dataset-specifie
BLEU-1
33.69
72,00
82.12
Slake VQA
Visual Question Answering
35.83
73.07
83.07
BLBU-1
37.79
56.86
58.07
Path-VQA
FI
38.55
57.51
58.74
BLEL
4.86
4.96
6.46
ROUGE-1 1
26.11
28.28
28.81
CXR
ROUGE-2 2
7.66
9.01
9.62
ROUGE-I
20.61
22.58
METEOR
8.73
8.89
10.59
Report Generation
BLEU-1
15.11
14.63
15,80
BLEU-4
2.4
1.75
2.47
MPx-Single
ROUGE-1
13:22
13.03
14.32
ROUGE-2
2.19
2.68
ROUGE-L
10.85
12.29
METEOR
5.83
5.79
5.92
Image Classification
DermaMNIST
Accuracy
25.84
79.95
76.96
OrganSMNIST
Accuracy
66.80
77.84
78.07
Toll
22.41
37.71
Slake-REC
R@0,5
39.30
Referring Expression Comprechesiium
loU
8.64
17.67
21.60
SA-MoIDDREC
R@0,
4.75
9.98
14.42
BLEU-I
27.21
50.79
75.78
Rlake-REG
Fl
30.97
53.15
77.35
Accuracy
20.40
44.78
68.16
Referring Expression Generation
61.47
SA MedZID REG
47.11
55.98
62.17
Accuracy
40.80
50.92
57.69
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly state our contributions and scope in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
26

---

Justification:
Although the paper does not include theoretical results, but uses and combines existing theoretical methods for interpretability analysis about the tug-of-war problem. All assumptions are clearly stated or referenced in the statement.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We fully disclose all the information needed to reproduce the main experimen- tal results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
27

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Code and resources are
available at https: com/
tsinghua- tsinghua-msiippUne www.mmmmiiippaaaeeeeee
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided all the training and test details including data splits, hyper- parameters, how they were chosen, type of optimizer, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In overall comparison, we report the mean and standard deviation of perfor- mance of our model on all tasks, which is calculated through three different random seed configurations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided the information about computer resources such as GPU and time of execution the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: cccppppp.cccpppeeeee http://p.....p.bb.c//ccccccc.uda
Answer: [Yes]
Justification:
This research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed the potential positive societal impacts in section of Intro- duction and Conclusion, and negative societal impacts in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
For our open-source model, providing effective safeguards a challenge.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All creators or original owners of assets used in the paper are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have produced new datasets for several tasks and described the processing process in Appendix B.2.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "N1VBrmOFTR", "reviewer": "Reviewer_mvVf", "summary": "This paper introduces Uni-Med, which applies mixture of experts at the connector level for efficient training toward a unified medical multi-modal foundation model. The contributions of this work include 1.) curation of indexes to quantify tug-of-war problem in multi-modal multi-task modal; 2.) a novel perspective of applying Connector MoE for multi-modal multi-task model, which enables efficient training; 3.) comprehensive ablative studies to evaluate various configurations for different modules; 4) commitment to providing open-source code and weights of the proposed method.", "questions": "1. Line 143 is missing an introduction about the soft router.", "limitations": "Most of the limitations are discussed in the Appendix, while some of them are not addressable.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. It is a technically-solid work. The mitigation of the tug-of-war problem is justified from multiple perspectives, including developed indexes, parameter statistics scores, routing weights, and tSNE feature visualization. \n2. The proposed framework is evaluated thoroughly in ablation studies. \n3. Consistent improvements over existing open-source medical foundation models are observed. \n4. The presentation is clear and easy to follow.", "weaknesses": "1. While the reviewer appreciates the acknowledgment of several limitations of this work, it would be better to mention them in the main text, especially limitation no. 5 in lines 662-663. If the space is not sufficient, at least they should be briefly mentioned in the main text, and a reference to detailed limitations should be provided.\n2. Confusion about the training/fine-tuning details: for models presented in Table 3, are they individually fine-tuned for each dataset based on the split introduced in the appendix? How is the fine-tuning implemented? Is it end-to-end or LoRA fine-tuning? For the Uni-Med, the reviewer understands that it is trained on all datasets appearing in Table 3, but there is no individual fine-tuning. Is this understanding correct?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments!\n\n \n\n**Q1:** While the reviewer appreciates the acknowledgment of several limitations of this work, it would be better to mention them in the main text, especially limitation no. 5 in lines 662-663. If the space is not sufficient, at least they should be briefly mentioned in the main text, and a reference to detailed limitations should be provided.\n\n \n\n**A1:** Thank you for your suggestion. We have realized that the limitations of our work and related references should be mentioned in the main text. We will add a limitation section in the main text of the revised version.\n\n \n\n**Q2:** Confusion about the training/fine-tuning details: for models presented in Table 3, are they individually fine-tuned for each dataset based on the split introduced in the appendix? How is the fine-tuning implemented? Is it end-to-end or LoRA fine-tuning? For the Uni-Med, the reviewer understands that it is trained on all datasets appearing in Table 3, but there is no individual fine-tuning. Is this understanding correct?\n\n \n\n**A2:** Thank you for your questions. We will elaborate on the implementation details of the models in Table 3 to reduce your confusion.\n\n- The understanding of Uni-Med is basically accurate. Uni-Med achieves **joint training** on 6 six distinct medical tasks and 12 datasets, requiring only **one-stage training** on a single A800 GPU and **no task/dataset fine-tuning**. It strictly follows the dataset split introduced in the appendix.\n\n- For model comparison, we use readily **available model checkpoints for testing**. The details are as follows: (1) **About training data and model type**. Except for Med-Flamingo, the raw training data of the comparison models all contain some of the datasets in Table 3. For example, LLaVA-Med uses full parameter fine-tuning on the Slake-VQA and Path-VQA, respectively, which means it offers different dataset-specific model checkpoints. XrayGPT is task-specific model, and it traines on MIMIC-CXR. RadFM is a generalist foundation model and its training data includes Slake-VQA, MIMIC-CXR and MPx-Single. (2) **About datasets split**. We use the **official test set split** for all datasets, except for Slake-VQA, as we utilize it to build data for other tasks. In this case, there is an unfair comparison between Slake-VQA, but it's actually unfair to Uni-Med because part of the test data is used for training in other models.\n\n- A completely fair comparison across different model architectures is to use the **same dataset split for training and testing**. The medical MLLMs used for comparison all followe the **standard architecture** consists of a vision encoder, a connector (e.g. XrayGPT: linear layer; LLaVA-Med: MLP), and an LMM. From this perspective, the experiments of the connector using linear layer or MLP in Table 2, **to some extent, represent** the results of XrayGPT and LLaVA-Med frameworks that are **fully consistent with the Uni-Med training and fine-tuning strategy**, respectively.\n\nThank you for recognizing our extensive experiments and analyses. To reduce the confusion of readers, we will add more implementation details in the revised version.\n\n \n\n**Q3:** Line 143 is missing an introduction about the soft router.\n\n \n\n**A3:** Thank you for your meticulous review and reminder. The soft router receives input tokens and calculates the routing weights for each expert. We will add the missing introduction about the soft router in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eFQTDrVXwg", "reviewer": "Reviewer_Snz8", "summary": "The authors propose to build a medical generalist multi-modal foundation model using a novel \"connector mixture of experts\" module to solve the problem of \"multi-task\" learning. Their connector-MOE technique introduces a projection and routing module from the visual encoder into the LLM that is explicitly conditioned on the underlying task. The model is tested on a set of medical tasks, and benchmarked against several other multimodal medical models demonstrating improved performance.", "questions": "In the discussion of MOEs and the paper motivation I think that it could be clarified substantially. Line 97-105 clearly disambiguates the usage of the term MoE, and it might be helpful to do this sooner in the introduction to help clarify this work for readers.\n\nGeneralist foundation model seems to be redundant? Isn't a \"Foundation model\" by definition \"generalist\"?\n\nA simple linear projection and purely autoregressive design with visual instruction fine-tuning like LLaVA learns to implicitly condition the projected visual tokens on task and is the appropriate benchmark for this explicit routing framework. \n\nI feel like the paper would, in general, benefit from framing it as an investigation into novel connectors with which to build any foundation model - medical or otherwise. Framing it as a novel medical foundation model focuses on the wrong thing, and makes me wonder why it isn't tested on a broader range of medical tasks and datasets, scaled up and down, and so forth.", "limitations": "No limitations are mentioned, and I think that this is a missed opportunity. There are clearly limitations with regards to the comparisons to other models, datasets involved with these comparisons, and overall size of the involved models utilized in Uni-Med.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The explicit task conditioned projection is novel, and integrates an older concept of MoEs into a SOTA MLLM framework.\n\nThe authors do an excellent job sourcing and assembling a large, multi-task, multi-modal set of medical benchmarks and performing an extensive set of ablations.\n\nThe paper is benchmarked broadly across multiple tasks - fitting the definition of a foundation model.", "weaknesses": "I disagree with the claim that there is limited research on connecting modalities in multi-modal models. This is an area of immense interest and extensive research broadly within the field of machine learning. \n\nI don't think that it is a unique medical issue, and the benchmarking of a novel architecture would be better served using more common datasets to the ML community. This is particularly important because, as the authors note, the medical datasets and models involved were hard for them to control for data leakage. Why not compare the routing technique against non-medical models on non-medical datasets where this issue won't be the case?\n\nThe results are hard to follow in two very large Tables Table 2 and 3, particularly Table 3 which is the comparison to existing standards. The explanation of these comparisons is quite brief, and not well described in the text despite being the primary comparison for the paper. \n\nThe paper cites data leakage as being an issue, but isn't dataset shift an issue too? If I understand correctly, Uni-Med is being tested on held-out samples from datasets it was trained on, while the other LLMs are being tested on a mixture of data that in some cases was even in their training datasets? A fairer comparison would be to utilize the same datasets across model architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments!\n\n \n\n**Q1:** I disagree with the claim that there is limited research on connecting modalities in multi-modal models.\n\n \n\n**A1:** There is no doubt about that researches about connecting modalities in multi-modal models is popular and extensive. As mentioned in lines 4-6, 50-52, and 104-105, our viewpoint is that research focus on the field of **connectors in MLLMs**. We will highlight this scope of our research in the revised version to reduce your misunderstanding.\n\n \n\n**Q2:** I don't think that it is a unique medical issue, and the benchmarking of a novel architecture would be better served using more common datasets to the ML community. As the authors note, the medical datasets and models involved were hard for them to control for data leakage. Why not compare the routing technique against non-medical models on non-medical datasets?\n\n \n\n**A2:** Thank you for your suggestion.\n\n- First, we agree that this is not a unique medical issue. However, when we construct medical MLLMs based on multi-modal and multi-task scenarios, we observe the tug of war problem at the connector level within standard MLLM architectures. This is the **background** of our proposal of Uni-Med. \n- Second, it is of great significance to research and develop the medical generalist foundation model. Providing a superior solution to the tug-of-war problem which is particularly serious due to the diversity of image modalities and tasks in the medical field becomes our basic **motivation**. We believe that **\"solving problem in prominent fields\" and \"exploring generalization in general fields\" are equally important contributions** to the machine learning community.\n- Third, not only in the medical field, but also in general field, model evaluation is plagued by data leakage issue. In our work, the data leakage issue is only observed on the MPx-Single using the model provided by RadFM. We ensure that all experiments of Uni-Med are free of data leakage issue and the results are **reliable**. \n\n- Fourth, we conduct **preliminary exploration of generalization** of our method in general field. We fully follow the training strategy of LLaVA-1.5 and report metrics on 9 benchmarks with/without CMoE in **Table Re.2**. The results show that the introduction of CMoE brings significant improvements on all benchmarks.\n\n \n\n**Q3:** The results are hard to follow in two very large Table 2 and 3. The explanation of these comparisons is quite brief, and not well described in the text despite being the primary comparison for the paper.\n\n \n\n**A3:** Thank you for your suggestion. Due to the page limit of the paper, we acknowledge the inadequacies in our writing regarding the explanation of experimental results. We will **add these details** in the revised version. We hope to make the readers feel that the presentation is clear and easy to understand.\n\n \n\n**Q4:** If I understand correctly, Uni-Med is being tested on held-out samples from datasets it was trained on, while the other LLMs are being tested on a mixture of data that in some cases was even in their training datasets? \n\n \n\n**A4:** Your understanding is accurate. For other medical MLLMs, we use readily available model checkpoints for testing. We clearly know that a completely fair comparison would be to utilize the same datasets across model architectures. But we have already achieved relative fairness:\n\n- We use the **official test set split** for all datasets, except for Slake-VQA, as we utilize it to build data for other tasks. In this case, there is an unfair comparison between Slake-VQA, but it's actually unfair to Uni-Med because part of the test data is used for training in other models.\n- As for the fact that we report RadFM's data leakage in the MPx-Single dataset (RadFM opens source this dataset and provides split), we conduct testing **strictly according to the split**, which only indicates that RadFM's model checkpoint may not have been trained according to this split.\n- **None of the ablation experiments has data leakage issues**, and the effectiveness of CMoE in any configuration is **reliable**.\n\n \n\n**Q5:** Line 97-105 clearly disambiguates the usage of the term MoE, and it might be helpful to do this sooner in the introduction to help clarify this work for readers.\n\n \n\n**A5:** Thank you for your suggestion. As mentioned in line 50-53, current research to mitigate the tug-of-war problem mainly tailors the MoE approach to the language model components, overlooking the potential benefits of exploring and enhancing the connector. We will provide a clearer presentation of our motivation and the usage of MoE in the introduction section.\n\n \n\n**Q6:** LLaVA learns to implicitly condition the projected visual tokens on task and is the appropriate benchmark for this explicit routing framework.\n\n \n\n**A6:** In fact, all experiments with linear and MLP connectors in Table 2 have model architectures **consistent with LLaVA and LLaVA-1.5**, respectively. In addition, CMoE with token-level router strategy is also an implicitly condition projection architecture. We hope these explanations are helpful for you to understand our benchmark settings.\n\n \n\n**Q7:** The paper would benefit from framing it as an investigation into novel connectors with which to build any foundation model. Framing it as a novel medical foundation model focuses on the wrong thing, and makes me wonder why it isn't tested on a broader range of medical tasks and datasets.\n\n \n\n**A7:** We elaborate on the background and motivation for why we chose the medical field for research in the first and second point in **A2**. With current computing resources, we have trained and tested on as wide a range of medical tasks as possible. **Compared to the existing medical MLLMs**, Uni-Med have added more diverse tasks and datasets. We do indeed look forward to having more data and a wider variety of tasks to validate our method. We will continue to focus on this in our future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pz1WhROLAp", "reviewer": "Reviewer_b87p", "summary": "The paper presents Uni-Med, a medical generalist foundation model designed to perform multiple medical tasks efficiently through multi-task learning. This model introduces a Connector-Mixture-of-Experts (CMoE) module to mitigate the tug-of-war problem in multi-modal, multi-task optimization, which is a common issue in current models. Uni-Med achieves competitive or superior performance across six medical tasks without requiring task-specific fine-tuning.", "questions": "Is the number of projection experts correlated with the number of tasks or the number of image modalities?\n\nFigure 5 shows that visual features of the same task are more tightly distributed. How would t-SNE behave for different modalities? Why are visual features related to tasks? For a single image, it can be used for both classification and report genertaion.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Multi-modal multi-task optimization is a complex and important question for large multimodal models, the introduction of the Connector-Mixture-of-Experts (CMoE) module, employs a mixture of projection experts to align visual and language embedding spaces, shows superior performance on multiple tasks.\n2. The paper conducts a comprehensive interpretation analysis of the problem from the perspective of gradient optimization and parameter statistics.\n3. Extensive experiments demonstrate Uni-Med's effectiveness across multiple tasks and datasets.", "weaknesses": "1. The model currently supports only 2D images, whereas most commonly used medical imaging modalities, such as CT and MRI, are in 3D.\n2. For the report generation task, the evaluation should include metrics like RadGraph Score and RadCliQ, as BLEU and ROUGE cannot fully assess the semantic accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments!\n\n \n\n**Q1:** The model currently supports only 2D images, whereas most commonly used medical imaging modalities, such as CT and MRI, are in 3D.\n\n \n\n**A1:** Thank you for your advice. Same as most medical MLLMs, we input **2D slices and corresponding questions for 3D images** such as CT and MRI. We acknowledge that Uni-Med has certain limitations in handling genuine 3D medical image inputs. The primary challenge lies in the need for different visual encoders to process 3D images effectively [1]. Replacing the visual encoder to handle 3D images would compromise our ability to process 2D image datasets effectively. This remains an area for future research.\n\n \n\n**Q2:** For the report generation task, the evaluation should include metrics like RadGraph Score and RadCliQ, as BLEU and ROUGE cannot fully assess the semantic accuracy.\n\n \n\n**A2:** Thank you for your suggestion. Firstly, we learne the concepts of the two metrics mentioned above:\n\n- RadGraph-based metrics. The RadGraph model [2] parses radiology reports into graphs containing clinical entities and relations between them. The RadGraph F1 metric computes the overlap in entities and relations separately, then reports their average.\n\n- RadCliQ. Radiology Report Clinical Quality (RadCliQ) is a composite metric that integrates RadGraph F1 and BLEU score in a linear regression model to predict the total number of errors in a report [3].\n\nSecondly, we calculate and report RadGraph entity F1, RadGraph relation F1, RadCliQ-v0 and RadCliQ-v1 on MIMIC-CXR dataset in **Table Re.1**, using the code released by Yu et al. [3]. The improvement of RadGraph-based metrics and the decrease of RadCliQ both indicate that **Uni-Med achieves better semantic accuracy in the report generation task**.\n\n \n\n**Q3:** Is the number of projection experts correlated with the number of tasks or the number of image modalities?\n\n \n\n**A3:** Thank you for your question. We hold the opinion that both are important, and the answer to this question needs to be **based on the actual situation**. We analyze the following three scenarios:\n\n- **Single modal, multi-task.** A same image may need to complete different tasks, and the number of experts should be related to the number of tasks.\n\n- **Multi-modal, single task.** The setting of the number of experts should consider the number of modalities.\n\n- **Multi-modal, multi-task.** In this scenario, further analysis of the data is required. Taking Uni-Med as an example, when we try to visualize visual features separately by task and modality, we find that the distribution of features by task is more chaotic, while the distribution of features by modality is more orderly (Detailed in Q4 & A4). Therefore, CMoE needs to consider task information more, and the number of experts is more related to the task.\n\n \n\n**Q4:** Figure 5 shows that visual features of the same task are more tightly distributed. How would t-SNE behave for different modalities? Why are visual features related to tasks? For a single image, it can be used for both classification and report generation.\n\n \n\n**A4:** Thank you for your question.\n\n- In **Figure 5**, we visualize the distribution of visual features by tasks. It can be clearly observed that the distribution of features by task is chaotic in **Figure 5 (a)**, which means that there is no obvious discrimination between different tasks after passing through the frozen visual encoder. Visual features of the same task are more tightly distributed after CMoE in **Figure 5 (c)** than MLP in **Figure 5 (b)**. \n\n- We use t-SNE to visualize the distribution of visual features by modalities and provide the results in **Figure Re.1**. Specifically, we first observe the visual feature distribution of different modalities under the same task in **Figure Re.1 (a-c)**. We find that the feature distributions of CT and MRI modalities in the REG task have good discriminability after passing through the frozen visual encoder. After passing through the connector, the improvement in Silhouette score (from 0.3049 to 0.3335) is relatively limited. In addition, we select 100 samples from each of the 8 modalities and observe their visual feature distributions after passing through different visual encoders in **Figure Re.1 (d-f)**. It can still be observed that the majority of modality distributions are ordered and tightly packed.\n\n- The above findings also provide a new perspective for explaining the explicit task conditioned projection in CMoE. When aligning visual and language embedding spaces through the connector in Uni-Med scenario, **task information is more difficult to distinguish than modality information**.\n\n- As described in the question, for a single image, it can be used for different tasks. If the visual features are not related to the task, then the tokens of the same image input into LLM are exactly the same (through linear, MLP, and token-level CMOE in Table 2). In this case, achieving multitasking relies entirely on the capabilities of LLM. On the contrary, we assume that different tasks require attention to different image features. After passing through the connector, the features of the same image for different tasks have adaptively changed, which can alleviate the negative impact of the tug of war problem in multi-task learning on LLM. The significant improvement in experimental results confirms the latter assumption.\n\n \n\n**References**\n\n[1] Bai F, Du Y, Huang T, et al. M3d: Advancing 3d medical image analysis with multi-modal large language models[J]. arxiv preprint arxiv:2404.00578, 2024.\n\n[2] S Jain, A Agrawal, A Saporta, et al. RadGraph: Extracting clinical entities and relations from radiology reports. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, December 2021.\n\n[3] Yu F, Endo M, Krishnan R, et al. Evaluating progress in automatic chest x-ray radiology report generation[J]. Patterns, 2023, 4(9).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fz6rWG4hzM", "reviewer": "Reviewer_wK4x", "summary": "The paper introduces Uni-Med, a medical generalist foundation model designed for multi-task learning across six different medical tasks. The proposed CMoE module leverages a mixture of projection experts to align visual and language embedding spaces effectively. The model demonstrates significant performance improvements across diverse medical tasks, validated through extensive experiments and ablation studies.", "questions": "- More detailed ablation studies.\n- The complete interpretation analysis.\n- More complete reference work.", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The introduction of the CMoE module to address the tug-of-war problem at the connector level is novel and well-executed.\n- The paper provides thorough ablation studies to validate the effectiveness of the proposed CMoE module under various configurations.\n- Uni-Med achieves impressive performance with minimal training computational overhead, highlighting its efficiency in handling large-scale multi-modal medical data.", "weaknesses": "- The ablation studies show that certain configurations (e.g., using a high number of projection experts) might lead to overfitting. This aspect could be discussed in more detail, including strategies to mitigate overfitting.\n- While the interpretation analysis only focus on visual features on different tasks, the analysis of visual features on different medical image modalities should be considered.\n- It could be better to add more work on evaluation of medical vision-language models in the section of related work to make sure that the relevant work is fully discussed, such as [1,2].\n\n[1] Yan Q, He X, Yue X, et al. Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQA[J]. arXiv preprint arXiv:2405.20421, 2024. \n\n[2] Xia P, Chen Z, Tian J, et al. CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models[J]. arXiv preprint arXiv:2406.06007, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments!\n\n \n\n**Q1:** The ablation studies show that certain configurations (e.g., using a high number of projection experts) might lead to overfitting. This aspect could be discussed in more detail, including strategies to mitigate overfitting.\n\n \n\n**A1:** Thank you for your suggestion. \n\n- Choosing the optimal parameter configuration in multi-modal multi-task scenarios has always been a concern, which aims to achieve a balance between performance and computational efficiency [1-2]. It is a challenging study as the complexity of the scenario can end up overfitting to simpler modalities or tasks or underfitting complex ones.\n- As shown in Table 2 (c), in the experiment exploring the key parameter of the number of experts, increasing the number of experts still brings performance gains on some datasets, but the **average gain tends to stabilize across all tasks and datasets**.\n- To our knowledge, recent studies [3-5] have also conducted ablation experiments on these key parameters. However, there are different observations in different scenarios (number of modalities/tasks/data volumes). Therefore, we believe that **selecting the optimal parameter settings using the development set is a simple and effective method** to achieve the balance between performance and computational efficiency.\n\nThanks to your feedback, we will provide more detailed discussions in the revised version. We will continue to focus on this in our future research.\n\n \n\n**Q2:** While the interpretation analysis only focuses on visual features on different tasks, the analysis of visual features on different medical image modalities should be considered.\n\n \n\n**A2:** Thank you for your suggestion. We also use t-SNE method to **visualize the distribution of visual features on medical image modalities** and provide the results in **Figure Re.1**.\n\n- Specifically, we first observe the visual feature distribution of different modalities under the same task in **Figure Re.1 (a-c)**. We find that the feature distributions of CT and MRI modalities in the REG task have good discriminability after passing through the frozen visual encoder. After passing through the connector, the improvement in Silhouette score (from 0.3049 to 0.3335) is relatively limited.\n- In addition, we select 100 samples from each of the 8 modalities and observe their visual feature distributions after passing through different visual encoders in **Figure Re.1 (d-f)**. It can still be observed that the majority of modality distributions are ordered and tightly packed.\n\nBased on the above observations, the **distinction of medical image modalities is achieved effectively** through the visual encoder, while task differentiation requires the well-designed connector. We will add the analysis of visual features on different medical image modalities in the revised version.\n\n \n\n**Q3:** It could be better to add more work on evaluation of medical vision-language models in the section of related work to make sure that the relevant work is fully discussed.\n\n \n\n**A3:** Thank you for your suggestion. We attach great importance to your suggestions and will add cutting-edge developments [6-7] on evaluation of medical vision-language models to the related work section in the revised version.\n\n \n\n**References**\n\n[1] Liu Q, Wu X, Zhao X, et al. When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications[C]//Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2024: 1104-1114.\n\n[2] Chen S, Jie Z, Ma L. Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning mllms[J]. arXiv preprint arXiv:2401.16160, 2024.\n\n[3] Chen T, Zhang Z, JAISWAL A K, et al. Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers[C]//The Eleventh International Conference on Learning Representations.\n\n[4] Dou S, Zhou E, Liu Y, et al. Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment[J]. arXiv preprint arXiv:2312.09979, 2023, 4(7).\n\n[5] Gou Y, Liu Z, Chen K, et al. Mixture of cluster-conditional lora experts for vision-language instruction tuning[J]. arXiv preprint arXiv:2312.12379, 2023.\n\n[6] Xia P, Chen Z, Tian J, et al. CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models[J]. arXiv preprint arXiv:2406.06007, 2024.\n\n[7] Yan Q, He X, Yue X, et al. Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQA[J]. arXiv preprint arXiv:2405.20421, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We would like to express our heartfelt gratitude for your invaluable time, expertise, and meticulous attention in reviewing our manuscript. The insightful comments and constructive feedback have immensely enriched the quality and rigor of our work.

 

We appreciate that the reviewers acknowledge the advantages of our work:

- **About module designs.** The introduction of the CMoE module to address the tug-of-war problem at the connector level is **novel and well-executed** (Reviewer wK4x); The explicit task conditioned projection is **novel**, and integrates an older concept of MoEs into a SOTA MLLM framework (Reviewer Snz8);

- **About experiments.** The paper provides **thorough ablation studies** to validate the effectiveness of the proposed CMoE module under various configurations (Reviewer wK4x); **Extensive experiments** demonstrate Uni-Med's effectiveness across multiple tasks and datasets (Reviewer b87p); The authors do an **excellent job sourcing and assembling** a large, multi-task, multi-modal set of medical benchmarks and performing an **extensive set of ablations** (Reviewer Snz8); The paper is **benchmarked broadly** across multiple tasks—fitting the definition of a foundation model (Reviewer Snz8); The proposed framework is **evaluated thoroughly** in ablation studies (Reviewer mvVf).

- **About results and meaning.** Uni-Med achieves **impressive performance with minimal training computational overhead**, highlighting its **efficiency** in handling large-scale multi-modal medical data (Reviewer wK4x); Multi-modal multi-task optimization is a **complex and important question** for large multimodal models, the introduction of the CMoE module, employs a mixture of projection experts to align visual and language embedding spaces, shows **superior performance** on multiple tasks(Reviewer b87p); It is a **technically-solid** work (Reviewer mvVf); **Consistent improvements** over existing open-source medical foundation models are observed (Reviewer mvVf).

- **About analysis.** The paper conducts a **comprehensive interpretation analysis** of the problem from the perspective of gradient optimization and parameter statistics (Reviewer b87p); **The mitigation of the tug-of-war problem is justified from multiple perspectives**, including developed indexes, parameter statistics scores, routing weights, and tSNE feature visualization (Reviewer mvVf).

- **About writing.** The presentation is **clear and easy to follow** (Reviewer mvVf).

 

On the other hand, we actively adopt the suggestions put forward by the reviewers and diligently address all the issues. Allow me to summarize the revisions made in the rebuttal:

- **Exploring visual features distribution on image modalities.** We conduct additional interpretation analysis which focus on visual features on different medical image modalities. We also use the t-SNE method for visualization and get instructive observation results.

- **Adding metrics for the report generation task.** To fully assess the semantic accuracy, RadGraph F1 and RadCliQ are uesd to evaluate the results of different models on the MIMIC-CXR dataset.

- **Emphasizing contribution to the ML community.** Although this is not a unique medical issue, we have observed that the tug-of-war problem is particularly serious due to the diversity of image modalities and tasks in the medical field. We believe that "solving problem in prominent fields" and "exploring generalization in general fields" are equally important contributions to the machine learning community. Through extensive experiments and interpretability analysis from multiple perspectives, Uni-Med has shown its effectiveness in mitigating the tug-of-war problem in medical field, and the main idea is instructive for the general field.

- **Clarifying fair comparison.** None of the ablation experiments has data leakage issues, and the effectiveness of CMoE in any configuration is reliable. For model comparison, we use readily available model checkpoints for testing. We use the official test set split for all datasets, except for Slake-VQA, as we utilize it to build data for other tasks. In this case, there is an unfair comparison between Slake-VQA, but it's unfair to Uni-Med because part of the test data is used for training in other models.

- **Preliminary exploration of generalization in general field.** We fully follow the training strategy of LLaVA-1.5 and report metrics on 9 benchmarks with or without CMoE. The results show that the introduction of CMoE brings significant improvements on all benchmarks.

- **Writing revision and explanation.** Thanks to the reviewers' feedback, in the revised version, we will (1) provide more detailed experimental discussions; (2) provide clearer explanation for the usage details of comparative models; (3) supplement, emphasize and refine some textual expressions; (4) adjust the placement of the limitation section to the main text.

- **Adding references.** We will add cutting-edge developments on evaluation of medical vision-language models to the related work section.

- **Future work.** Addressing the limitations of handling 3D image data, the optimal setting of key parameters such as the number of experts in different scenarios, validation on more data and a wider variety of tasks will be part of our future work.

Meta-review: This paper introduces a novel approach to addressing the tug-of-war problem in multi-modal, multi-task optimization within the medical field. The authors propose a unified medical generalist foundation model, Uni-Med, which leverages a Connector Mixture-of-Experts (CMoE) module to efficiently align visual and language embedding spaces. The model is evaluated across six distinct medical tasks, including question answering, visual question answering, report generation, and image classification, demonstrating competitive or superior performance compared to existing medical multimodal large language models (MLLMs). Extensive experiments, including ablation studies, validate the effectiveness of the proposed CMoE module. The paper also provides a comprehensive interpretation analysis of the tug-of-war problem from the perspective of gradient optimization and parameter statistics.
After the rebuttal, the paper received mixed feedback, including one strong accept, one accept, one weak accept, and one reject. The reviewers generally found the paper technically sound, with a well-executed novel approach to solving a critical issue in medical MLLMs. The CMoE module was particularly praised for its innovative design and its ability to improve performance across multiple tasks. However, there were concerns about the brief explanation of the comparisons and the fairness of the comparison. For fairness, the authors used the official test set split for all datasets, except for Slake-VQA, as the authors utilized it to build data for other tasks. After carefully considering the paper and all comments, the AC believes the detailed explanation issue can be resolved in the revised version, and the effectiveness of the proposed method can be demonstrated by the results on other testing datasets, except for Slake-VQA. Therefore, the AC recommends accepting this paper.
Decision: Accept (poster)