Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing
Authors: Jie Peng
Jenna L. Ballard
Mohan Zhang
Sukwon Yun
Jiayi Xin
Qi Long
Yanyong Zhang
Tianlong Chen
Venue: ICML 2025 poster (2025)
Keywords: Multi-modal
Medical
Primary area: applications->health_medicine
TL;DR: We propose a novel fusion method that adaptively integrates all modalities for medical multi-modal tasks.

Abstract: Medical multi-modal learning requires an effective fusion capability of various heterogeneous modalities.
One vital challenge is how to effectively fuse modalities when their data quality varies across different modalities and patients.
For example, in the TCGA benchmark, the performance of the same modality can differ between types of cancer. 
Moreover, data collected at different times, locations, and with varying reagents can introduce inter-modal data quality differences ($i.e.$, $\textbf{Modality Batch Effect}$).
In response, we propose ${\textbf{A}}$daptive ${\textbf{M}}$odality Token Re-Balan${\textbf{C}}$ing ($\texttt{AMC}$), a novel top-down dynamic multi-modal fusion approach.
The core of $\texttt{AMC}$ is to quantify the significance of each modality (Top) and then fuse them according to the modality importance (Down).
Specifically, we access the quality of each input modality and then replace uninformative tokens with inter-modal tokens, accordingly.
The more important a modality is, the more informative tokens are retained from that modality.
The self-attention will further integrate these mixed tokens to fuse multi-modal knowledge.
Comprehensive experiments on both medical and general multi-modal datasets demonstrate the effectiveness and generalizability of $\texttt{AMC}$.

Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing
Jie Peng" Jenna L. Ballard 22
Mohan Zhang?
Sukwon Yun Jiayi Xin
Qi Long²
Yanyong Zhang'
Tianlong Chen
Abstract
Pathology Image
Pathologic Diagnosis Text
RNA-seq Expression
Medical multi-modal learning requires an effec- tive fusion capability of various heterogeneous modalities. One vital challenge is how to ef- fectively fuse modalities when their data qual- ity varies across different modalities and patients. For example, in the TCGA benchmark, the per- formance of the tween types of cancer. Moreover, data collected at different times, locations, and with varying reagents can introduce inter-modal data quality differences (i.c., Modality Batch Effect). In response, we propose Adaptive Modality Token Re-BalanCing (AMC), novel top-down dynamic multi-modal fusion approach. The core of AMC is to quantify the significance of each modality (Top) and then fuse them according to the modal- ity importance (Down). Specifically, we assess the quality of each input modality and then re- place uninformative tokens with inter-modal to- kens accordingly. The more important a modal- ity is, the more informative tokens are retained from that modality. The self-attention will fur- ther integrate these mixed tokens to fuse multi- modal knowledge. Comprehensive experiments on both medical and general multi-modal datasets demonstrate the effectiveness and generalizability of AMC. Code is available at https: / /gi thub. con/ com/ PengJieb/ amc.
Folding
Ju Vilaaeee See look
Deveriphhee
Reyunoel In inseak
Good
No Cracking
B5ock SUatiAn Hig Detailed
c0. Thak & &Ilformative
No Ari Rubbie
Eran Stain
Dist-ibatoon
Gcar
Littee Details
Atmuntmal Roprestie Rastarice
Tieme Cracking
modality differ be-
Orear Deseripptoon
Tisnoe Folding
Artifect
pathology
C-tuddes
UCBC2DAD LBC BRGA B104
0100 LSA LGG BRCK 82A5
UCAC1AAD 6200 BRCA BACA
Type
Type
Type
Figure 1. The top part of the figure illustrates how the quality of different modalities can vary dynamically. The bottom part shows how variations within the same modality across different cancer types can directly impact final task performance. The red box indicates the best performance across these three modalities.
netic) (Han et al., 2024; Yun al., 2024a; Mobadersany et al., 2018; Yan et al., 2021; Yap et al., 2018; Bochm et al.,
2022a; Vanguri et al., 2022; Zhou et al., 2023; Lipkova
et al., 2022; Chen et al., 2022a; Steyaert et al., 2023; Boehm et al., 2022b; Im et al., 2023; Chen et al., 2024b). These data provide a comprehensive view patient health, en- abling personalized treatments and enhancing diagnosis and prediction across medical applications.
One vital challenge in multi-modal medical learning is ef- fective multi-modal fusion when data quality varies across different modalities and patients (see Figure 1) (Liu et al., 2024a; Chen et al., 2024a; Sparring et al., 2018; Ghalavand et al., 2024). Most previous fusion methods are designed for visual, text, and audio data (Lin et al., 2024b; Sun et al., 2021; Joze et al., 2020; Cao et al., 2023), but they do not address the specific needs of the medical domain. Medi- cal data involves more heterogeneous modalities, such as genomics, neuroimaging, and biofluid biomarkers, which require specialized fusion techniques.
1. Introduction
Multi-mode learning has advanced medical analysis by integrating diverse data sources (e.g., clinical, imaging, ge-
Equal contribution 'University of Science and Technology of China Uniierssity of Pennsylvania UUniversity of North Car- olina at Chapel Hill. Correspondence to: Yanyong Zhang <yany- ongz@ustc www.uuuussutuc..........
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Prior dynamic fusion (Wang et al., 2022b; Sun et al., 2021;

---

Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re- Re-balancing
Joze et al., 2020; Cao al., 2023) methods use mechanisms like self-attention (Yun et al., 2024a; Han et al., 2024), cross- attention (Wu et al., 2023), or gating networks (Rahman et al., 2020) to implicitly decide each modality's s contri- bution and fuse them accordingly. Recently, in the era of multi-modal large language models (MLLMs) (Lin et al., 2024b; Liu et al., 2025; Guo et al., 2025; Liu et al., 2023b; 2024b; Li et al.. 2023a). approaches like LLaVA (Liu et al.,
We demonstrate the effectiveness of AMC through extensive experiments several real-world datasets, including the MIMIC-IV dataset, the Alzheimer's 's Disease Neuroimag- ing Initiative (ADNI) dataset, and a subset of the TCGA benchmark covering five different cancer types. The re- sults confirm the robustness of AMC across diverse medical scenarios. The contributions of this work are as follows:
We introduce novel top-down dynamic fusion mecha- nism that adaptively integrates multi-modal informa- tion, addressing the variation in data quality across different modalities and patients in various medical domains.
2023b; 2024b) project other modalities into the language token space, utilizing self-attention to integrate all modali- ties. This has increased interest in improving self-attention for multi-modal fusion (Wu et al., 2024; Zhao et al., 2023). However, vanilla self-attention can be problematic, as it often assigns diluted attention weights to irrelevant contexts, leading sub-optimal performance (Wang et al., 2022b: Ye et al., 2024). A possible solution is token fusion (Wang et al., 2022b), which replaces uninformative tokens with inter-modal tokens to better integrate information from vari- ous modalities for dynamic fusion. Token fusion has shown effectiveness and efficiency in visual fusion tasks. However, it focuses on alignment-awand fusion, requiring explicit re- lationships between modalities, such as shared pixels or 3D coordinates. This requirement limits its application in the medical domain, where such relationships are difficult to ascertain, like linking a sub-region of a pathology image with specific genes.
AMC features a novel token fusion approach, includ- ing attention- based modality importance calculation, token score assignment, token replacement policy, and a customized transformer block design.
Extensive experiments on real-world datasets, includ- ing ADNI, MIMIC-IV, and TCGA, demonstrate the consistent and robust performance of AMC in handling variations in data quality.
2. Related Work
Multi-modal Learning with Transformers. Recently, Transformer-basee Multimodal learning has achieved sub- stantial progress(Liu et al., 2023a; Li et al., 2023b; Lin et al., 2024a; Dai et al., 2023), due to their unique advantages and scalability in modeling different modalities, such as language, image, audio, video, etc. Inspired by the great success of maassformer-baseed pretraining in the NLP do- main(Touvron et al., 2023; Devlin et al., 2019; Brown et al., 2020; Bai et al., 2023; DeepSeek-Al et al., 2024), series of works(Dosovitsse et al., 2021; Sun et al., 2019; Radford et al., 2021; Li et al., 2022a; Lu et al., 2019; Chen et al., 2020) have demonstrated that if pre-trained on large-scale multi-modal datasets, Transformer-based models outper- form other domain-speciff models in a wide range of multi- modal tasks, and more importantly, achieve remarkable zero-shot generalization ability. For example, (Liu et al., 2024c) proposes a tight fusion solution to fuse language and vision modalities, and develops open-set object detector that can detect arbitrary objects with language inputs such as category names or referring expressions.
In response, we start with the perspective that modalities contribute unequally. Specifically, we propose a top-down dynamic multi-modal fusion method called the Adaptive Modality Token Re- Re-Balancing method (AMC), which dynam- ically balances modality contributions via self-attention. Un- like previous methods that learn modality contributions im- plicitly, we emphasize that explicitly recognizing unequal contributions of modalities could be more beneficial. Our AMC method works in two main steps. First (Top), it iden- tifies the importance of each modality. We calculate this using attention distribution. This step helps determine how many tokens from each modality should be replaced with to- kens from other modalities. This explicit calculation makes each modality's 's contribution interpretable, which is cru- cial in the A14Medical field (Duan et al., 2024; Wang et al., 2024). Second (Down), we replace tokens with lower scores with tokens from other modalities, based on the modality's 's importance. The token score, also derived from attention distribution, guides this token replacement process. This approach ensures that we prune uninformative tokens and utilize informative tokens from other modalities for fusion. In our model design, we use a single transformer branch to process all modalities. We introduce specific modules within the transformer block to enhance parameter efficiency and fusion quality, supporting the AMC. These include an improved self-attention design, a sparse mixture-of-expert (SMoE) module, and a two-level contrastive learning loss.
Dynamic Multi-modal Fusion. In multi-modal learning, the importance of modalities can vary based on the task and input data quality. Thus, dynamic fusion methods that adapt to these changes are crucial (Zhang et al., 2024; Guan et al., 2019; Han et al., 2020; Tian et al., 2020; Chen et al., 2022b; Sun et al., 2021; Hazarika et al., 201 8; Joze et al., 2020; Cao et al., 2023; Yun et al., 2024a). TensorFusion
(Zadeh et al., 2017) integrates all modalities by using a tensor fusion network to handle both intra- modality and inter-modality dynamics. FlexMoE (Yun et al., 2024a) and
2

---

Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing
LiMoE (Mustafa et al., 2022) use self-attention with sparse mixture-of-experts to adaptively integrate multiple modal- ities and reduce interference. FuseMoE (Han et al., 2024) and MAGGate (Rahman et al., 2020) apply gating networks for dynamic fusion. MUSE (Wu et al., 2023) uses graph net- works and contrastive learning to fuse modalities effectively for predictions. Token fusion (Wang et al., 2022b) mini- mizes computation and prevents attention from focusing on irrelevant contexts by using inter-modal tokens. Despite these advancements, most methods focus on traditional sce- narios like visual detection, semantic segmentation, and audio-visual tasks. Evaluating each modality's 's contribution and effectively fusing heterogeneous modalities is still a challenge, especially in medical multi-modal tasks.
map one modality in into another modality m'' for replace- ment. In many medical tasks, such explicit relationships difficult to establish. For example, the relationship between pathology images and genetic profiles is unclear, and there is no exact mapping between them. (2) The setting of the pa- rameter requires additional effort. A reasonable value for varies across tasks. For instance, tasks in the original pa- per set 0 = 0.01. However, in our medical tasks, no tokens would be replaced with 0 0.01. (3) The values in a (m) are randomly predefined. Consequently, if am (m)))m))) the n-th token will never be replaced from m to m. It as- sumes that all modalities can capture the right information about the same objects, neglecting the heterogeneity and varying abilities of different modalities.
O,
3. Preliminaries and Notations
4. Top-down Dynamical Fusion: AMC
Token Fusion. Token fusion (Wang et al., 2022b) is origi- nally designed for fusing vision transformers with homoge- neous or heterogeneous modalities. After each transformer layer, it employ addition food-fo MLP layer as scoring layer 5m(+) to dynamically predicts the importance of to- kens for the m-th modality. Given input multi-modal tokens {xm) ) m= and the inter-modality token is well aligned!, the token fusion of n-the token is defined as:
In this section, we introduce our overall network architec- ture (Section 4.1) 1) and the proposed Adaptive Modality To- ken Re-BalanCing (AMC) method. AMC accounts for the varying contributions of cach modality across instances. It comprises two key operations: Modality Importance Calculation (Section 4.2) and Customized Token Fusion (Section 4.3). The Modality Importance Calculation com- putes the importance of each modality using attention dis- tribution. It dynamically identifies contribution differences between modalities. The Customized Token Fusion then replaces these tokens with inter-modal features based on their calculated importance. For each instance, we calculate the modality importance to ensure effective fusion. Finally, we introduce two key network designs that enhance token fusion quality (Section 4.4).
Nm [n]]
Xm [n]] @ [n])20
(1)
Σ am (m) o (xxm (n]) @ I (n))
where M is the number of all modalities, N is the number of tokens of each modality, and is indicator asserting the subscript condition, therefore it outputs mask ten- sor (0, 13. The xmIni [n] denotes the n-th token of m-th modality. The parameter 0 is a small threshold, and the operator C resents the element-wise multiplication. The (m) E {0, 1) and (m)) [n] = 1 indicators the n-th token of the m-th modality can be substituted by the corresponding token of the m'-th modality, otherwise am am [m]] = 0. When the number of modality M 2, the am [n] is set to 1 by default. The h(-) is used to select tokens that correspond to the same things between modalities m and m'. For homogeneous modalities, 11()) is an identity function. For heterogeneous modalities, h(-) is defined as a projection function that uses modality-relevan domain knowledge to map the input token to its equivalent in another modality.
4.1. Overall Network Architecture
Task Head
Modality Specific Encoder
Pathology
im improved Transformer Blocks
Add & Norm
Sparse Mixture-of Exppers
Clinical
Add & Norm
Differential Attention
However, the original token fusion has several drawbacks which limit it applicability: (1) It assumes an explicit map- ping relationship between modalities, which is necessary to
'The n- token x.. [n], x [n] for modality m and m' are represent the same things in the physical world. For instance, xm [n] and x,, [n] denote the same object on corresponding RGB and depth images.
Genomics
M
Figure 2. Network Architecture Overview.
As shown in Figure 2, our overall network architecture com-
3

---

Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing
prises three components in turn: the modality-specffic en- coder, consecutive improved transformer blocks with AMC, and task head. O The modality-specific encoder encodes modality m into a sequence of tokens Xm E RN*D, WDD, where N is the number of tokens, and D is the feature dimen- sion. The structure of the modality-speeific encoder relies on corresponding domain knowledge, For each task, we use Q-Former (Li et al., 2023c) in the modality-specifie encoder to ensure the number of tokens N is the same across input modalities. @ The improved transformer block replaces the self-attention module with the differential attention mod- ule (Ye et al., 2024) and substitutes the MLP layer with a Sparse Mixture-of -f-Experts (SMoE) layer. Subsequently, we concatenate multi-modal input { {xm)n) JM along the batch di- mension, resulting in the transformer input a E MxMxNX D where M is the number of modalities. After passing through the consecutive transformer layers, split back into {xm J.M. Each O. undergoes average pooling, and the m pooled outputs are concatenated and reshaped into a single feature vector. 3 This vector is then fed into the task head, which employs a linear layer to produce the final prediction. To enhance modality fusion, we incorporate AMC between adjacent transformer layers. This approach facilitates the effective integration of information across modalities.
all sequences. Last, we normalize a and interpret as the modality importance E [0, 1]M, The vector emphasizes which modalities have the most consistent maximum val- ues, indicating their relative importance. In summary, the modality importance & E [0, 1] M is obtained as follows:
a [mm] Mean Mean(max Am.
a(m]
8 Σ. a/m]
The notation [m, indicates selecting all elements along the sequence dimension for the m-th modality. The colon "." signifies selection across the entire dimension. The function Mean(-) returns the average of the given vector. The variable s represents the attention weight for each modality. We define the number of uninformative tokens for the m-th modality as Km = floor (s|m) x N). Thus, we prune Km tokens in the m-th modality and impute them using inter- modal tokens (x,n') ) /' This approach avoids setting a threshold for token fusion as in vanilla token fusion.
Improved Transformer Block > Improved Tranformer Block
X1
Xa|1]
XM
x
Xa|2]
XM
K
[r]]
ope (n) 2 (nimd al mm
NNo qim, nl C Bottom (nfm)))
[
The computational increase from multi-modal input pri- marily arises from the modality-specific encoders. During fusion, the token sequence length remains the same as for a single modality input, NN but the batch size increases by a factor of M, the number of modalities. The computational complexity of concatenating all modalities into longer se- quence for the fusion is (MNN)) Using cross-attention for fusion, ensuring attention between every pair of modali- ties also has a complexity of ((²)) A4 (N)} = ((MN)) However, our computational complexity in the attention module is ((M(N)) (N))). The increased computational load from multiple modalities of our method can be handled by the parallel computing power of GPUs. Thus, with sufficient GPU power, the actual complexity in the attention module reduces from (N))) to O(N2).
< Mey,
Pooling
Token Replacement
Sean <
m
Xm
X
[2]
Kn.
[N]]
n/m, 1/nim, p/m, N
dggi 0..
Token Fusion
Figure 3. Customized Token Fusion. RQI, RQ2, and RQ3 cor- respond the three research questions detailed in Section 4.3. We first identify the token score (Equation 3) and modality impor- tance (Equation 2) via the attention distribution. Then we utilize this information to decide which token should be replaced and selected to impute intra-modi tokens (Equation 4). Finally, we use {zm J for the final prediction.
4.3. Customized Token Fusion
After obtaining the number of uninformative tokens for each modality, the next step is to perform token fusion. However, as discussed in Section 3, several challenges hinder the application of token fusion from visual fusion scenarios to broader medical applications. To adapt to medical multi- modal learning, we propose a novel token fusion method, illustrated in Figure 3. In this subsection, introduce our token fusion by addressing three specific research questions.
4.2. Modality Importance Calculation
To obtain the importance of each input modality for sub- sequent customized token fusion. We utilize the attention distribution from the last transformer block, represented by the attention map A = M First, we compute the maximum value across the last dimension for each element, resulting max; (A)..,*) E M This operation helps us capture the significance of each token. Next, we compute the average across the sequence dimension (the last dimen- sion) of max: (Am,n,)) to obtain the statistical information as the modality importance indicator. This yields the vector E RRM, where each element o [m] corresponds to the av- erage of the maximum values for the m-th modality across
RQ1: How to identify uninformative tokens? A1: The attention distribution tells you.
Unlike the original token fusion uses an additional scoring network with token-wise pruning loss to obtain the token score that indicate the informative level of tokens. Here, we use the attention distribution as the token score which avoids additional scoring network and corresponding token-wiss
4

---

Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing
pruning loss. Specifically, the token score is obtained by:
sequently, the token fusion operation of xm [n] is reformu- lated as follows:
n/m, n| = max((aaa)) (k),
(3)
&
Xax [n] Xm [n] @ FufmmiiidBootcoc.... (xfml]
M
(4)
where r/fm, n) denotes the token score for m-th modality, n-th token. This computing is overlapped with modality importance caleulation, so we do not involve additional computation here. According to the token score r/mm,;) :], we select these Bottom-Km tokens as uninformative tokens.
Xex (1) O Bajun.jccBussee. (ofme) @ Fnfon' milzot" my
RQ3: How to use {
m= for task prediction?
A3: Average pooling and concatenate.
RQ2: Which token from other modalities should re- place an uninformative token?
After our transformer backbone, each Xm incorporates in- formation from other modalities information via the self- attention module. Then, the challenge is whether to select one Xm. from fxm JM or use the entire set for task pre- diction. In our AMC, use the entire set {xm JMM for the final prediction. Even with contrastive learning efforts to align modalities, modality-specific tokens can still carry heterogeneous information. The token replacement does not always ensure that the selected token X,, [x]] has the high- est token score among Xm' Consequently, using only one Xm. for task prediction might overlook critical and unique information from another modality m'. Specifically, we apply average pooling to each Xm to obtain X.m E RR... We then concatenate the set {xm J to form a longer vector z E RDM This concatenated vector z is subsequently fed into the task head for prediction.
A2: Contrastive loss narrows the search space, then selecting the token with the highest score.
Due to the heterogencity between medical multi-modal data, the initial search space for the candidate replace token is Fmmm for the uninformative token Xm. [n]. The token in {xm" JM mm contains both redundant and complemen- tary. The token score 17 cannot distinguish whether the token holds redundant or complementary information for m-th modality. Consequently, we introduce a two-level contrastive loss on the modality-specific encoded output. The goal of contrastive learning is to learn representations where positive inputs are closer in feature space than nega- tive inputs. The first-level contrastive learning objective L uses features from the same instance as positive inputs and features from other instances as negative inputs (Li et al., 2022b). The second-level contrastive learning objective Cr uses the n-th token from different modalities of the same in- stance as positive inputs, while tokens from other instances serve as negative inputs (Zhou et al., 2024). The detailed definitions of these two-level contrastive learning objectives can be found in Appendix A.
{Xme LM
4.4. Improved Transformer Block
Although our proposed AMC provides the method to effec- tively remove uninformative tokens and retain critical infor- mation from important modalities for the final prediction, two challenges remain.
D Diluted Attention Distribution. Our method heavily relies on the attention distribution for modality importance calcu- lation and token score assignment. However, the original token fusion can dilute the inner-modal attention weights of the self-attention mechanism (Wang et al., 2022b). This dilution can significantly undermine the final performance, potentially leading to inaccurate assessments of modality importance and token scores in AMC.
The first-level contrastive learning improves the quality of representations. In the second- contrastive learning, the goal is to encourage the n-th token across different modali- ties to capture the same information, eliminating the need for an explicit mapping relationship between modalities. This alignment ensures that tokens in {xm, [r]] m am can com- plement xm [ni]. Consequently, the token replacement search space is narrowed from {xm' +m
m to (xm [n]] M ymm Ad-
To address this problem, we employ the differential atten- tion module (Ye et al., 2024). This approach is inspired by nancelcan headphones and differential amplifiers in electrical engineering. The technique involves partition- ing the query and key vectors into two groups, computing two separate attention maps, and then using the difference between these two maps as the final attention map. This differential operation can help reduce attention weight on irrelevant context. For simplicity, we illustrate this using single-head head attention. Given the input sequence xm, the query, and key are calculated as follow:
ditionally, the alignment in the second level reduces irrele- vant information in {xme [r]] + + mitigating information loss during the token replacement operation.
Note that the attention module is shared across all modali- ties, and input tokens are encouraged align through con- trastive learning objectives. As result, can compare attention values across different modalities, even if they are not processed in the same forward pass (multiple modalities are concatenated in the batch dimension, not the sequence dimension). This approach avoids randomly predefining which modality's ' token can replace the current token. Con-
[Q Qz) = x,, 8 [Ki; K2] = x, WK.
(5)
5

---

Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing
where Wº, W E DDx2 are parameters, and Q1, Q2, K1, K2 E RNND. Then the attention map cal- culation is re-formulated from sof toftmax(
cer types: uterine corpus endometrial carcinoma (UCEC), lung adenocarcinoma (LUAD), brain lower grade glioma (LGG), breast invasive carcinoma (BRCA), and blad- der urothelial carcinoma (BLCA). We use three modali- ties-whole slide images (WSIs), pathology reports, and RNA-seq data-to predict patient survival (time-to-event) separately for each cancer type, WSIs are histopathological images containing spatial and morphological information vital for diagnosis and prognosis. Pathology reports, which are text-based descriptions of tumor samples, are vectorized before inputting to our model. Gene expression profiles mea- sured via bulk RNA-seq provide molecular-leve insights into pathophysiology. We do the patient survival time pre- diction on this dataset and report the c-index metric (higher is better) (Alabdallah et al., 2024).
OKT to:
VD
ViKT Asoftmax( vD
Q2K vD ),
A = tmax
(6)
where the À is a learnable scalar. We use this attention map for our modality importance calculation and customized token fusion. For a detailed description of the differential transformer, including the multi-head extension, can refer to Appendix A.2.
> Modality Conffict. Intuitively, we can use modality- specific transformer layers for our token fusion, which is also the network architecture of the original token fusion paper. When the number of modalities increases, this de- sign is parameter inefficient. However, using one branch of consecutive transformer layers to process all modalities introduces the gradient conflict optimization issue between modalities (Yun et al., 2024a;b). Therefore, we integrate the mixtureoffexpert design replace the original MLP layer in the transformer block to alleviate the gradient con- flict optimization issue. For detailed description of our mixture-of-expert design, please refer to Appendix A.3.
MIMIC-IV is large database of de re-identifiedd patient data from the emergency department or intensive care unit at the Beth Israel Deaconess Medical Center in Boston, MA. Our dataset consists of 9,003 patients with laboratory measure- ments (denoted as Lab), clinical notes (denoted as Note), and billing codes (denoted as Code). Using these modalities, we predict patient mortality within one year of admission, framing it as binary classification problem.
ADN/ includes biofluid biomarker measurements, clinical assessments, genomic data, and neuroimages from patients diagnosed as cognitively normal (CN), with mild cognitive impairment (MCI), or Alzheimer's ' dementia (AD), Biospec- imen data include biomarkers from plasma, serum, cerebral spinal fluid (CSF), and urine samples. Clinical assessments consist of cognitive tests, questionnaires, physical exam measurements, and medical history. Genomic data contain genotype dosages for 135,995 genetic variants. Neuroim- ages are structural MP-RAGE MRIs aligned to an average template and segmented using MUSE. Our dataset consists of 2,380 subjects, and the inference task is diagnosis classi- fication, framed as a three-class problem.
5. Experiment
Setup. To ensure fair comparison with baselines, we use the best hyper-parameter settings from the original pa- pers. If these are not available, we conduct hyper-parameter searches, including learning rate, hidden dimension, and batch size, with ranges of [le- - 3, le - 4, 5e - 5, le - 5]. [32, 64, 128], and [32, 64, 128). respectively. For our pro- posed method, we additionally search for the number of experts and the weights of L,, Ln and the load balancing loss of SMoE, with ranges of [4, 8, 16). [1.0, [L.0, 0.1]. and 0.1], respectively. For the dataset split, we use 70% for training, 15% for validation, and 15% for testing. All experiments were conducted using RTX 3090 GPUs By default, each experiment was run five times with dif- ferent seeds to ensure reproducibility, and the results were averaged. The final hyper-parameter settings for AMC are in Appendix B.I. For more implementation details for each dataset, see Appendix B.6.
Result. TCGA (Table I): AMC attains the highest c-index on three cancer types (UCEC, LUAD, LUAD,BLCA} with {0.068 0.510, 0.009 - 0.182, 0.025 0.128) improvement, ranks the third best performance on LGG cancer, and the sec- ond best performance on the BRCA cancer. Overall, AMC achieves the best average C-index across all cancer types. These results demonstrate that our method consistently out- performs existing multi-modal fusion baselines in survival prediction accuracy. ADNI (Table 2): We achieve improve- ments of (3.64 16,13, 3.32 14.3, 2.23 22.74] in terms of Accuracy, Recall, and F1 metrics. These improvements show that our method is reliable for the majority class and also effectively identifies rare but important cases, making it useful for general applications. However, our method scores lower than MUSE and FlexMOE in Precision, sug- gesting that more attention is needed to reduce false posi-
5.1. Medical Application Evaluation
Dataset. We apply AMC on three medical multi-modal real-world datasets: Medical Information Mart for The Can- cer Genome Atlas (TCGA), Intensive Care (MIMIC)-IV, and Alzheimer's Disease Neuroimaging Initiative (ADNI). We choose 5 common used cancer types in TCGA for the performance report.
TCGA involves data from 2,585 individuals across five can-
6

---

Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing
Table 1. Performance on TCGA dataset.
The bold denotes the best performance and the underline denotes the second-hest performance.
Cancer Type
FlexMOR
PuseMOF
LIMOR
MAGGGat
Malf
TF
MUSE
Preposed
UCBC
0.51020.013
0.353a0.117
0.79540:050
0.61340.223
0.66540.071
0.722:0.051
0.501
0.86320.023
LUAD
0.50210.004
0615200017
0.67440.014
0.62120.031
0.616:0.017
0.66120.026
0,50110.25
0.683:0.030
LGG
0,50440.066
0.567+0.06
0.77146:023
0:79900.019
0.82840.0199
0.696=0.031
0.499+0.22
0.795e0.019
BRCA
0466440066
0,49+0.09
0.73
0.684+0.085
0,725=0.008
0.721=0.018
| 0.501=0.26
0.72740.000
BLCA
0,17810.086
0.535.0.068
0,57840:055
0,17540.068
0.56910.037
0.565L0.075
0.501
0,60310.055
Average
0/192
0.512
0.711
0.638
0
0.674
0.501
0.734
Table 2. Performance on ADNI dataset.
Table 4. Different modality importance calculation methods.
Acc
Recall
Precision
F1
Acc
Recall
Precision
F1
FlexMOE
52.27+2.99
48,9726.41
57.58+2.67
47.4749.56
AMC
68.51=1.59
61.65=0.99
59.4620.81
59.75=1.00
FuseMOR
48.85
49.81=3.51
52.76x2.00
49.65+3.88
AMC w/ STD
67.24+2.00
1..24=1.05
59,53+1.21 1,25
LiMOE
46.55+7.56
41.69=4.10
51.92#6.44
34.80+8.20
AMC w/ Max
70.23=3.65
59,4624.77
55.40=8.35
56,66#6.52
MAGGate
47.68+7.71
46,48=7.71
50.99+4.93
40.79#11.88 1.80
Table 5, Different token score assignment methods.
MulT
44.87+7.85
40.73=6608
51.00++4.34
32.08
Acc
Recall
Precision
F1
TF
19.78+9.08
41.85=4.87
45.22+3.67
35.27=10.20
AMC
68.51#1.59
61.65+0.99
59.46is0.81
59.75=1.00
MUSE
51.51
51.71=1.82
58.72+0.91
52.59+1.65
AMC
65,17=1.88
61 64:5440.80
58.59+0.55
58.03=1.01
Proposed
55.91+2.49
55.03+2.58
55.38+2.55
54.82+2.41
w/o Differential Attention
AMC
66.41#1.21
61.59=1.50 588
$8.89+0.49
58.82=0.74
w/ Original Token Fusion
Table 3. Performance on MIMIC-IV dataset.
5.2. In-depth Analysis
Acc
Recall
Precision
In this subsection, we conduct in-depth analysis experiments to show the insights provided by AMC. Specifically, we eval- uate the effect of different methods for assigning modality importance (Paragraph 1), and token score (Paragraph 2). Then, we provide the token exchange (Paragraph 3) and the modality score (Paragraph 4) analysis.
FlexMOF
74.5740.74
51.76=1.21
54,07+2.01
49.96+2.31
FuseMOE
77.31
50.20=0.22
53.08&14.11
44.3520.61
LiMOE
69.9942.24
56.88=1.33
57.13+1.02
56.6911.04
MAGGate
77.42+0.0
50.00#0.0
38.71
43.64=0.0
MulT
68.66#1.15
58.62=0.57
57.69+0.74
57.96=0.72
TF
77.41=0.03
49.99=0.02
38.71#0.00
43.63+0.01
1. Modality Importance Calculation. To determine modality importance, a key operation is an average (Mean) to capture the statistical information of attention distribution for each modality. Intuitively, the standard deviation (STD) could also represent this statistical information. Addition- ally, we can use Max to consider the maximum token score of the modality as its importance.
MUSE
66.70+0.74
50.02=1.87
33.9340.42
40.38+0.38
Proposed
68.51+1.59
61.65=0.99
59.46+0.81
59.75=1.00
tives. MIMIC-IV (Table 3): We observe improvements of (3.03 11.66,
1.79 19.37) in Recall, Preci- sion, and F1, respectively. These gains demonstrate that our method achieves more reliable and precise predictions for the target class, particularly in scenarios where minimizing false negatives and false positives is critical. The slightly lower overall accuracy compared to baselines suggests that prior methods may be overly reliant on the majority class, potentially reflecting overfitting to class-imbalanced training data rather than generalizing to meaningful patterns.
We compared the performance differences between Mean, STD, and Max. The results Table 12 show that Mean and STD yield similar results. Mean performs better in Accuracy and Fl metrics, while STD is better in Recall and Precision. Overall, Mean is slightly better, with higher Fl metric and lower standard deviation across all four metrics. The use of Max is clearly worse than both Mean and STD, as it results in lower Recall, Precision, and Fl metrics. This indicates that statistical information about token scores better reflects how each modality contributes to the final prediction.
7 Non- Mod-Melicall Task Validation. We apply our method to traditional multi-modal dataset to test the generalizability of the proposed AMC and network design. Due to space constraints, the results are provided in Appendix B.2. These results demonstrate that our method remains effective in non-medin on-medical tasks.
2. Token Score Assignment. Here, we compare our method with the original token fusion method (the "AMC w/ Original Token Fusion" in Table 5) in the token score assign- ment. The original token fusion method uses an individual linear network to predict the token score, while we use
7

---

Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing
the attention map from the previous transformer layer for this purpose. Our approach employs differential attention module to enhance attention map quality and mitigate the issue of over- alloocctan attention to irrelevant contexts. We also consider using a vanilla attention module to obtain the attention map for token score assignment as an additional method (the "AMC w/o Differential Attention" in Table 5).
tokens in the Note modality, as seen in the "Code-+Note" plot. The detailed distribution of token replacements and contributions makes the fusion process more transparent. This transparency can be communicated to stakeholders, such as clinicians, to build trust in automated systems that leverage multi-modal data for decision-making.
The results in Table 5 show that using the attention map from differential attention to calculate the token score is bet- ter than using individual linear layer (AMC outperforms "AMC w/ Original Token Fusion" in Accuracy, Recall, Preci- sion, and F1 metrics). The vanilla attention map faces the over-allocation problem, resulting in poorer performance ("AMC w/ Original Token Fusion" better than "AMC w/o Differential Attention" in all metries). This indicates that the over-allocation issue decreases the quality of using attention maps for token score calculation.
*f lnstoooee
Moudility Saur
Madullly Saure
CR
Mociaity Sooce
Nate
1ab
Oute
Tolkss
Tokoro Soore
Token Score
Figure 5. The modality importance (Top) and token score (Bot- tom) statisties. The red dash line is the average value modality score and token score.
Note
1ab
Code
of Tastanne
4. Modality Importance Visualization. In Figure 5, the top row shows the distribution of modality scores for Note, Lab, and Code. Each modality has a distinct score dis- tribution. This indicates they contribute differently to the model's decisions. For example, the Code modality has more instances with higher scores. This suggests it often provides more valuable information. In contrast, the Lab modality has a broader and flatter distribution. The bottom row presents the distribution of token scores for each modal- ity. Note and Code have narrower distributions around their means compared to Lab. This implies more consistent token quality in these modalities. The Lab modality shows more variation in token informativeness. This variation can affect the overall contribution of the Lab modality. Visualizing modality importance and token scores is useful for building interpretable and efficient models. It is especially important in applications where understanding the information source impacts decision making.
Replaced Token
Replaced Token
Replated Token
Nole
Cado
Late
of Lrveantee
of Eralance
Condicate
Condidate Token
Condidate Token
TabesNote
Sarellah
Noar Cade
Conndiatee Tokes
Condidate Tolenn
Condidate Token
The statistic of token replacement on the MIMIC-IV testing set. The first row shows the distribution of tokens in each modality that have been selected for replacement by tokens from other modalities. The last two rows show the distribution of the number of tokens where a modality was chosen to replace another modality's 's tokens. Since token fusion applied cach modality's tokens, denoted as am, and MIMIC-IV includes three modalitics total, we have two rows for this data. In the format "MI-+M2" M1" indicates the candidate modality, while "M2" represents the modality that will be replaced by modality "MI". The dashed red vertical line indicates the average value.
Figure 4.
decision-making.
5.3. Ablation Study
We conducted an ablation study to examine the impact of the four key modules proposed by AMC on performance. The results in Table 6 indicate that removing Contrastive Loss and Differential Attention leads to lower F1 scores. This highlights their effectiveness in enhancing token fu- sion. The decrease in performance when SMoE is removed underscores its role in managing gradient conflicts between modalities. This results in the lowest Fl and Precision met- rics. Lastly, the drop in performance when Token Fusion is excluded demonstrates its importance in balancing contribu- tions from different modalities.
3. Token Exchange Analysis We visualize the token ex- change statistics in Figure 4. The results show that more tokens from the billing codes modality (referred to as Code) replaced by inter-modal tokens. It shows that the num- ber of tokens replaced varies across modalities, with Code having a higher frequency of token replacement. The last two rows of Figure 4 demonstrate the dynamics of token replacement, where one modality replaces another. For ex- ample, tokens from the Code modality frequently replace
8

---

Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y. Ge, W., Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D.. Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang, P. Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang, Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing
Cancer, 3(6):733-733, June 2022a. ISSN 2662-1347,
Table 6.
The ablation study on the MIMIC-IV dataset.
doi: 1..10088/-43118-22--00333
Token Pusion
Coninaive
Loss
Diftreeiia Ameriion
SMaB
Ace
Recall
Precision
F1
08.1111.99
61 65.0500090
59.4600.11
59.75a1.00
Boehm, K. M., Khosravi, P., Vanguri, R., Gao, J., and Shah, S. Harnessing multimodal data integration to advance precision oncology. Nature Reviews Cancer, 22(2): 14- 126, February 2022b. ISSN 1474-1768. doi: 10.1038/
60.9741667
62.94
59.24a31.23
57.712090
84
63.2440.85
59/00.0.40
38.4660.77
v
68.1771188
61.54a0.80
58.59a11.55
5803a100
5
5
65.2762.37
60/8ge1.12
58:2360:43
57.712079
6. Conclusion
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
We propose AMC, a dynamic fusion method that first iden- tifies the importance of each modality and then fuses them accordingly. Our experiments demonstrate that AMC, along with the corresponding network architecture, can effectively fuse multi-modal information in the medical domain. Addi- tionally, our method provides interpretable insights into how each modality and token contributes to the final prediction.
Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan. T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners, 2020. URL https:// arxiv. org/ abs / 200000055.. 14165.
Impact Statement
Cao, Y., Bin, J., Hamari, J., Blasch, E., and Liu, Z. Multi- modal object detection by channel switching and spatial attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 403-411, 2023.
This paper aims to advance the field of Medical Machine Learning. All datasets used are publicly available. While work may have societal consequences, none need to be specifically highlighted here.
Chen, M., Pu, L., Gan, Y., Wang. X., Kong. L., Guo, M., Yang. H., Li, Z.., and Xiong, Z. The association between variability of risk factors and complications in type diabetes mellitus: A retrospective study. Scientific Re- ports, 14(1):6357, March 2024a. ISSN 2045-2322. doi: 1.10388/41198-022 024-56777-.
References
Alabdallah, A., Ohlsson, M., Pashami, S., and Rögnvaldsson, T. The Concordance Index decomposi- tion: A measure for deeper understanding of survival prediction models. Artificial Intelligence in Medicine, 148: 102711, February 2024. ISSN 0933-3657. doi: 1.10166..rtteedd224410002
Chen, R. J., Lu, M. Y., Williamson, D. K., Chen, T. Y.. Lipkova, J., Noor, 7.,, Shaban, M., Shady, M., Williams, M., Joo, B., and Mahmood, F. Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell, 40(8):865-878.. c6, August 2022a. ISSN 1535-6108, 1878-3686. doi: 10. 10.1016/f..cell (clf)).ceel.20220..000
Chen, W., Yang. J., Sun, Z., Zhang, X., Tao, G., Ding, Y., Gu, J., Bu, J., and Wang, H. DeepASS:: A deep addersarial-regulare
A., Yang. H., Yang, J., Yang, S., Yao, Y., Yu, B., Yuan,
H., Yuan, Z., Zhang, J., Zhang, X., Zhang, Y., Zhang, Z., Zhou, C., Zhou, J., Zhou, X., and Zhu, T. Qwen technical report, 2023. URL htw.hhtt://///et /ttp://wxxxvv.////// abs/2309. 16609.
graph learning method for ASD diagnosis with multimodal data. Translational Psychiatry, 14(1):1-9, September 2024b. ISSN 2158-3188. doi: 10. N0.1038/s41330 0001113------0000 0297222.
Boehm, K. M., Aherne, E. A., Ellenson, L., Nikolovski, I., Alghamdi, M. Vázquez-García, García, I., Zamarin, D., Long Roche, K., Liu, Y., Patel, D., Aukerman, A., Pasha, A., Rose, D., Selenica, P., Causa Andrieu, P. I., Fong, C., Capanu, M., Riis-Filho, J. S., Vanguri, R., Veeraragha- van, H.. Gangai, N., Sosa, R., Leung S., McPherson, A., Gao, J., Lakhman, Y., and Shah, S. P. Multimodal data integration using machine learning improves risk stratification of high-grade serous ovarian cancer. Nature
Chen, Y.-C., Li, L., Yu, L., Kholy. A. E., Ahmed, F., Gan, Z.., Cheng, Y., and Liu, J. Uniter: Universal image-text representation learning, 2020. URL https: //arxiv. beg/abs/110 11740.
Chen, Y.-T., Shi, J., Ye, Z., Mertz, C., Ramanan, D., and Kong. S. Multimodal object detection via probabilistic en- sembling. In European Conference on Computer Vision, pp. 139-158. Springer, 2022b.

---

Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing
Dai, W. Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. Instructblip: Towards -eneral-purpose vision-language models with instrue- tion tuning, 2023, URL wwtttttss//aaiiiii org/ abs/ 2305. 06500.
Perceiving Any Aspect Ratio and High-Resolution Im- ages. In Leonardis, A., Ricci, E.,, Roth, S., Russakovsky, O., Sattler, T., and Varol, G. (eds.), Computer Vision ECCV 2024, pp. 390-406, Cham, 2025. Springer Nature Switzerland. ISBN 978-9-031--7310--8 doi:
10. 10.1007998833-33--73000
Ae, ;, Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., Gao, H., Gao, K., Gao, W., Ge, R., Guan, K., Guo, D., Guo, J., Hao, G., Hao, Z., He, Y., Hu, W., Huang, P., Li, R., Li, G., Li, J., Li, Y., Li, Y. K., Liang, W., Lin, F., Liu, A. X.X. Liu, B., Liu, W., Liu, X., Liu, X., Liu, Y., Lu, H., Lu, S., Luo, F., Ma, S., Nie, X., Pei, T., Piao, Y. Qiu, J. Qu, H., Ren, T., Ren, Z., Ruan, C., Sha, Z., Shao, Z., Song, J., Su, X., Sun, J., Sun, Y., Tang, M., Wang, B., Wang. P., Wang. S., Wang. Y., Wang, Y., Wu, T., Wu, Y., Xie, X., Xie, Z., Xie, Z., Xiong, Y., Xu, H., Xu, R. X., Xu, Y., Yang. D., You, Y., Yu, S., Yu, X., Zhang, B., Zhang. H., Zhang, L., Zhang, L., Zhang, M.. Zhang, M., Zhang, W., Zhang, Y., Zhao, C., Zhao, Y., Zhou, S., Zhou, S., Zhu, Q. and Zou, Y. Deepseek Ilm: Scaling open- source language models with longtermism, 2024. URL attps://arxxe org/ abs/ / 220/24111..0 02954.
Han, X., Nguyen, H., Harris, C., Ho, N., and Saria, S. Fuse- MoE: Mixture-of-Expers Transformers for Fleximodal Fusion, May 2024.
Han, Z., Zhang,
C., Fu, H., and Zhou, J. T. Trusted multi- view classification. In International Conference on Learn- ing Representations, 2020.
Hazarika, D., Gorantla, S., Poria, S., and Zimmermann, R. Self-attentive feature-level fusion for multimodal emotion detection. In 2018 IEEE Conference on multimedia in- formation processing and retrieval (MIPR), pp. 201. IEEE, 2018.
Im, J.EE., Yoon, S.-A., Shin, Y. M., and Park, S. Real- Time Prediction for Neonatal Endotracheal Intubation Using Multimodal Transformer Network. IEEE Journal of Biomedical and Health Informatics, 27(6):2625-2632 doi: 10.1109/BBH1.023..
Devlin, J. Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding, 2019. URL https://aras org/ hts/abs/188000 04805.
June 2023. ISSN 2168-2208 3267521.
Joze, H. R. V., Shaban, A., Iuzzolino, M. L.,, and Koishida, K. Mmtm: Multimodal transfer module for enn fusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13289-13299, 2020.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M.. Heigold, G., Gelly. S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https: // //arxiv. org/ abss/////111 20100000011111 11929.
Leiva, L. A., Hota, A., and Oulasvirta, A. Enrico: A Dataset for Topie Modeling of Mobile UI Designs. In 22nd In- ternational Conference on Eluman-Compute Interaction with Mobile Devices and Services, MobileHCI '20, pp. 1-4, New York, NY. USA, February 2021. Association for Computing Machinery. ISBN 978-1-4503-8052222 doi: 10.1145534032444411077
Duan, J., Xiong. J., Li, Y., and Ding, W. Deep learning based multimodal biomedical data fusion: An overview and comparative review. Information Fusion, 112:102536, December 2024. ISSN 1566-2535. doi: 10.1016/j.inffit 2024.102536.
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., and Gao, J. LLaVA-Med: Training a Large Language-and-Va Vision Assistant for Biomedicine in One Day. Advances in Neural Infor- mation Processing Systems, 36:28541-28564, December 2023a.
Ghalavand, H., Shirshahi, S., Rahimi, A., Zarrinabadi, Z., and Amani, F. Common data quality elements for health information systems: A systematic review. BMC Medical Informatics and Decision Making, 24(1):243, September 2024. ISSN 1472-6947.
doi: 10.1186/
Li, J., Li, D.,, Xiong, C., and Hoi, S. Blip: Bootstrapping Eaguage-image pre-training for unified vision-language understanding and generation, 2022a. URL https:/ arxiv. arxiv.org/ otgg/bs/////ooo...///.be. abs 22011 002220122000 12086..
1-2911--02--2644477
Guan, D., Cao, Y., Yang, J., Cao. Y., and Yang, M. Y. Fusion of multispectral data through illumination-wware deep neural networks for pedestrian detection. Information Fusion, 50: 1 148-157, 2019.
Li, J., Li, D., Xiong, C., and Hoi, S. BLIP: Bootstrap- ping Language- Image Pre-training for Unified Vision- Language Understanding and Generation. In Proceed- ings of the 39th International Conference on Machine Learning, pp. 12888-12900. PMLR, June 2022b.
Guo, Z., Xu, R. Yao, Y., Cui, J., Ni, Z., Ge, C., Chua, T.S.. Liu, Z., and Huang, G. LLaVA-UHD; An LMM
10

---

Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing
Ji, Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023b. URL https:// arxiv. org/ abs 2oos/2301.10 12597.
Liu, S., Cheng, H., Liu, H., Zhang, H., Li, F., Ren, T., Zou,
X., Yang, J., Su, H., Zhu, J., Zhang, I.., Gao, J., and
Li, C. LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents. In Leonardis, A., Ricci, E., Roth, s., Russakovsky, O., Sattler, T., and Varol, G. (eds.), Computer Vision - ECCV 2024, pp. 126-142, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031- 72970-6. doi: 10.
Li, J., Li, D., Savarese, S., and Hoi, S. BLIP-2: Bootstrap- ping Language- Language-Image Pre-training with Frozen Image Encoders and Large Language Models. In Proceedings of the 40th International Conference on Machine Learning, Pp. 19730-19742 PMLR, July 2023c.
Lu, J., Batra, D., Parikh, D., and Lee, S. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision- and-language tasks, 2019. URL setps://arxxr org/ (0c//bss/9900 62.
Liang, P. P.. Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J.. Chen, L.. Wu, P., Lee, M. A., Zhu, Y.. Salakhutdi- nov, R., and Morency, L.-P. MultiBench: Multiscale Benchmarks for Multimodal Representation Learning. arXiv:2107 07.005022022 [cs], November 2021.
Mobadersany, P., Yousefi, S., Amgad, M., Gutman, D. A., Barnholtz-Sloan, J. S., Velázquez Vega, J. E., Brat, D. J., and Cooper, L. A. D. Predicting cancer outcomes from histology and genomics using convolutional networks. Proceedings of the National Academy of Sciences, 115 (33:E2970--22971 March 2018. doi: 10.1073/pnas. 1717139115.
Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., and Yuan, L. Video-Ilava: Learning united visual representation by alignment before projection, 2024a. URL https:
/////aaaaa /arxiiiv. org/ abs 22111. 10122.
Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., and Yuan, L. Video-LLaVA: Learning United Visual Representation by Alignment Before Projection, October 2024b.
Mustafa, B., Riquelme, C., Puigcerver, J., Jenatton, R., and Houlsby, N. Multimodal Contrastive Learning with LIMoE: The Language-Image Mixture of Experts, June 2022.
Lipkova, J., Chen, R. J., Chen, B., Lu, M. Y., Barbieri, M.,
Shao, D., Vaidya, A. J., Chen, C., Zhuang, L., Williamson, D. F. K., Shaban, M., Chen, T. Y., and Mahmood, F.
Paul Pu Liang, Lyu, Y., Fan, X., Mo, S., Yogatama, D., HIGHMMT:
Morency, L.-P., and Salakhutdinov, Towards Modality and Task Generalization for High- Modality Representation Learning. arXiv:2203.01311 [cs], March 2022.
Artificial intelligence for multimodal data integration in oncology. Cancer Cell, 10910):1095-1110 October 2022. ISSN 1535-6108, 1878-3686. doi: 10.1016fj.ccell.022 09.012.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger. G., and Sutskever, I. Learning transferable visual models from natural language supervision, 2021. URL https: http//xxvvvv///////./ .rg abs 221./2113.0000
Liu, B., Zhou, H., Tan, L., Siu, K. T. H., and Guan, X.-Y. Exploring treatment options in cancer: Tumor treatment strategies. Signal Transduction and Targeted Therapy. 9 (1):1-44, July 2024a. ISSN 2059-3635. doi: 10.1038/ $41392-024-01856.
Rahman, W., Hasan, M. K., Lee, S., Zadeh, A., Mao, C., Morency, L.-P., and Hoque, E. Integrating Multimodal Information in Large Pretrained Transformers. Proceed- ings of the conference. Association for Computational Linguistics. Meeting, 2020:2359-2369, July 2020. ISSN 0736-587X. doi: 10. 1ttp:///w..0000...mmmmmmmm..2m.
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tuning, 2023a. URL https://arxiv. wwwp:/iiioorrooooooooooo abs/ 2304.088 08485.
Liu, H,, Li, C., Wu, Q., and Lee, Y.J. Visual Instruction Tun- ing. Advances in Neural Information Processing Systems, 36:34892-34916 December 2023b.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer In International Conference on Learning Representations, July 2022.
Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved Basclines with Visual Instruction Tuning. In Proceedings of the IEEE/VVF Conference on Computer Vision and Putlern Recognition, pp. 26296-26306, 2024b.
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q. Li. C., Yang, J., Su, H., Zhu, J., and Zhang, L. Grounding dino: Marrying dino with grounded pre- training for open-set object detection, 2024c. URL https: //tp://aavvvv////////ae org/ abs///b///// 2303 05499.
Sparring, V., Granström, E.,, Andreen Sachs, M., Brom- mels, M., and Nyström, M. E. One size fits none a qualitative study investigating nine national quality reg- istries' conditions for use in quality improvement, re- search and interaction with patients. BMC Health Ser-

---

Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing
vices Research, 18(1):802, October 2018. ISSN 1472- 6963. doi: 10.1 10.1186/s11 1etiss1111350 2913-018-3621-4
Wang, Y., Chen, X., Cao. L.., Huang, W., Sun, F., and Wang. Y. Multimodal Token Fusion for Vision Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12186-12195 2022b.
Steyaent, S., Pizurica, M., Nagaraj, D., Khandelwal, P., Hernandez-Boussarrd, T., Gentles, A. J., and Gevaert, O. Multimodal data fusion for cancer biomarker dis- covery with deep learning. Nature Machine Intelli- gence, 5(4):351-362, April 2023. ISSN 2522-5839. doi: 10.1038842255--023--03333
Wu, Q. Ye, W. Zhou, Y., Sun, X., and Ji, R. Not All Attention is Needed: Parameter and Computation Effi- cient Transfer Learning for Multi-modal Large Language Models, June 2024.
Sun, C., Myers, A., Vondrick, C., Murphy, K., and Schmid, C. Videobert: A joint model for video and language representation learning, 2019. URL https:/: //rxivv. brg/abs/1900 1904. 01766.
Wu, Z., Dadu, A.. Tustison, N., Avants, B., Nalls, M., Sun, J., and Faghri, F. Multimodal Patient Representation Learning with Missing Modalities and Labels. In The Twelfth International Conference on Learning Represen- tations, October 2023.
Sun, L.,, Liu, B., Tao, J., and Lian, Z. Multimodal cross-and self-attention network for speech emotion recognition. In ICASSP 2021-202 IEEF International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4275 4279. IEEE, 2021.
Yan, R., Zhang, F., Rao, X., Lv. Z., Li, Zhang, L., Liang, S., Li, Y., Ren, F., Zheng, C., and Liang, J. Richer fusion network for breast cancer classification based on multimodal data. BMC Medical Informatics and Decision Making, 21(1):134, April 2021. ISSN 1472-6947. doi: N0.1186/s1291 020-0134066.
Tian, J., Cheung, W., Glaser, N., Liu, Y-C.,, and Kira, Z. Uno: Uncertainty-aware noisy-or multimodal fusion for unanticipated input degradation. In 2020 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 5716-5723, IEEE, 2020.
Yap. J., Yolland, W., and Tschandl, P. Multimodal skin lesion classificatton using deep learning. Experimental Dermatology, 27(11):121-1126; 2018. ISSN 1600-0625. doi: 10. 137777.
Touvron, H., Lavril. T., Izacard, G.. Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation lan- guage models, 2023. URL https: / /ttwxaaxivvv../// org/ 13971.
Ye, T., Dong, L., Xia, Y., Sun, Y., Zhu, Y., Huang, G., and Wei, F. Differential Transformer, October 2024.
Yun, S., Choi, I., Peng, J., Wu, Y., Bao, J., Zhang, Q.,
Xin, J., Long, Q.. and Chen, T. Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture- of offExxerts, October 2024a.
Vanguri. R. S., Luo, J., Aukerman, A. T., Egger. J. V., Fong, J., Horvat N., Pagano, A., Araujo-Filho, J. d. A. B., Geneslaw, L., Rizvi, H., Sosa, R., Boehm, K. M., Yang, S.-R., Bodd, F. M., Ventura, K., Hollmann, T. J., Gins- berg. M. S., Gao, J., Vanguri, R., Hellmann, M. D., Sauter, L.,, and Shah, S. P. Multimodal integration of radiology, pathology and genomics for prediction of response to PD-(L)1 blockade in patients with non-small cell lung cancer. Nature Cancer, (110))1511-1164 October 2022. ISSN 2662-1347. doi: 10. (0.1138//43018-022-2000
Yun, S., Peng, J., Lee, N., Zhang. Y., Park, C., Liu, Z., and Chen, T. scMoE: Single-cell Multi-Modal Multi- Task Learning via Sparse Mixture-of -f--xxerts, November 2024b.
Zadeh, A., Chen, M., Poria, S., Cambria, E.,, and Morency, L.-P. Tensor Fusion Network for Multimodal Sentiment Analysis, July 2017.
Wang, G., Lu, Y., Cui, L., Lv, T., Florencio, D., and Zhang, C. A Simple yet Effective Learnable Positional Encoding Method for Improving Document Transformer Model. In He, Y., Ji, H., Li, S., Liu, Y., and Chang, C.-H. (eds.), Findings of the Association for Computational Linguis- tics: AACL-IJCNLP 2022, pp. 453-463, Online only, November 2022a. Association for Computational Lin- 12t5533//1/2222.mmmnn.. a2cl.42.
Zhang, B. and Sennrich, R. Root Mean Square Layer Nor- malization. In Advances in Neural Information Process- ing Systems, volume 32, Curran Associates, Inc., 2019.
Zhang, Q. Wei, Y., Han, Z., Fu, H., Peng, X., Deng, C., Hu, Q., Xu, C., Wen, J., Hu, D., et al. Multimodal fusion on low-quality data: A comprehensive survey, arXiv preprint arXiv:2404: 18947, 2024.
guistics. doi:
Wang, L., Wang, H., Yang, H., Mao, J., Yang, Z., Shen, J., and Li, X. Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks, October 2024.
Zhao, B., Tu, H., Wei, C., Mei, J., and Xie, C. Tuning LayerNorm in Attention: Towards Efficient Multi-Modal L.M Finetuning, December 2023.
12

---

Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re- Re-balancing
Zhou, H.-Y., Yu, Y., Wang, C., Zhang, S., Gao, Y., Pan, J., Shao, J., Lu, G., Zhang, K., and Li, W. A transformer- based ietiseentatio--eerning model with unified process- ing of multimodal input for clinical diagnostics. Nature Biomedical Engineering, 7(6):743-755, June 2023. ISSN 2157-846X. doi: 10. 10.1033/441511-03-000000
Zhou, Q. Xu, H., Li, H., Zhang, H., Zhang, X., Wang, Y., and Gao, K. TekenlLevel Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recog- nition. Proceedings of the AAAI Conference on Artificial Intelligence, 11 March 2024, ISSN 2374-3468, doi:
12866..
13

---

Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing
A. Method Details
A.1. Definition of Two-level Contrastive Learning objectives
Given pairs data from m-th and m'-th modality, the corresponding tokens is ZB = {aim, 1 where a NR We denote the instance-level features as the average pooling of a across the first dimension for each elements, resulting in x RD_ Then define the instance-level pair-wised representation as X' (x x -i
The contrastive training
objective at the instance level with pre-defined temperature T is:
(x M/T //T
(x(
log
log
Ej=1 e (m D/T
2
2j=1
(x (xmmxx "n//TT
where the (,,) denotes the dot product operation. Then contrastive leartning objective at the token level is temperature T is:
clain [n]. [n])/T
e elal af (m))/T
(Z - log
log
(8)
-==1 En (at (n), (x) (x1/T
Σ 1==
(al (wo), a [m))/T
where the [ [n] denotes the n-th token of a Consequently, the C, and Lr defined as:
N N
L = Cr
Σ
inn (3B).
(9)
1=1
A.2. Differential Self-Attention Details
Given the input sequence am, the query, and key is calculated via:
QQ Q2] a. w N [2] a.
a.nw W = =am an
(10)
where Wº, W WV Dx2D are parameters,
Q1 Q2, K1, NND V E NX2D
Then the attention map
calculation is re-formulated from sof softmax(
QK
to:
A softmax QKKT Asoftmax
O2K2
(11)
VD
Then the attention module output is: DAttn AV. The À is defined as:
exp(An AE,)) exp(>m= >Ag)) + Ainit:
(12)
where Agos Agos Aby Akg RRD are learnable vectors, and Ainit E (0, 1) is a constant number. We set Asnit = 0.1 in default.
Multi-Head Extension. When we have h attention heads, the projection parameters is defined as WºO W.K. W., , {0, 1,..., h). The scalar À is shared cross heads within the same layer. The computation pipeline results as follows:
[ Q5] = am Q [KT; K3] = am W.K V am W.V.
Qi(K Asoftmax (5(K))
A sof tmax
VD
VD
head; A.V; head, (1 Amma) LN(head;),
MultiHead am) Concat(head).. head,,
where W0 E DxD is the learnable projection layer, LN(-) uses RMSNorm (Zhang & Sennrich, 2019) for each head, and Concant(-) concatenates the heads together along the channel dimension. We compute the average of {A A to obtain the attention mapp A, which is used to determine modality importance and perform customized token fusion.
14

---

Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing
A.3. The Mixture-of-Expert in the Transformer Block
In our transformer block, we replace the MLP layer with Mixtureoof-Experts (MoE) module, consisting of multiple experts denoted {fi fz......... ... fF), where E' is the number of experts. We employ the sparse activated MoE (SMoE) variant, which includes router R. The router responsible for the routing mechanism, sparsely selecting the top-k experts. For given token the router R. selects the top-k experts based on scores derived from softmax function applied to learnable gating function g()). We implement this gating function using single linear layer. The router then outputs R(x), identifying the indices the selected experts. This process described as follows::
E Σ R(x), fi(x),
y
(13)
Top-K Hopp-K(ep (So (softmax(ggxx) k),
(14)
if is in the top k,
Top-K
(15)
0, otherwise.
where g(x) computes the scores for each expert, and the top-k function selects the experts with the highest scores, To avoid the load imbalance problem SMoE, we use the load and imbalance loss (Shazeer al., 2022).
B. Experiment Details
B.1. Hyper-parameter on Each Dataset
Table 7. The hyper-parameter setup for AMC.
TCGA
ADNI
MIMIC-IV
ENRICO
UCEC
LUAD
LGG
BRCA
BLCA
Leurning rute
le-4
le-3
le-3
le-3
le-3
5e-3
# of Experts
8
8
8
8
8
8
Top-K
2
2
2
2
2
2
2
# of Transformer Layers
2
2
2
2
2
4
Training Epochs
30
100
30
30
100
Warm-up Epochs
5
Hidden dimension
64
64
128
64
64
64
64
128
Batch Size
32
64
64
64
64
64
64
128
# Attention Heuds
8
8
B.2. Non-Medical Multi-modal Evaluation
Dataset. We select Enhanced Rico (ENRICO) dataset (Leiva et al., 2021) evaluates the generalizability of AMC. The ENRICO is the dataset of Android app sereens categorized by their design motifs. It contains 1, 1,460 Android app screens (image modality) and the corresponding view hierarchy (set modality). The instances in this ENRICO can be categorized into 20 design categories.
Result. We compare existing fusion methods tested on this dataset as described in (Liang et al., 2021; Paul Pu Liang et al., 2022). We use the same evaluation setup as these references, which includes searching for paper-parameters like learning rate and hidden dimension. We report the mean and standard deviation from 10 repetitions, ensuring fair comparison with the results from (Liang et al., 2021; Paul Pu Liang et al., 2022). All baseline results are taken from the original papers (Liang et al., 2021; Paul Pu Liang et al., 2022). However, we lack the standard deviation for HighMMT because it was not reported in the original paper. From Table 8, we see that our method, AMC, shows an improvement {16.3 24.5) over multi-modal basclines and {21.9 22.8} over hinle-modaa baselines. This result demonstrates that our method also effective in a non-medical multi-modal benchmark.
15

---

Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing
Table Performance the ENRICO dataset. We refer the evaluation metric previous works (Liang al., 2021; Paul Pu Liang et al.. 2022), and report the 20 classes' prediction accuracy (%).
Unimodal (MuliiRench)
Unimodal (MulnRench)
Late Fasion
TensoPPusion
IRTP
MI-Marix
CCA
RefNer
GradBlend
HighMMT
Proposed
Modality
Image
Set
Image, Set
Acc (20)
47.021.6
16.121.3
50.812.0
46.621.9
47.112.
46.71244
50.12114
44,42.2
510011
$2.6
68.913.7
Table 9. Extend AMC for the modality missing scenario.
Ace
Recall
Precision
F1
AMC
55.91+2.49
55.03+2.58
55.38+2.55
54,82+2.41
AMC w/ missing modality
56.3+2.21
55.7743.32
57.39=2.29
54.81=2.18
B.3. Extra Experiments
Modality Missing Scenario. We extend AMC to solve the missing modality problem. In such cases, we treat the importance score and token score of any missing modality as zero. This approach enables AMC to function effectively even when some data is unavailable. To demonstrate AMC's adaptability to the missing modality scenario, we conducted experiments using the ADNI dataset. This dataset is naturally suited for testing missing modality scenarios, providing a robust environment to validate our approach. The results in the Table 9 indicates that AMC's performance decreases only slightly when dealing with missing modalities. This demonstrates that AMC can effectively handle the missing modality problem through simple extensions. We use the same model but add examples with missing modality problems. The similar performance indicates that AMC still keeps its prediction capability in these additional examples.
Efficiency Experiment. We report the Mean time per iteration during training, and GFLOPs during testing across baselines and AMC. The result in Table 10 shows our method is computationally efficient.
Modality Importance Varies Across Tasks, We evaluate the single modality task performance across MIMIC-IV and ADNI. The results in Table 12 show that the importance of different modalities varies across different tasks.
B.4. Importance of Modality Score
Dataset Level Modality Importance. We assess modality importance at the dataset level using the MIMIC-IV dataset. Each modality is evaluated individually using the same AMC backbone. The results in Table 12 indicates that the 'Code" modality is less important (the lowest F1, Recall, and Precision), aligning with Figure 4 where 'Code' has higher frequency of token replacement.
Instance Level Modality Measurements. We investigate the effectiveness of our modality score in reducing prediction uncertainty. We compare the uncertainty of predictions with their original modality scores against AMC with equal modality scores. The uncertainty of predictions is defined - EE Pi log(p,), where p, is the predicted probability of class i. Our findings show that the modality importance score effectively reduces prediction uncertainty (from 0.1218 increase to 0.1343), demonstrating its utility in identifying significant modalities.
B.5. Interpretability Experiment
Gradient Conflict. We analyze the distribution of cosine distances between training gradients derived from the different modalities in the MIMIC-IV dataset. These gradients are computed from model configured with experts and dense MLP, both following the same configuration as in the AMC and Dense Model setups. The gradients are extracted from the final transformer layer. Higher positive cosine distances indicate reduced gradient conflict, which suggests better alignment between the modalities in terms of learning direction. Our result in Figure 6 shows that the SMoE effectively alleviates the gradient conflict between different modalities.
Modality Importance (Case Study). We propose a case study using images from the TCGA dataset. The image is easier for readers to realize the modality data quality. This study illustrates the variation in modality importance, providing a visual representation of data quality. The results in Figure 7 show that images with lower modality importance scores generally
16

---

Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re- Re-balancing
Table 10.
The computation efficiency comparison.
Metric
FlxxMOE
FuscMOE
LiMOE
MAGGatc
MulT
TF
AMC
Mean Time (s)
12.73
18.68
12.65
11.64
12.85
12.4
11.5
GFLOPs
59.07
59.76
59.41
59.06
60.12
59.39
45.23
Table 11.
The modality importance across different tasks.
MIMIC-IV
Code
Note
Lab
ADNI
Genomic
Image
Biospecimen
Clinical
Acc
67.95
67.58
64.77
Ace
52.94
53.21
50.4
53.50
Precision
55.82
59.18
Precision
53.23
55,91
49.1
53.30
Recall
55.88
56.51
62.50
Recall
53.55
54.70
32.6
52.43
FI
55.60
56.02
58.67
F1
52.30
54.12
39.1
47.32
exhibit poor quality, while images with higher scores indicate better quality.
Token Replacement (Case Study). We provide several case studies about how each modality's s tokens are replaced by other modalities' tokens in the Figure 8.
B.6. Implementation Details
The implementation difference between each dataset is the modality-speciff encoder.
MIMIC-IV. In MIMIC-IV, each modality is represented feature vector. We begin by splitting each feature vector into N sub-vectors and projecting these sub-vectors into the model's hidden dimension. Learnable positional embeddings (Wang et al., 2022a) are then added to transform these projections into input tokens. To enhance feature quality, we first pass these tokens through modality-speciffe vanilla transformer layers. The outputs from these layers are then fed into our improved transformer blocks, where token fusion is performed to integrate the modalities effectively. Finally, these integrated tokens are used to make the final prediction.
ADNI. In ADNI, genomic, biospecimen, and clinical modalities are represented as feature vectors. As with MIMIC-IV, we split each feature vector into N sub-vectors which are projected into the model's hidden dimension. The imaging modality represented 3D tensor which we first apply multiple 3D convolutions with batch normalization. This is followed by average pooling and linear layer projection produce feature vector that is split into N sub-vectors in the model's hidden dimension. The resulting representations are in the same format as the other modalities and undergo the same downstream steps as described for MIMIC-IV. Namely, learnable positional embeddings are added to the vectors to produce input tokens, which are then passed through modality-specifff vanilla transformer layers. This is followed by token fusion via our improved transformer blocks, resulting in integrated tokens that are utilized for prediction.
TCGA. In TCGA, each modality is represented as a feature vector. RNA-seq data is already tabular, whereas the WSIs and text data were preprocessed extract feature vectors. Inputs are first projected into the model's hidden dimension and then split into N sub-vectors. Query tokens are learned by applying single transformer layer with attention between these tokens and the sub-vectors. The resulting outputs are N tokens per modality. The following steps the same as outlined above: learnable positional embeddings are added to these input tokens, these are then input to modality-specific vanilla
Table 12. Dataset-level modality importance on the MIMIC-IV dataset.
Modality
Code
Note
Lab
Acc
67.95
67.58
64.77
Precision
55.44
55.82
59.18
Recall
55.88
56.51
62.50
F1
55.60
56.02
58.67
17

---

Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing
Gradient Cosine Similarity between Code and Lab
Gradient Cosine Similarity between Code and Note
Gradient Cosine Similarity between Note and Lab
120
AMC MoR
AMC w/ MoF
1641
AMC w/ MoR
w/o
w/o
1.0
w/o
100
200
120
100
60
60
40
90
-1000
-0.75
.0:55
0.20
0.35
1.00
-1.00
-055
0.00
0.55
0:75
1,00
-1.00
-0.75
-0.5
.035
0.00
0.55
0.59
100
Figure 6. Gradient Conftict. The gradient conflict between different modalities in the MIMIC-IV dataset. The "AMC w/ MoE" is the proposed AMC method, and "AMC w/o MoE" replaces the MoE to its dense counterpart.
Modality Score
Modality Score
Image: 0.31, RNA-seq: 0.34
Image: 0.34, RNA-seq: 0.32
Text: 0.35
Text: 0.33
No Folding
Tissue Cracking
Little Cracking
No Ari Bubble
Evan Stain Distribution
Tissue Folding
Figure 7. Modality Importance. The case study about the modality data quality identified by our proposed modality importance. The image the left side obtains the lowest modality importance, and the right side image obtains the highest modality importance.
transformer layers, and token fusion is applied to generate integrated tokens for the prediction task.
18

---

Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing
Code
Lab
Note
Instance
Figure 8. Token Replacement. The case study token replacement within the MIMIC-IV IV dataset. Tokens are color-coded by modality: red blocks represent Code tokens, purple blocks indicate tokens, blue blocks signify Note tokens. first row shows token sequence before replacement. Each subsequent row displays the token sequence after replacement for a different instance. We randomly select 100 instances from the testing set for the visualization.
19


Reviews:
[{"review_id": "ni9rqdHfJQ", "reviewer": "Reviewer_joLr", "summary": "This paper proposes Adaptive Modality Token Re-balancing (AMC), a transformer-based fusion method that dynamically adjusts token contributions from different modalities based on estimated modality importance scores derived from attention distributions. AMC selectively replaces less informative tokens from weaker modalities with more informative ones from stronger modalities. The approach also integrates a differential attention mechanism, a Sparse Mixture-of-Experts (SMoE) layer to address inter-modal gradient conflicts, and a two-level contrastive loss for feature alignment. Although each component individually builds upon existing methods, their combined application specifically targets modality imbalance in medical AI. The method demonstrates improved interpretability, generalizability, and predictive accuracy on three multimodal medical datasets, TCGA (cancer survival prediction), ADNI (Alzheimer’s disease classification), and MIMIC-IV (hospital mortality prediction), and one non-medical benchmark (ENRICO), outperforming state-of-the-art multimodal fusion methods such as FlexMOE, LiMOE, MAGGate, MulT, TensorFusion, and MUSE.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and thoughtful feedback. We are encouraged that the reviewer finds our method **novel in how it integrates token-level rebalancing** with **reasonable reproducibility and evaluation**. Below, we address each concern in turn.\n\n### [Cons1] Overstated About the Modality Importance\n\nTo respond to the reviewer's concern regarding the importance of modalities, we propose two experiments to clarify the role of modality scores in identifying the significance of each modality.\n1. **Dataset Level Modality Importance:** We assess modality importance at the dataset level using the MIMIC-IV dataset. Each modality is evaluated individually using the same AMC backbone. The results in the following table indicate that the 'Code' modality is less important (the lowest F1, Recall, and Precision), aligning with Figure 4 where 'Code' have a higher frequency of token replacement.\n\n|Modality|  Code |  Note |  Lab  |\n|--------|:-----:|:--:|:----:|\n| Acc       | 67.95 | 67.58 | 64.77 |\n| Precision | 55.44 | 55.82 | 59.18 |\n| Recall    | 55.88 | 56.51 | 62.50 |\n| F1        | 55.60 | 56.02 | 58.67 |\n\n2. **Instance Level Modality Measurements:** We investigate the effectiveness of our modality score in reducing prediction uncertainty. We compare the uncertainty of predictions ($-\\sum_{i=1}^{N}p_i\\log(p_i)$, where $p_i$ is the predicted probability of class $i$) using AMC with its original modality scores against AMC with equal modality scores. Our findings show that the modality importance score effectively reduces prediction uncertainty, demonstrating its utility in identifying significant modalities.\n\n|                                    | Uncertainty |\n|------------------------------------|:-----------:|\n|                 AMC                |    0.1218   |\n| AMC w/ average modality importance |    0.1343   |\n\nThese experimental results indicate that our modality importance adequately captures modality contribution for the final prediction.\n\n### [Cons2] Clinical Validation of the Interpretableability of AMC\n\nPlease refer to the **Cons1** in our response to the **Reviewer CN7m** for this clarification due to limited space.\n\n### [Cons3] Extend to Missing Modality Scenario\n\nPlease refer to the **Cons3** in our response to the **Reviewer JTwf** for this clarification due to limited space.\n\n### [Cons4] The Measurement of Modality Quality Definition\n\n**We define the modality quality as the performance of each modality with the same modal backbone.** As we show in Figure 1, the bottom part shows the performance variations with the same network backbone but different modalities as the input. **These performance differences of each modality across different cancer types show the challenge of data quality variation.**\n\n### [Cons5] Gradient Conflict between Different Modalities.\nWe analyze the distribution of cosine distances between training gradients derived from the different modalities in the MIMIC-IV dataset. These gradients are computed from a model configured with experts and a dense MLP, both following the same configuration as in the AMC and Dense Model setups. The gradients are extracted from the final transformer layer.\n**Higher positive cosine distances indicate reduced gradient conflict**, which suggests better alignment between the modalities in terms of learning direction.\nOur result in Figure 1 of [Anonymous](https://anonymous.4open.science/r/amc_rebuttal-1247/README.md) shows that the SMoE effectively alleviates the gradient conflict between different modalities.\n\n### [Cons6] Training Efficiency\nPlease refer to the **Cons2** in our response to the **Reviewer CN7m** for this clarification due to limited space.\n\n### [Cons7] Illustrating the customized token fusion workflow of Figure 3.\nThanks for pointing that out. \nIn our token fusion, we first identify the token score (*Equation 3*) and modality importance (*Equation 4*) via the attention distribution. Then we utilize this information to decide which token should be replaced and selected to impute intra-modal tokens (*Equation 4*). Finally, we use $\\{{x}_{m}\\}_{m=1}^{M}$ for the final prediction. In the revision, we will revise the figure captions for better standalone clarity.\n\n### [Cons8] Fail/Underperform Analysis of AMC\nWhen **all modalities possess high quality**, determining which token to replace in AMC's token fusion process becomes challenging. This can lead to potential information loss due to token replacement, which could explain why AMC underperforms compared to MulT and MAGGate in the LGG cancer type of the TCGA dataset.\nDespite this, AMC still achieves a top-3 performance among seven baseline methods in the TCGA LGG dataset. This demonstrates that while AMC may not be the best performer in this specific case, it still maintains competitive performance overall.\n\nWe sincerely thank you for your time and consideration. Please let us know if you have any further questions; we are more than happy to address them comprehensively.", "claims_and_evidence": "The paper makes three primary claims:\n\nFirst, the authors assert that explicitly recognizing modality contributions, as done by AMC, offers advantages over previous implicit methods. However, AMC computes modality importance explicitly using self-attention scores, which themselves are implicitly learned and subject to dataset biases. Since token replacement is guided by these scores without external validation (e.g., human-labeled importance or uncertainty measures), the claim of explicit modality contribution recognition is overstated and requires clarification.\n\nSecond, the authors highlight improved interpretability as a major contribution. However, this interpretability remains limited to computational visualization of modality importance and token reallocation. The paper lacks clinical validation, such as alignment with clinical decisions, biological plausibility, or qualitative assessments by domain experts. Comparisons to clinically validated explainable medical AI methods are also absent, limiting the strength of this claim.\n\nThird, the authors claim the Sparse Mixture-of-Experts (SMoE) layer effectively manages gradient conflicts between modalities. Although numerical results indicate performance drops when removing SMoE, no direct evidence or analysis demonstrates how SMoE resolves gradient conflicts, particularly from a clinical perspective.", "methods_and_evaluation_criteria": "The paper evaluates AMC on diverse medical datasets (TCGA, ADNI, MIMIC-IV), which are widely used benchmarks in multimodal medical research. However, real-world medical datasets often suffer from missing modalities. AMC assumes all modalities are consistently available, limiting its practicality for real-world clinical deployment.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The authors did not conduct a failure case analysis, leaving it unclear under what conditions and why AMC might fail. For example, AMC is not the best-performing model for certain cases, such as the LGG cancer type in TCGA, yet the authors do not investigate or discuss possible reasons. Additionally, since AMC's re-balancing process relies entirely on self-attention distributions, which can be unreliable indicators of feature importance, biased or inconsistent attention scores might cause the token replacement mechanism to amplify errors instead of reducing them.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "The paper focuses exclusively on transformer-based approaches and does not discuss alternatives.", "other_strengths_and_weaknesses": "Strengths: The proposed method is novel in how it integrates token-level rebalancing within a transformer-based model. The reproducibility is reasonable. Experiments are reasonably thorough in terms of dataset selection and metric evaluation.\n\nWeaknesses: The paper lacks discussion of computational costs, specifically how token rebalancing affects training or inference time compared to existing fusion methods. Absence of error analysis, failure-case examination, and handling of missing modalities limits AMC's practical value as a real-world medical AI solution.", "other_comments_or_suggestions": "1. While the paper emphasizes data quality variations, it does not formally define or objectively measure \"quality,\" relying instead on performance differences and attention scores as proxies. An explicit metric or clearer definition would enhance clarity.\n\n2. Figure 3, illustrating the customized token fusion workflow, could be presented more clearly.", "questions_for_authors": "1. How does AMC handle missing modalities? If a modality is unavailable (e.g., missing genetic data in TCGA), does AMC fail, or can it adapt accordingly?\n2. Can the authors provide an analysis or insights into scenarios where AMC underperforms or fails, and explain the reasons behind such outcomes?\n3. How do the authors quantitatively demonstrate that the SMoE layer effectively resolves inter-modal gradient conflicts? Additionally, why was SMoE chosen over alternative gradient stabilization techniques?\n4. The paper discusses efficiency claims qualitatively but lacks quantitative evaluation. Could the authors provide specific metrics regarding AMC’s computational efficiency?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6k9LefkDnA", "reviewer": "Reviewer_jTYn", "summary": "The paper proposes Adaptive Modality Token Re-Balancing (AMC), a dynamic fusion method tailored specifically to handle varying data quality across modalities in medical multi-modal learning. AMC explicitly quantifies the importance of each modality by computing statistical information from attention distributions (mean attention values). Based on these calculated importance scores, AMC adaptively replaces less informative tokens with more valuable inter-modal tokens, guided by a two-level contrastive learning approach and an improved differential attention mechanism, thereby ensuring precise and robust integration of multi-modal medical data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback. We address the key concerns and suggestions below.\n\n### [Cons1] Interpretability Analysis in the Medical Setting\nFor the interpretable aspect, **we are not primarily aiming to propose a method that focuses on giving interpretableability**.\nHowever, AMC can provide modality-level insights.\nTo illustrate this, we present visualizations in Figures 4 and 5. \nThese figures demonstrate our model's ability to identify how each modality differently contributes to the final prediction.\nTo further address reviewer concerns, we conducted a **case study** on the image modality within the TCGA dataset. Please refer to the **Cons1** in our response to the **Reviewer CN7m** for the case study details due to limited space.\n\n### [Cons2] Recent SoTA Fusion Baselines.\nFor our comparisons, we include classical fusion methods such as MAGGate, MulT, and TF. Additionally, we evaluate against recently proposed baselines: **FlexMoE and FuseMoE (both from NeurIPS 2024)**, and **MUSE (from ICLR 2024)**. The FlexMoE and MUSE are designed for medical tasks.\nIf reviewers have specific baselines they recommend for inclusion, we are open to incorporating them into our analysis.\n\n### [Cons3] Theoretical Justification about the Calculation of Modality Importances.\nAs discussed in Section 5.2, Paragraph 1, we explore two perspectives to calculate modality importance: **the statistical perspective** (Mean v.s. STD) and **the value perspective** (Mean v.s. Max) of attention \nIn this paper, we propose a novel fusion mechanism to address the challenge of modality quality variation. Our empirical evidence supports the effectiveness of using Mean for determining modality importance. Although this paper does not include a theoretical analysis, our contributions have significant practical implications. Rigorous proof is valuable, but it is outside the scope of this work.\n\n### [Cons4] Dataset in Table 4&5\nThanks for pointing this, the datasets used in Table 4&5 are the MIMIC-IV. We will revise the table captions for better standalone clarity.\n\n### [Cons5] Citation about Contrastive Learning\nThanks for the reminder. We will include the related citations [1], [2] for instance level and token level contrastive learning respectively in our revision.\n\n[1] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation\n[2] Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognition\n\n### [Cons6] Novelty Clarification\n\nPlease refer to the **Cons3** in our response to the **Reviewer CN7m** for this clarification due to limited space.\n\n### [Cons7] Section 3 is Hard to Follow\nSection 3 introduces token fusion and discusses its drawbacks. The basic idea of token fusion is to prune single-modal transformers and reuse pruned units for multimodal fusion. Each single-modal transformer is pruned individually, and pruned units are replaced by **projected alignment features** from other modalities. \n\nThe drawbacks include: (1) **the requirement for well-aligned modalities with explicit mapping**, (2) **the need to manually set a threshold $\\theta$ for pruning, which requires additional effort to determine a feasible value**, and (3) when the number of modalities exceeds two, tokens are randomly selected from other modalities as substitutes, **assuming all modalities capture the correct information.** We are open to refining this section further if the reviewer provides specific comments.\n\n### [Cons8] Typos\nThanks for the feedback. We will correct these typos in our next revision.\n\n### [Cons9] The Intuitive Justification and Robustness Clarification of Modality Importance Calculation\n**Intuitive Justification:** For a reliable modality, attention values on each token should **vary to capture meaningful information**. At the same time, **the overall attention value should be high** to ensure that useful information is integrated effectively. This suggests that the attention map captures meaningful information across the input data. Both standard deviation (STD) and maximum (Max) can be affected by outliers. In contrast, a higher mean value is more robust.\n\n**Robustness Clarification:** We conducted an ablation experiment on the ADNI dataset to assess robustness. The F1 scores for STD and Max were 53.93 and 52.99, respectively, which are lower than the F1 score of 54.82 achieved by Mean. This demonstrates that robutness of Mean operation for modality importance.\n\n### [Cons10] Missing Modality Scenarios\nPlease refer to the **Cons3** in our response to the **Reviewer JTwf** for this clarification due to limited space.\n\nWe are very thankful for your time and consideration. Please let us know if you have any further questions; we will be happy to address them fully.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "1. The method lacks a thorough analysis of its interpretability in clinical settings, which is critical for the adoption of medical AI systems.\n2. The paper lacks comparisons with more recent state-of-the-art medical multi-modal fusion methods, which would strengthen its claims of superiority.", "theoretical_claims": "**Insufficient Theoretical Justification** \nThe paper lacks rigorous theoretical justification for key design choices, such as the choice of modality importance calculation methods (Mean vs. STD vs. Max), eq 2, the replacement of indicator with eq 4.", "experimental_designs_or_analyses": "Which dataset is used for the experiments in Table 4&5?", "relation_to_broader_scientific_literature": "The main components of AMC, including Token Fusion, Contrastive Loss, Differential Attention, and SMoEs, are adapted from existing works rather than being entirely novel.", "essential_references_not_discussed": "Are the employed contrastive loss (Appendix A eq 7, eq 8) proposed by the authors? I believe citations are necessary.", "other_strengths_and_weaknesses": "Section 3 is hard to follow.", "other_comments_or_suggestions": "Typos: line 124, 161 n-the; line 231 nn-th token", "questions_for_authors": "**1. Originality Clarification.** Token Fusion, Contrastive Loss, Differential Attention, and SMoEs are undoubtedly the most effective components driving AMC's performance gains. However, it appears that all of these components are derived from existing works.  \nI urge the authors to clearly **identify one contribution** that they believe most innovates or extends beyond these established techniques to demonstrate AMC's originality and impact.\n\n\n**2. Choice of Modality Importance Calculation** \nThe authors tested various methods (Mean, STD, Max) to calculate modality importance—could you provide theoretical or intuitive justification for why the attention mean outperforms other statistics? How robust is this choice across different datasets and tasks?\n\n**3. How does AMC perform when one or more modalities are partially or completely missing in a given patient's data?**\n\n\n**4. Which dataset is used for the experiments in Table 5?**", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LVAqK3fCQU", "reviewer": "Reviewer_JTwf", "summary": "The paper proposes Adaptive Modality Token Re-Balancing (AMC) to address the challenge of unequal modality contributions in medical multi-modal learning. AMC dynamically quantifies modality importance and rebalances token contributions by replacing less informative tokens with inter-modal ones. It integrates differential attention, Sparse Mixture-of-Experts (SMoE), and contrastive learning to improve interpretability and efficiency. Extensive experiments on TCGA, MIMIC-IV, and ADNI datasets show that AMC outperforms existing methods, demonstrating strong generalization in multi-modal learning.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and constructive comments. We are grateful for the recognition of our contributions, including **the interesting approach of token replacement, solid experiments, and innovative ideas.** We address the concerns and questions raised as follows.\n\n### [Cons1] Typos\nThe D in line 163 and line 170 should be N. Meanwhile the nn-th should be replaced with n-th. We will modify these typos in our revision.\n\n### [Cons2] Misalignment Issue for the Multimodal Images.\nThis is a good question. Before tokens are fed into our improved transformer blocks, we use Q-Former to encode each modality. In Q-Former, the query comprises modality-specific learnable embeddings, while the key and value derive from the input modality tokens. Q-Former extracts key information from input images, as discussed in [1], and integrates this into learnable query embeddings. Spatial information is implicitly encoded within these embeddings. **Our contrastive loss not only retains the spatial information of modalities, such as images, but also enhances the Q-Former's ability to extract critical information from image tokens.**\n\n[1] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models\n\n### [Cons3] Modality Missing Scenario\nThis paper primarily examines the contribution of different modalities to final predictions. Although handling missing modalities is not our main focus, **it is easy to extend AMC to solve the missing modality problem**. In such cases, **we treat the importance score and token score of any missing modality as zero**. This approach enables AMC to function effectively even when some data is unavailable.\nTo demonstrate AMC's adaptability to the missing modality scenario, we conducted experiments using the ADNI dataset. This dataset is naturally suited for testing missing modality scenarios, providing a robust environment to validate our approach.\nThe results in the following table indicate that **AMC's performance decreases only slightly when dealing with missing modalities**. This demonstrates that AMC can effectively handle the missing modality problem through simple extensions.\nWe use the same model but add examples with missing modality problems. The similar performance indicates that AMC still keep prediction capability in these additional examples.\n\n| |Acc|Recall|Precision|F1|\n|-|:-:|:-:|:-:|:-:|\n| AMC| 55.91±2.49 | 55.03±2.58 | 55.38±2.55 | 54.82±2.41 |\n| AMC w/ missing modality | 56.3±2.21 | 55.77±3.32 | 57.39±2.29 | 54.81±2.18 |\n\n### [Cons4] How does the importance of different modalities vary across different tasks? \nAs shown in Figure 1, the important modality varies even for the same task across different datasets. This variation is mainly due to differences in data collection times and locations, which can affect the quality of medical data. To address the reviewer's concern, we further evaluated the single modality task performance across MIMIC-IV and ADNI (Table 1 in [Anonymous](https://anonymous.4open.science/r/amc_rebuttal-1247/README.md)). This approach helps us assess how the importance of different modalities varies across different tasks.\n\n### [Cons5] Are there certain redundant modalities that are deemed necessary?\nIn this paper, we focus on the effects of different modalities. The necessity of certain redundant modalities is beyond the scope of this work. However, this is an important question that we would like to investigate in the future.\n\n### [Cons6] Intuitive Visualizations\nThanks for your reminder. We provide several case studies about how each modality's tokens are replaced by other modalities’ tokens in the Figure 3 of [Anonymous](https://anonymous.4open.science/r/amc_rebuttal-1247/README.md).\n\n### [Cons7] Comparing and Citing More Works for Novelty Certification.\nAfter comparing our AMC with the mentioned MLLM based methods, we summarize the following similarities and differences.\n**Similarity:**\n- All methods use the attention score to evaluate token importance.\n**Differences:**\n- Our approach uses the attention score to calculate modality importance.\n- We fuse multimodal information through our proposed token fusion framework, whereas existing MLLM methods use self-attention within the LLM backbone.\n- We consider both token-level and modality-level feature importance, unlike previous methods that focus solely on token-level importance.\n\n**Compare with More Baselines:** The input modality, medical tasks, learning objectives, and fusion framework of AMC differ significantly from those of existing methods, making it challenging to include them as baselines in our comparison.\n\n**Novelty Clarification:** Please refer to the **Cons3** in our response to the **Reviewer CN7m** for this clarification due to limited space.\n\nWe are very thankful for your time and consideration. If you have any further questions, please let us know, and we will be happy to address them fully.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "There is no clear theoretical proof in the text, and most of the content is cited from previous research findings.", "experimental_designs_or_analyses": "The experiments are relatively solid.", "relation_to_broader_scientific_literature": "The core idea of this paper is that token replacement is an interesting concept that can be applied to broader multi-modal learning tasks. It cleverly processes data along the batch dimension to avoid the computational cost of quadratic complexity while replacing tokens across different modalities during the process.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper is well-written, with solid experiments and innovative ideas. While it builds on existing work, the improvements lead to strong final results.\n\nSee weaknesses in the comments/questions.", "other_comments_or_suggestions": "1. line 163 : “... in the modality-specific encoder to ensure the number of tokens D is the same across input modalities.”  D -> N ?\n2. line 170: “We define the number of uninformative tokens for the m-th modality as Km = floor(s[m] × D).” Why is D here?\n3. line 231: “level contrastive learning objective LT uses the nn-th token” nn?", "questions_for_authors": "1. My primary concern is that in multimodal images, there often exists a misalignment issue. The proposed feature replacement and contrastive loss in the paper appear to rely on strict spatial positional constraints, which may not align with many real-world scenarios. Has the author conducted any analysis regarding the model's effectiveness in the presence of significant misalignment?\n2. Frameworks based on batch processing generally facilitate testing with missing modalities. Has the author conducted tests on the performance of different modalities in the absence of certain inputs? Furthermore, how does the importance of different modalities vary across different tasks? Are there certain redundant modalities that are deemed necessary?\n3. The paper lacks some intuitive visualizations. For instance, if there are corresponding examples where the token at position (2, 4) of modality A is replaced by modalities B or C, it would be beneficial to clearly demonstrate how the details of modalities B or C are more evident in this instance. Is there any intuitive visualization available to support your claims?\n4. In fact, using attention mechanisms to demonstrate feature importance has become a common practice, whether in existing works on token pruning based on MLLMs, such as FastV and VisionZip, or in specific domains like EDITOR and DeMo, where attention is utilized for feature selection and MoE weighting. The author should consider comparing and citing these works to clarify their core contributions; otherwise, the overall novelty of the paper may appear insufficient.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C6wA9zTJMo", "reviewer": "Reviewer_CN7m", "summary": "This paper proposed AMC, a top-down dynamic multi-modal fusion approach. AMC firstly quantifies the importance of each modality and then fuse the information from different modalities based on the importance. Experimental results show that AMC is effective on several public datasets. This paper addresses a significant challenge in the field of medical multi-modal learning: effectively fusing heterogeneous data from various modalities, especially under conditions where their data quality can vary.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and constructive comments. We are especially grateful for the recognition of our contributions, including **well-supported claims** and **experiments that demonstrate a comprehensive understanding of the capabilities and limitations of the AMC method.** We address the concerns and questions raised as follows.\n\n### [Cons1] Clinical Interpretability Evaluation\nWe should emphasize that interpretableability is **not our main contribution**, and our main contributions lie in three points: \n- **Top-Down Dynamic Fusion Mechanism:** A novel top-down dynamic fusion mechanism to address the variation in data quality challenge\n- **Token Fusion for Heterogeneous Modalities:** A novel token fusion approach that can work on heterogeneous modalities and is not limited to modalities with explicit mapping relationships\n- **Extensive Real-World Validation:** Validate our method by extensive experiments on real-world datasets.\n\n#### *Interpretability and Clinical Validation*\nAlthough our method is **not primarily designed to enhance interpretability**, it does offer modality-level insights. Through our top-down fusion framework, we assess the importance of each modality. This helps in understanding each modality's contribution to the final prediction. Unlike traditional methods such as the Shapley value, which focus on feature-level interpretability, our approach provides insights at the modality level. Therefore, we do not directly compare our method with clinically validated medical AI methods. However, we can include specific methods in future comparisons if the reviewer suggest.\n#### *Case Study*\nTo further address clinical validation concerns, we propose a case study using images from the TCGA dataset. The image is easier for readers to realize the modality data quality. This study illustrates the variation in modality importance, providing a visual representation of data quality. The results in Figure 2 of [Anonymous](https://anonymous.4open.science/r/amc_rebuttal-1247/README.md) show that images with lower modality importance scores generally exhibit poor quality, while images with higher scores indicate better quality.\n\nWe will revise our introduction for clear clarification about our contribution and add the case study to our appendix.\n\n### [Cons2] Empirical Validation for Efficiency.\n\nTo address the reviewer’s concern, we report the **Mean time per iteration during training**, and **GFLOPs** during testing across baselines and AMC. The result shows our method is computationally efficient.\n|     Metric    | FlexMOE | FuseMOE | LiMOE | MAGGate |  MulT |   TF  |    AMC    |\n|:-------------:|:-------:|:-------:|:-----:|:-------:|:-----:|:-----:|:---------:|\n| Mean Time (s) |  12.73  |  18.68  | 12.65 |  11.64  | 12.85 |  12.4 |  **11.5** |\n|     GFLOPs    |  59.07  |  59.76  | 59.41 |  59.06  | 60.12 | 59.39 | **45.23** |\n\n### [Cons3] “situates itself within the existing body of AI and machine learning literature” and has limited novelty.\nWe respectfully disagree. What we really focus on advocating is the novel and practical angle of medical learning with considering multimodal data quality variance for the first time. Motivated by this fresh perspective, our solution integrates the attention mechanism and contrastive learning into the proposed token fusion framework. This token fusion approach is distinctly developed beyond existing methods. Specifically, to address previous drawbacks in token fusion, our framework offers significant differences and improvements over prior models. \n\nIn addition, our novelty is consistently recognized by two reviewers. Our paper was rated as an “innovative idea“ by **Reviewer joLr**: “The proposed method is novel in how it integrates token-level rebalancing within a transformer-based model.”. Then, our novelty is also highly acknowledged by **Reviewer JTwf** as innovative ideas.\n\nFinally, our method brings novelty from the following perspectives:\n- **Novel Motivation:** we are motivated by the important observation “in medical tasks, data quality varies across different modalities and patients.”\n- **Technical Novelty:** Novel token fusion framework that does not require explicit mapping between modalities, and a novel token replacement policy when the number of modalities is larger than 2.\n- **Application Novelty:** We propose a unified learning framework that can address different kinds of medical applications: patient survival time prediction, patient mortality prediction, and diagnosis classification prediction.", "claims_and_evidence": "The paper makes several claims regarding the effectiveness of the Adaptive Modality Token Re-Balancing (AMC) method in enhancing medical multi-modal learning. Most of the claims are supported well. \n\nHowever, 'its interpretability in AI4Medical field' requires supplementation with specific clinical interpretability evaluation. \n\nBesides, claims regarding computational efficiency and scalability are supported by theoretical explanations but would benefit from empirical validation, especially in terms of runtime and resource usage on large-scale datasets.", "methods_and_evaluation_criteria": "The proposed methods and/or evaluation criteria make sense for the problem. The choice of datasets and evaluation metrics provides a comprehensive understanding of the AMC framework’s capabilities and limitations.", "theoretical_claims": "There is no obvious issue.", "experimental_designs_or_analyses": "Experimental designs are valid.", "relation_to_broader_scientific_literature": "Overall, the paper situates itself within the existing body of AI and machine learning literature by extending known concepts with innovative techniques tailored for medical applications, thereby contributing both specialized knowledge and potential broader impact applications.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths: Proposes a top-down dynamic fusion method combining modality-level weighting with token-level replacement. The results shows this method is effective on several public datasets.\nWeaknesses: Limited novelty. While the paper integrates various existing techniques from multi-modal learning, attention mechanisms, and contrastive learning, the method seems to be a combination of different techniques. And there should be more valid proof or evaluation of the claim about interpretability.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Can you explain more about your novelty and give more evaluation or prove the interpretability? I will reconsider my rating if your reply address my concern.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes Adaptive Modality Token Re-balancing (AMC), a transformer-based fusion method that dynamically adjusts token contributions from different modalities based on estimated modality importance scores derived from attention distributions. 

Reviewers generally appreciated the novel idea of token-level rebalancing within a transformer architecture, and recognised the solid experimental results supporting the proposed method. The rebuttal was also helpful in clarifying several technical concerns, particularly around modality misalignment handling and robustness to missing inputs.
Decision: Accept (poster)