Dynamic Modeling of Patients, Modalities and Tasks via Multi-modal Multi-task Mixture of Experts
Authors: Chenwei Wu
Zitao Shuai
Zhengxu Tang
Luning Wang
Liyue Shen
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Learning
Medical Imaging
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A Sample and Task-dynamic Model for Multi-modal Multi-task Medical Image Learning

Abstract: Multi-modal multi-task learning holds significant promise in tackling complex diagnostic tasks and many significant medical imaging problems. It fulfills the needs in real-world diagnosis protocol to leverage information from different data sources and simultaneously perform mutually informative tasks. However, medical imaging domains introduce two key challenges: dynamic modality fusion and modality-task dependence. The quality and amount of task-related information from different modalities could vary significantly across patient samples, due to biological and demographic factors. Traditional fusion methods apply fixed combination strategies that fail to capture this dynamic relationship, potentially underutilizing modalities that carry stronger diagnostic signals for specific patients. Additionally, different clinical tasks may require dynamic feature selection and combination from various modalities, a phenomenon we term “modality-task dependence.” To address these issues, we propose M4oE, a novel Multi-modal Multi-task Mixture of Experts framework for precise Medical diagnosis. M4oE comprises Modality-Specific (MSoE) modules and a Modality-shared Modality-Task MoE (MToE) module. With collaboration from both modules, our model dynamically decomposes and learns distinct and shared information from different modalities and achieves dynamic fusion. MToE provides a joint probability model of modalities and tasks by using experts as a link and encourages experts to learn modality-task dependence via conditional mutual information loss. By doing so, M4oE offers sample and population-level interpretability of modality contributions. We evaluate M4oE on four public multi-modal medical benchmark datasets for solving two important medical diagnostic problems including breast cancer screening and retinal disease diagnosis. Results demonstrate our method's superiority over state-of-the-art methods under different metrics of classification and segmentation tasks like Accuracy, AUROC, AUPRC, and DICE.

Published as a conference paper at ICLR 2025
DYNAMIC MODELING OF PATIENTS, MODALITIES AND TASKS VIA MULTI-MODAL MULTI-TASK MIXTURE OF EXPERTS
Chenwei Wu, Zitao Shuai; Zhengxu Tang; Luning Wang Liyue Shen Department of Electrical and Computer Engineering, University of Michigan, liyues @unich.eui
ABSTRACT
Multi-modal multi-task learning holds significant promise tackling complex diagnostic tasks and many significant medical imaging problems. It fulfills the needs in real-world diagnosis protocol to leverage information from different data sources and simultaneously perform mutually informative tasks. However, med- ical imaging domains introduce two key challenges: dynamic modality fusion and modality-tast dependence. The quality and amount of task-related informa- tion from different modalities could vary significantly across patient samples, due biological and demographic factors. Traditional fusion methods apply fixed combination strategies that fail to capture this dynamic relationship, potentially underutilizing modalities that carry stronger diagnostic signals for specific pa- tients. Additionally, different clinical tasks may require dynamic feature selection and combination from various modalities, phenomenon we term "modality-tane dependence." To address these issues, propose M' novel Multi-modal Multi-task Mixture of Experts framework for precise Medical diagnosis. M' comprises Sodelity-Specific (MSoE) modules and a Modality Task MoE (MToE) module. With collaboration from both modules, our model dynamically decomposes and learns distinct and shared information from differ- ent modalities and achieves dynamic fusion. MToE provides a joint probability model of modalities and tasks by using experts link and encourages experts to learn modality-task dependence via conditional mutual information loss. By doing so, M' offers sample and population- contributions. We evaluate M oE on four public multi-modal medical benchmark datasets for solving two important medical diagnostic problems including breast cancer screening and retinal disease diagnosis. Results demonstrate our method's superiority over state-of-the-art methods under different metrics of classification and segmentation tasks like Accuracy, AUROC, AUPRC, and DICE
Modality-
interpretability of modality
INTRODUCTION
Multi-modal and multi-task learning holds strong potential for tackling many significant and chal- lenging medical imaging problems (Tu et al. 2024 Khara et al. 2024b Cai et al., 2024 He et al.) 2021], as it mimics real-world clinical protocols of disease diagnosis, where clinicians leverage all available data sources and simultaneously perform multiple assessments that are inherently interre- lated and informative of each other (Stahlschmidt al. 2022 Acosta et al. 2022). For example, radiologists combine full-field digital mammography (FFDM) containing high-resolution details and 2D synthesized (2DS) images indicating high-contral tissue structures for breast density pre- diction and cancer risk screening (Brown et al.) (002);; Ophthalmologists integrate color fundus photographs with optical coherence tomography (OCT) that uses near-infrared light and interferom- etry for diabetic retinopathy diagnosis and ren-retinal leakage detection (Korot et al. 2021). All these applications request model that can understand the complementary information from these
'These authors contribute equally this work.
1 'Corresponding Author Our code could he found in M40F Official Implementation

---

Published as a conference paper at ICLR 2025
aok
TakieTada Matchtdaaties
Mati
Eqpet
Prper
Hand Eifeer Took
Ryoen Dapa Loup
Figure 1: Introduction: Traditional multi-modal multi-task modeling vs M'OE in medical imaging
different input modalities for predicting various tasks, drawing a lot of research attention (Yang
2024
Zhao et al.) 2022).
Modeling the complex cross-modal and cross-task relationships of medical imaging modalities is non-trivial challenge. Different modalities can maintain both modality-specifie and modality- shared information: for example, while MRI and PET can both be used for Alzheimer's disease analysis. MRI allows for higher soft-tissue contrast and PET for metabolic activity visuals (Pichler et al., 2008 Zhang et al., 2023). Such information should be inherently dynamic across patient samples due to different patient subgroups and environmental variances. This is analogous to the well-known dynamic fusion of infrared and RGB images in the natural domain. In RGB-Infared fusion, the relative importance of each modality naturally varies with environmental conditions RGB provides better information in daylight while Infrared excels in low-light conditions. In breast cancer screening, parenchymal tissues in younger women often appear similar and confuse with tumor tissues in FFDM images, whereas it can be visualized more elearly by 2DS, due to reduced tissue overlaps (Khara et al. 2024a). For older patients, where calcifications are a key sign of early cancer, FFDMs more reliable in showing these small spots (Aujero et al. 2017] Brown et al.) 2023). These insights from practical medical applications inspire us to raise the first research ques- tion: How can we capture the patient-dyamiic modality-hhard and modality-ppccific information in multimodal data fusion?
Existing multi- motti fusion methods for medical images can be mainly categorized into early fu- sion (Jiang et al., 2019), late fusion (Donnelly et al. [2024] Weisman et al. 2022). and information- theory-based methods (Zhang et al., 2024] Chen et al.] (023b). However, most of these approaches are developed based on the assumption that different modalities contain fixed information distribu- tion across different samples, which rarely holds true in the real world. We conjecture that ignoring such reality leads to the problem of "Modality Competition." " "Huang et al. 2022) where only single modality dominates the joint latent space, suppressing distinct domain-specttn
features that should commonly exist in all modalities. This can result in severely compromised predictions when crucial disease information is only present in an underrepresented modality. We hypothesize that models should be able dynamically integrate effective information from the respective modalities while not losing generalizable information shared in different modalities. That is, dynamically learning sample-dynamil podality-speciffe and modality-shane
information.
Another challenge comes with the various interrelated target tasks. Different diagnostic tasks require task-specific way to select and fuse features from different imaging modalities, depending on the modality-task relevance. For instance, in a comprehensive breast cancer screening utilizing top and side views of both FFDM and 2DS images (4 modalities) (Khara al., (024a). density assessment relies more on FFDM due to its ability to provide higher-resolution details, while the disease de- tection task, particularly for calcified cancers and lesions dense tissue, benefits significantly from 2DS with better visibility scores (Aujero et al., 2017: Destounis et al.) 2020). This implies the sec- ond important research question: How do we model the dynamic modality dependence on different tasks? Current multi-task learning methods either aim to learn a shared set of features that generalize across different tasks (Swamy et al. (2024) or focus on separating task gradients or task modules for single modality (Chen et al. (023c)) ignoring the dynamic nature of task-relevant modality shared information. We hypothesize that models should be able to dynamically leverage the most relevant features from each modality to fuse for each specific task while not compromising the diversity of
2

---

Published as a conference paper at ICLR 2025
shared modti- features generalizable across different tasks. In other words, the goal is to learn the task-specific and task-shared manifolds in the
latent space.
In short, as illustrated in Figure II effective fusion of multi multi-modalities should be both sample- dynamic and -askddepe..ent. Motivated by this, in this work, we propose M a new Multi-modal Multi-task Mixture of Experts (MoE) framework for Medical imaging, dynamically linking modal- ities, samples and tasks via soft mixture of expert (Puigcerver et al., 2023) networks. M' consists of two main modules: Modality-Speciin (MToE). The combination of modality-shared and modality-specifie experts offers a promising so- lution to the modality competition problem, achieving sample dynamic fusion. By leveraging the input-dependent nature of MoE, our proposed model dynamically learns distinct and shared infor- mation from different modalities. Besides, MToE explicitly models the modality-task dependence by learning a joint distribution of modalities, experts, and tasks. By optimizing conditional mutual information loss, we encourage experts to learn diverse patterns of task spe-isseiiiic modality-share information. Our experiments and analysis show that M OE can be well generalized to solve dif- ferent multimodal medical problems in various applications. More importanly, we show that our proposed MToE module can be flexibly combined with existing multi-modal fusion strategies to further improve performance.
MoE (MSoB) and Modality-Share Modality-Task MoE
Our main contributions can be summarized as follows:
1) We propose a novel framework for multi-modal multi-task learning, achieving both sample- dynamic modality fusion and dynamic modality-task dependence modeling. It provides sample- level and dataset-level interpretation of modality contribution to different prediction tasks.
2) Our proposed MToE module provides simple yet effective way to jointly model and interpret modality, expert, and task dependency. We devise a new conditional mutual information loss to encourage experts dynamically learn diverse patterns of task-dependent modality-share infor- mation for different target tasks, respectively.
3) Extensive experiments are conducted on four multi-modal medical benchmark datasets to address two important multi-modal and multi-task medical problems: breast cancer screening and retinal diagnosis. Results show that M' outperforms the baselines from both medical and natural image domains, demonstrating the effectiveness and generalization of our approach.
RELATED WORK
Multi-modal learning in Medical Imaging. Multi-modal learning in medical imaging focuses on improving prediction and mimics the multi-modal nature of clinical expert maeckign-mmking by fus- ing disparate data sources (Kline et al., (022). It has wide applications in fields such as breast cancer screening (two views of FFDM and two views of 2DS) (Yala et al.) (022), retinal diagnoses (Reti- nal Fundus images and OCTs) (Zou et al. 2024b), and head-neck cancer survival analyses (Meng) jet al. 2023) (Computed Tomography(CTT) Positron Emission Tomography (PET), and Magnetic resonance imaging (MRI)). Existing works can be categorized into early fusion, late fusion (usually caantion-based]) and intermediate fusion methods. Late fusion focuses on learning a combined or joint latent space: Donnelly et al. (2024) learns local and global features from multi-modal breast cancer images via a combination of convolutional and transformer networks; [Oyelade et al.) (2024) extracts both low and high-level features using twin-CNNs and uses binary optimization method to eliminate non-discriminant features in the search space. However, late fusion is prone to lose modality-specifi. information (Zhang al. 2024). Early fusion concatenates different modalities into a single input and is shown to outperform late and middle fusion for brain tumor segmentation from MRI/PET/CT data (Marinov et al. 2023). For example, Xue al. (202T) segments liver le- sions by combining predictions from low and high-level feature maps in separate PET/CT decoders. In intermediate fusion, the features, respective to each modality, are joined in intermediate layers before feeding to the final prediction head (Huang et al. (2022). This type of model allows for more convenient modeling of modality and shared information. Chen et al. (2019); Yao et al. (2024b): Wang al. (2023) disentangles the features shared across modalities and those unique within each modality to address challenges like modality inconsistency and modality missingness. Marinov et al. (2023) disentangles unimodal and multi-modal features using modality-speciff de- coders and a multi-modal decoder. Zhang et al. (2024) uses a mutual information disentangled transformer decompose modality information into shared and specific latent space to predict can-
3

---

Published as a conference paper at ICLR 2025
cer risks with mullii-odee histopathology data. Apart from prediction tasks, recent works like Yao (et al. (2024a) have also extended similar fusion techniques for individualized chest X-ray generation via latent diffusion models. However, all these existing methods are either limited to single-task prediction or do not consider the dynamic dependence of modality-shared information on specific tasks, trying to learn shared set of features for different clinical outcomes. Moreover, they all ignore the dynamic changes of modality quality in reality, which diminishes the patient-conditioned advantages of each modality, e.g., the high structural contrast from 2DS for patients with denser breasts and high-resolution details in FFDM for patients with less-dense breasts.
Multi-task learning in Medical Imaging. In medical imaging, multi- task learning (MTL) has wide applications as it aims to enhance the performance on interrelated tasks by leveraging shared infor- mation between them, given limited data availability (Malhotra et al.] 2022). Existing medical MTL. methods typically utilize hard parameter sharing methods, designing task-shared networks to cap- ture common patterns and task task- specific networks to learn individual task features (Meng et al., 2022 Ju et al. 2021). However, these methods require strong correlations among tasks (Zhang & Yang, 2018). which are not necessarily satisfied medical scenarios. Another type of methods consider soft soft-parameter sharing, which utilize regularization loss terms to implicitly seperate parameters for each task (Graham et al.) 2023 Zhou et al.] (02I). (Shao et al., (024) have also considered asso- ciating multi-instance medical image data with multiple tasks using attention-based fusion blocks. However, these methods assume a fixed association between tasks and input data modalities (Zhang & Yang 2018). making them unable to dynamically capture modality-task dependencies, and thus limiting their applicability in multi-modal,, multi-task medical imaging scenarios.
METHOD
3.1 PRELIMINARY
Problem Formulation. We start with the definition of multi-modal multi-task learning. Suppose we are given m image modalities MI,..., Am and p tasks ti...., tpp where the tasks are related but distinct. Each modality contains its own unique information, and the combination of different modalities elicits task-specific shared information. Multi-modal multi-task learning seeks im- prove model performance by integrating the information from the m modalities and leveraging the knowledge across the p tasks. Each task ta can be viewed as a function T3 that maps multi-modal image pairs / to the label for each task Vt, = (T). For each 4g. we have corresponding labeled dataset Dx. each data point of which has m paired images = (I,,..., Im) and a task label Y,, (e.g.. paired 4 images, 2 from FFDM, 2 from 2DS, and label of risk of breast cancer). For simplicity of analysis, we assume that each image pair 1 is associated with a task-specific label Yr, for cach task tx, forming the task dataset Dr as (I, Y., Our approach generalizes to the more worl-world scenario where each D2 consists of different image pairs.
Soft Mixture-of- Mixture-of-Expeer(MMEE) MoE enables dynamic fusion of multi-modal data and creates data- dependent flows during inference (Cao et al.] (202)). Soft MoE (Puigcerver et al. (023) replaces the top-K selection in the sparse Moł (Mustafa
al. 2022) with noftmax-based assignment in the token-expert mapping, allowing for a soft weighted combination of input tokens. Specifically, for a Soft-MoE layer with n experts, given input tokens X E 12' where / the number of tokens and d is their dimension, the Soft MoE assigns a weighted combination of these tokens to each expert using a dispatch matrix D E RR'"" Each expert then projects the input tokens into output tokens, and the final output is obtained as the weighted combination of the output tokens, with the weights provided by a combine matrix C. The matrix D and C' are created based on the input X and a learned matrix & of the MoE layer.
Hypothesis. We propose our hypotheses for medical multi-modal multi-task learning, which will be verified by our analysis experiments in the Sec. 4.3
Dynamic modality-shared and specific information: be decomposed into modality-specific and modality-shared. This interaction should be dynamic, e.g., the high structural contrast from 2DS for patients with denser breasts and high-resolution details in FFDM for patients with less-dense breast. Ignoring this fact will cause multi-modal multi-task models to focus on optimizing parameters of a specific modality while neglecting the others, aka Modality-Comptiiinn
Information between multi-modalities can
4

---

Published as a conference paper at ICLR 2025
(o). Prumework Part I: Project modality imput
Part 2: M4ofE
Purt 3: Feed into task heads
Fp
G -....
4
Task head
Task head
Frihedding X
Emhadding X2
Fibbedding X
MToB
Moll H,
Embbederr
Embedker
Embbecker
MSeF
X550E
f
Modality
Modaliey
Modality
X2
X
X1
X
(b). Seddulliy--Speeifii MoE]MMSEE]
(c). Shared Modality-Tans Task MoR [MTrE]
(d). Basie Soft-MoR Fusion block
Modallty Speetiic Mell: (Mtsa)
Cusdibkaal
Inchedinn Fool
Dalee Sulh Moll Fixiss block
Middally
Exceedling
of Madiltyy
D- PEMUR)
of Maditty
Meddallly an
X,
Figure 2: (a) The framework first embeds different input modalities with feature embedders, then goes through MSoE to learn and retain the modality-specific features, MToE to model shared modal- ity information and modality-task dependence, and final fusion block connected with task heads. (b) The MTоE block creates data-dependent pathways for each multi-modal input, dispatching modality tokens experts and combining the expert outputs generate task-specific features. (c) Modality-specifit MoEs are used to extract and retain input-dependent podality-speciff features. (d). We apply a basic Soft MoE block after O' fuse all features.
Dynamic modality-tase dependence: In mudti- multi-task learning. the shared information between different modalities should be modeled differently to a given task instead of using a static shared set of features, forming modality dependence.
3.2 OVERALL FRAMEWORK
In this section, will demonstrate our M'OE framework for multi-modal multi-task learning and illustrate how it dynamically links the input modalities to different tasks. Our M' design com- prises modality-specific feature embedders, -opelity Soft MoEs, and shared Modality-Task Soft MoËs. We customize the MoEs into Vision Transformer blocks so that it creates a data- dependent path for each input. This enables more flexible modeling of dynamic modality depen- dence and dynamic mudti-modal fusion in a multi-modal multi-task learning setting. Specifically, as shown in Fig.
2 a), we first utilize feature embedder to project input modalities to embeddings X1, Xm E R Rexd To mitigate the modality competition phenomenon, where modality-specific information lost for most modalities, for each modality M, we use modality-speciff MoE blocks features Z, E lx from X,. Meanwhile,
to extract and retain ieput-dependent modality-speciioe we feed XI....,X,, into our proposed MToE block and obtain the task-dependent fused modality features, producing task- speccfic features pool GI...., Gp. Finally, each task task-specific feature pool from G1. Gp is fused with coodlltyy-speciffe specific and shared pool H1 ..., Hp- For each task Tr. H₁ is passed to the task heads after a basic soft MoE block for final predictions. The overall framework can be seen in Fig. 2
task-shared feature pool Zi. Zm to form task-
The mopelity- MoE is Soft MoE, as described in Sec. 3.1 As shown in Fig. b)), each MSoE for modality i contains n experts and a learnable matrix T E Rddx,, where d represents the embedding dimension. Given input tokens X R we compute the dispatch matrix D R by applying softmax along the columns of X and the combine matrix C' E R!xa by applying softmax along the rows of Xd. We pass the linearly weighted input Xz, where X = DX, to each expert j and obtain the output Y,. Finally, we apply a linear combination of Y, based on C' to obtain the final output Y CÝ for the MSoE module.
5

---

Published as a conference paper at ICLR 2025
As shown in Fig. 2 c). the MToE block connects tasks to input modalities by using experts link. It is distinct from the basic soft MoE in that it takes in mixed inputs from multi-modalities, directs the mixture of their information through task-specific slots, and differentiates with learnable task emheddings. The MoR layer of the MTTE consists of n experts and a learnable matrix Pask E different tasks. Given the Raxaxp, and each expert processes p input task slots corresponding input token-type modality features X1 X1,..., Xm E Rlxd, there is total of M = I x m feature tokens. And then they are assigned to the task slots of each expert through convex combination of .I.....X, Xm. using dispatch weights D E R Mxxxp, The dispatch weight is given by applying a softmax over columns of X where X E
R Mxd is the concatenation of Xx,..., Dijk
exp((X9),x)) X DX. Each task slot k of expert j is processed by the expert j's neural E. exp((X(),,z)
network f,. to produce the output slots: ,t = [X, + TEt), where TEg is the learnable task embedding of task k that is shared across experts. The final output Y of each task k is obtained by taking a convex combination of all the output slots Y. This process is similar to the previous one:
Cijk- 2 Eizr (xp((X)).sa)' Y C. where C, the combine weights, are computed by applying softmax over the rows of X This facilitates the dynamic connection between different
exp((X0)1a)
input modalities and tasks and enables probability modeling in Sec. 3.3
3.3 JOINT PROBABILITY MODELING OF MODALITY, TASK AND EXPERTS
To understand how each modality contributes to different tasks, we define a probability model over tasks T, experts E, and modalities M. We assume that when our trained network is deployed, it will receive a random task and a corresponding distribution P(T) The input-conditioned and dynamic multi-expert structure of MToE makes it well-suited for jointly modeling T and M, where the á E mmxaxp matrix and task slots collabora- tively connect modality M to expert E and subsequently task T.
modal image pair I based on a global task
In the MTоE module, we model the probability P(Eş[Tk) of selecting expert Ej for task Ts based on how much importance the routing network assigns expert E, task Tı. For example, in sparse MoEs, if the routing network assigns 30 out of 100 image pairs from task T3 to expert Ey, then (TE) = 0.3. In the MToE, rather than making hard assignments, we calculate these probabil- ities by summing the soft weights from the routing matrix X Given this matrix, the conditional probability of an input X can then be determined as follows: P(E)(T&))
(x*),x)),44 EE Ei_ (X+))...** where m is the number of modalities, n is the number of experts, and p is the number of tasks. Similarly, the conditional probability (TM) can be represented by the frequency of how a modality i is selected to contribute to task j, which can be written as follows: P(M[
n (X+):.1a
Ei=, (X To model how experts are assigned to each modality for different tasks, we need to model the conditional probability P(M, E)T). This can be given by dispatch weights as well. Given the input X and task k, the modality is assigned to the expert j with the probability
Σn -;= (X0).,1
P(M, E ) which can be expressed by: P(M,, ET(Ta) = E.s E, ET (X0),,
Based on our hypothesis of modality-task dependence, in the multi-modal multi-task learning set- ting, the most important experts and the selection of input modality features should depend on the task. This allows the MToE to assign specific experts to corresponding modalities for each task. To achieve so, could maximize the conditional mutual information between experts E and modali-
ties M given T: [(M; E|T)= = Σm Σj= E PMM,, E, (Tx) log P(M, Th P(F, (73)
P(M,E, ETT.
This term can be calculated based on our probabilistic modeling above in the MT
To understand this objective, we can decompose it as [(M; E/T) = (M H(MEE, Term (M E, T)) refers to the entropy of modality given task and experts, smaller H H E, T) encourages the MToE to learn a stronger dependency of the modality on both the expert and task, and leverage different combinations of modalities for different tasks. H (T refers the entropy of modalities given tasks, where a higher term ensures modality diversity instead of repeatedly relying
This distribution typically derived from the dataset, as not data points have labels for every task. In implementation, & is with the shape of (m I) x x since each modality has I tokens. For each modality, we sum up the probability of all of its tokens to marginalize probahility for each Modality m.

---

Published as a conference paper at ICLR 2025
on small subset of modalities. Given the training objective Lt, of each task tgp the final loss L can be written as: c=E:- Lig (M ElT), where a is the haper-prrameter that adjusts the strength of conditional mutual information loss.
EXPERIMENT
4.1 EXPERIMENT SETTING
Datasets. To test o'E' generalizability across populations and medical imaging modalities, conduct extensive experiments over four public datasets on Iwo use cases, Mammogram screen- ing: We use EMory BrEast Imaging Dataset (EMBED) (Jeong et al. 2023) collected in the United States, The 2023 RSNA Screening Mammography Breast Cancer Detection AI Challenge Dataset (RSNA) collected in Australia (Carr et al.). and VinDr-Mammo Dataset (VinDr) collected in Viet- nam (Nguyen et al. 2023). EMBED has four modalities (2 view FFDM and 2 view 2DS) and seven tasks: BI-RADS assessment (3-class), 1-5 years risk scoring (binary), and density classification(4 class). RSNA and VinDr have two modalities (2 view FFDM) and two tasks: density classification (4-class) and BI-RADS assessment (3-class). Ophthalmology screening: we use the Glaucoma grAding from Multi-Modality imAges (GAMMA) dataset (Wu et al. 2023) collected in China.
GAMMA has two modalities (color fundus photos and OCT) and two tasks: glaucoma detection (3-class) and optical cup segmentation.
Evaluation. Accuracy is used for classification tasks, and DICE score is used for segmentation
across 5-fold cross validation. For the o' model reported in Table 1 we use 128 experts for the MToE module and 32 experts for each of the MSoE modules. We use Adam optimizer with learning rate of lc-4 and StepLR scheduler with step size of 5 and gamma of 0.1. The model is trained on batch size of 32 and a total of 100 epochs. a is set to 0.05. Our hardware includes 4 NVIDIA A100s and 4 L40s. Detailed mean, standard deviation, AUROC and AUPRC, are in Appendix.
Baselines. We mainly compare our method with four groups of SOTA methods: 1) Multi-modal single-task baselines from medical AI. 2) Multi-modal multi-task baselines from medical AI. In I), we compare with MIRAI (Yala et al.) (022) and Asymmirai (Donnelly et al. (2224) for mammog- raphy, Eyemost (Zuu et al. 2024b) and Eyestar (Wu af., 2023) for ophthalmology. compare with the 4 methods under multi-task setting. 3) MoE-related baselines. We compare with a multi-modal soft mixture of experts (Puigcerver et al. (003) and the combination of MIRAI and Asymmirai with our MToE. 4) Multi-modal multi-task baselines adapted from the natural domain, including FULLER (Huang et al. 2023). AIDE (Yang et al. 2023). J. MModN (Swamy et al. 2024
2), we
EVIF (Geng et al. (2024). Comparison with Adapted Multimodal Soft MoE (Puigcerver et al.
2023
AdaMV-MoE (Chen et al. 2023a), and Puse (Han et al. 2024) are included in Appendix.
4.2
MAIN RESULTS
Our O significantly outperforms baseline methods. As shown in Table 1 the single-task MSoE outperforms all the single-task medical Al baselines, demonstrating our design's effective- ness in retaining mspelity-speeific information. As for multi-task settings, ' consistently out- performs baseline methods across all tasks on all benchmarks. This highlights the effectiveness of O' in the multi-modal, multi-task learning setting.
Multi-tasks benefit from each other in our O' method, unlike baselines. In Table | our 'E trained in multi-task setting shows superior performance compared to its single-task training counterpart. This indicates that tasks involved in multi-task learning contribute to shared features. We show in Sec. 4.3 that this also reduces gradient conflicts that typically hinder the learning of task-specific features. In contrast, such improvements are not seen baseline methods. This may be because these methods fail to capture the dynamic dependence between modalities and tasks.
Our MT framework can generalize to different modal learning backbones. We have adapted MToE to Miral and Asymiral baselines for mammography tasks, and have adapted MToE to Eyemost and Eyestar baselines for retinal tasks. After being combined with our MToE module, the single-tast task oriented medical AI baselines obtain significant performance improvement, most on par with or even excceding Multi-mode modti Multi-task baselines from the natural domain. This highlights MToE's flexibility and generalizability.
7

---

Published as a conference paper at ICLR 2025
Table I: Multi-modal single-task and multi-task results over four benchmarks from mammography imaging and retinal imaging. Complete 7-task results on EMBED dataset are in appendix.
Setting
Method
EMBED
RSNA
VinDR
Method
GAMMA
Mammo.
Risk
Density
Birads
Density
Birads
Density
Birads
Retinal
Glau.
Seg.
Single
Mirai
84.0
82.3
72.5
76.3
62.3
86.3
66.1
Eyemost
86.0
87.8
-Task
Asysmirai
79.0
80.2
69.4
74.1
60.1
78.9
62.4
Eyestar
85.4
85.9
Ours wo MSoE
83.2
82.1
71.9
76.8
62.2
85.4
65.8
Our MSoE
86.2
86.2
Ours
85.5
83.6
74.2
77.5
64.0
87.7
67.5
Ours
87.6
88.9
Multi
Mirai
83.1 1
83.2
72.3
76. 1
62.5
85.4
65.9
EyeMost
86.5
86.6
-Task
+MToE
84.6
83.3
73.4
76.8
63.1
85.9
66.4
+MToB
87.2
88.5
Asymirai
80.0
82.5
69.6
73.9
60.7
79.1
62.2
Eyestar
85.9
84,1
+MTOE
81.9
83.1
70.8
74.2
61.2
81.2
64.2
+MToB
87.5
86.9
EVIF
84.7
83.4
73.6
77.0
65.9
88.8
70.7
EVIF
88.7
87.0
Fuller
84,3
83.5
72.5
76.9
65.4
87.1
68.7
Fuller
87.8
86,2
AIDE
82.9
73.6
66.
88.2
69.8
AIDE
88.5
87.5
MModN
84.8
83.7
73.9
77.1
66.4
89.0
70.4
MModN
89.2
87.2
Ours wo MTaE
84.0
82.8
73.1
76.7
64.3
86.9
67.3
Ours wo MToE
88.1
87.4
Ours
85.9
84.1
75.1
77.8
66.7
89.6
71.8
Ours
90.4
89.7
00. Modally Coappiition
Mt. Modally Utttastin (Paaa Sett Medl
c1. Modality Utibaatia (Dun)
S/2
Figure 3: (a). Performance drop caused by modality competition. (b). Modality utilization of multi- modal soft MoE. (c). Modality utilization of M Our method has less modality competition.
4.3 ANALYSIS
Modality Utilization in Non-MoE Fusion, MoE vs M' To investigate our first hypothesis, we examine how well traditional (non-MoE) multi-modal models, multi-modal MoE, and our approach utilize the four different modalities on the EMBED dataset on the cancer risk prediction task. For tra- ditional method, we first train a multi-modal model where each modality is encoded by modality- specific ViT encoder. The representations from each modality are then averaged and then passed through dense fusion layer before being fed into the task heads. We also trained four unimodal models for each modality. As shown in Fig. 3a), we compared the four modality-specific (from the multi-modal model) encoders' performance with that of their unimodal counterpart. The en- coders from the multi-modal model exhibited a sigmificant performance gap compared to the model trained solely that modality, especially for modalities FFDM left-to-right (M2), 2DS up-to-down (M3), and 2Ds left-to-right (M4). This indicates that M2-M4 is much more poorly optimized in the multi-modal model compared to MI, i.e. losses the modality competition (Huang et al.) 2022).
For the MoE-based method, it is easier to evaluate the modality utilization across experts: For modality M,, we first obtain the sum of input-conditioned modality dispatch weights assigned to M,'s token for a data sample and a given expert Ey. then aggregate this value over all experts and over all data points. In Fig. we observe that when simply using a multi-modal soft MoE architecture, modality dominates the expert space. In contrast, O alleviates this issue by learning a much more diverse modality utilization and achieves higher performance as shown in Table. I (Without MSoE v.s. M.oE).
Dynamic Modality Dependence. Recall our second hypothesis that the fused features from dif- ferent modalities should be dynamically dependent on different tasks. Unlike M oF,,, traditional multi-modal fusion models lack interpretability of modality-task dependence. To ensure a fair com- parison, we introduce the synergy measure from partial information decomposition (PID) theory, which quantifics how much new task-specific information emerges when two modalities are fused.
We train synergy estimators using BATCH algorithm following Liang et al. (2024) to calculate pair- wise synergy between input modalities for a given task. For this experiment, we train models on the EMBED dataset with density prediction and 1 1-year risk prediction tasks. We utilize the two FFDM modalities and two 2DS modalities for training, and this results in 6 pairwise synergies per task, visualized as matrices in Fig. 4 We observe that the synergy matrix for the baseline MIRAI
8

---

Published as a conference paper at ICLR 2025
a). Task synergy of baseline
b). Task symergy of basellie
<). Task I synergy of ours
d). Task symergy of ours
0216
200
9
640
0012
0004
0000
0036
0321
0006
0x07
0002
064d
0012
0013
000
0099
0045
0005
Figure 4: Visualization of modality-task dependence. We report modality-pair wise synergies which measure how two modalities benefit from each other contributing the given prediction task.
Pepulation Lovel Madallly Consituticn (Per task, eddre doowet
PobeeT High maa, ligh
Overt
Somple Leed Modalty Contribttio Per patort, per bon))
IOO ude
201
Figure 5: Visualization of modality contribution on patient level and dataset level.
method shows similar value distributions across different tasks, where shared information of two- view FFDM is prevalent (MI and M2). While Task 1 (density) and Task (risk) synergy for baseline do capture different absolute values, the highest synergy pair still only peaks for M1 and M2 (within two views FFDM) for both tasks. However, this doesn't 't follow our dynamic modality-task depen- dence intuition. While density information is known to be mostly captured by FFDMs (Destounis jet al. 2020; Zuckerman et al. 2016), prior work (Kleinknecht et al.) 2020 Gastounioti et al. 2022) in mammogram screening suggests that for the cancer risk task, models should leverage usetul in- sights from all four modalities. Our model shows a clear contrast of synergy distributions between the two tasks, and the cancer risk task's distribution derives information from more diverse infor- mation from different modalities. The benefits can be further proven by the improved results after adding MToE in Table 1 (Without MToE vs Mt For density, our method peaks at M1-M2. For risk, we peak at not only M1-M2 synergy (within FFDM) but also M1-M4 synergy (between FFDM and 2DS). We also capture the M2-M3 and M3-M4 synergy much better than the baseline. This in- dicates a stronger modality-task dependency. This improvement is likely due to our method's ability to hetter learn desk-depnndent mutual information between different modalities. Further details on PID theory and BATCH algorithm can be found in Appendix
Sample and Population-Leve Modality Contribution Interpretability. M. has inherent sam- ple and population level interpretability of modality contribution to specific task, because the & matrix determines both how much weight each modality contributes to an expert and how much weight each expert contributes to task. We use two tasks from the EMBEDS dataset to illus- trate this. I) Sample-level (local) modality contribution for one patient input: Given a task, we calculate the importance weight of each modality for each expert, and average over all experts by expert-task assignment weights. 2) Population-level (global) modality contribution: we aggregate the sample-level contribution over the entire dataset. Fig. provides granular insights on how dif- ferent modalities contribute to the task predictions of two different patient samples and over the entire EMBEDS dataset. We observe distinct patterns across patients and clinically sensible global patterns: FFDM contributes more to density, and all modalities should contribute to risk prediction (Destounis al. 2020 Aujero et al. 2017 Zuckerman al., 2016).
Gradient Conflict Mitigation. Gradient conflict problem is a long-standing issue in multi-task learning that may harm the model's ability to learn task-specific information (Chen et al.) 2023c Zhou et al., 2024). This issue is also present in multi-modal scenarios, shown in Fig observe that O' also helps mitigate this issue. We train a ViT-based multi-task late fusion model using four modalities in mammography, predicting density and I-year risk. We collect the gradients from the last layer of the fusion block during back propagattnn for both task losses in each step of an epoch. As shown in Fig 6 we calculate the cosine similarities between the gradients of the two tasks and categorize the gradient relationships into three categories: "conflict" (similarity between -1 and -0.01), "neutral" (-0.01 to 0.01), and "enhance" (0.01 to 1). The baseline model shows
We
9

---

Published as a conference paper at ICLR 2025
a). Task Gradient Conflict Annlysis
b), Task-Expert Heatmap Base MoE
c). Tisk-EExprr Heatmup M'
0.6
0.2
Conflict
Neutal
Echance
Figure 6: (a). Gradients from different task heads may conflict, and method has mitigated this issue. (b). Compared to basic Soft MoE, our method has a sparser expert-task dependency that may have helped alleviate the gradient conflict problem.
high proportion of conflicts, indicating significant gradient conflicts in the multi-modal, multi-task setting. contrast, our M'oE model exhibits much lower proportion of conflicts, demonstrating that method effectively mitigates gradient conflicts. This improvement may be attributed to sparse task-cxpert dependency. where experts tend to specialize different tasks. As shown in Fig. 6 conduct reduced experiment (16 experts and 7 tasks) and generate heatmaps of the expert-task assignment scores from the combine matrix to visualize the expert-task dependencies in both the baseline MoE model and our O' Our method shows higher sparsity, enabling dynamic separation of task-specific parameters and helping to reduce gradient conflicts.
4.4 ABLATION STUDY
Ablation of Different Model Components. shown Table we conduct ablation studies on each component of the o' framework. Removing either the MSoE or MToE leads to significant performance drop. This is because: 1) the MSoE retains and contributes modality- specific features to the embedding pool and alleviates the modality competition issue. 2) The task-slot and task- specific embedding guided MToE data flow capture the dependence between modalities and tasks to find the most suitable modality feature selection and fusion for each task. Additionally, removing the cnatualliffommatiion regularization loss while retaining the MToE results in a slight performance decrease compared to original M' model, the loss encourages higher modality-tann -tak dependence and more diverse modality feature fusion.
Increasing the number of experts and regularization degree within certain range improves the performance. As shown in Fig. increasing the number of experts in the MToE from 16 to 128 improves the model's performance. Additionally, raising the value of a, which controls the degree of mutual information regularization, from 0.01 to 0.2 generally enhances the model's performance, but too heavy of weight makes the predictions slightly worse. Detailed results are in the Appendix.
37
Table 2: Ablation results on technical components in M
90
54
MToE
MSoE
MI Reg.
EMBED
65
1 year risk
Density
Birads
84,0
82.8
73.1
85.1
83.5
73.8
Figure 7: Ablation expert number and regularization degree a.
85.2
83.7
73.5
85.9
84.1
75.1
CONCLUSION
In this paper, we propose Mf a Multi-modal Multi-task Mixture of Experts framework for Med- ical multi-modas multi-task learning. O' enables dynamic, cample-adaptiv modality fusion and modality-tase dependence modeling. It jointly models the dependence of modalities, experts, and tasks and captures dynamic distinct and shared modality information. Extensive experiments show that M consistently outperforms baselines across diverse medical imaging benchmarks, offers interpretability of modality contribution, and can be flexibly combined with different backbones.
Acknowledgements. The authors acknowledge support from Michigan Institute for Computational Discovery and Engineering Catalyst Grant, and Michigan Institute for Data Science PODS Grant.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Julián N Acosta, Guido Falcone, Pranav Rajpurkar, and Eric J Topol. Multimodal biomedical ai. Nature Medicine, 187971773--1784 2022.
Mireille P Aujero, Sara C Gavenonis, Ron Benjamin, Zugui Zhang, and Jacqueline S Holt. Clinical performance of synthesized two-dimensional mammography combined with tomosynthesis large screening population. Radiology, 283(():70-76, 2017.
Nils Bertschinger, Johannes Rauh, Eckehard Olbrich, Jürgen Jost, and Nihat Ay. Quantifying unique information. Entropy, 16(4):2161-218) 2014.
Ann L Brown, Charmi Vijapura, Mitva Patel, Alexis De La Cruz, and Rifat Wahab, Breast cancer in dense breasts: detection challenges and supplemental screening opportunities. RadioGraphics, 43(10);e230024, 2023.
Zhiyuan Cai, Li Lin, Huaqing He, Pujin Cheng, and Xiaoying Tang. Uni4eye++: A general masked image modeling multi-modal pre-training framework for ophthalmic image classification and seg- mentation. IEEE Transactions on Medical Imaging, 2024.
Bing Cao, Yiming Sun, Pengfei Zhu, and Qinghua Hu. Multi-modal gated mixture of local-to-global experts for dynamic image fusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23555 235644, 2023.
Chris Carr, Felipe Kitamura, Kalpathy-Crann Cramer, J Mongan, K Andriole, M Vazirabad, M Riopel, R Ball, and S Dane. Rsna screening mammography breast cancer detection. 2022.
Cheng Chen, Qi Dou, Yueming Jin, Hao Chen, Jing Qin, and Pheng-Ann Heng. Robust multi- modal brain tumor segmentation via feature disentanglement and gated fusion. In Medical Image Computing and Computer Assisted Intervention-MIICAA 2019: 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part IIII 22, pp. 447-456. Springer, 2019.
Tianlong Chen, Xuxi Chen, Xianzhi Du, Abdullah Rashwan, Fan Yang. Huizhong Chen, Zhangyang Wang, and Yeging Li. Adamv-moc: Adaptive multi-task vision mixture-of-experte In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 17346-17357, 2023a.
Yuanyuan Chen, Yongsheng Pan, Yong Xia, and Yixuan Yuan. Disentangle first, then distill: A unified framework for missing modality imputation and alzheimer's 's disease diagnosis. IEEE Transactions Medical Imaging, 2023b.
Zitian Chen, Yikang Shen, Mingyu Ding, Zhenfang Chen, Hengshuang Zhao Erik G Learned- Miller, and Chuang Gan. Mod-squad: Designing mixtures of experts modular multi-task learn- ers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. (1828-11837, 2023c.
Stamatia V Destounis, Amanda Santacroce, and Andrea Arieno. Update on breast density, risk estimation, and supplemental screening. American Journal of Roentgenology. 214(2):2963305 2020.
Cuturi M Sinkhorn Distances. Lightspeed computation of optimal transport. Advances in neural information processing systems, 26:2292-2300. 2013.
Jon Donnelly, Luke Moffett, Alina Jade Barnett, Hari Trivedi, Fides Schwartz, Joseph Lo, and Cynthia Rudin. Asymmirai: Interpretable mammography-basee deep learning model for 1-5- year breast cancer risk prediction. Radiology, 310(3);232280, 2024.
Alexey Dosovitskiy. An image is worth 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1199 2020.
Aimilia Gastounioti, Shyam Desai, Vinayak Ahluwalia, Emily F Conant, and Despina Kontos. Artificial intelligence in mammographic phenotyping of breast cancer risk: a narrative review. Breast Cancer Research, 24(1):14, 2022.
11

---

Published as a conference paper at ICLR 2025
Mengyue Geng, Lin Zhu, Lizhi Wang, Wei Zhang, Ruiqin Xiong, and Yonghong Tian. Event- based visible and infrared fusion via multi-task collaboration. In Procediiggs of the IEEE/CVF Conference Computer Vision and Pattem Recognition, pp. 26929-26939, 2024.
Simon Graham, Quoe Dang Vu, Mostafa Jahanifar, Shan E Ahmed Raza, Fayyaz Minhas, David Snead, and Nasir Rajpoot. One model all you need: multi-task learning enables simultancous histology image segmentation and classification. Medical Image Analysis, 83:102685, 2023.
Xing Han, Huy Nguyen, Carl Harris, Nhat Ho, and Suchi Saria. Fusemoe: Mixture-of-experts transformers for fleximodal fusion. arXiv preprint wrXiv:2402.03220 2024.
Xingxin He, Ying Deng, Leyuan Fang, and Qinghua Peng. Multi-modal retinal image classification with modality-speciff attention network. IEEE transactions on medical imaging, 40(6):1591- 1602, 2021.
Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang, and Longbo Huang. Modality competition: What makes joint training of multi-modal network fail in deep carning?(provabll) In Interna- tional conference on machine learning, pp. 9226-9223 92599. PMLR, 2022.
Zhijian Huang, Sihao Lin, Guiyu Liu, Mukun Luo, Chaoqiang Ye, Hang Xu, Xiaojun Chang, and Xiaodan Liang. Fuller: Unified multi-modality multi-task 3d perception via multi-level gradient calibration. In Proceedings the IEEE/CVF International Conference Computer Vision, pp- 3502-3511, 2023.
Jiwoong Jeong, Brianna L Vey, Ananth Bhimireddy, Thomas Kim, Thiago Santos, Ramon Correa, Raman Dutt, Marina Mosunjac, Gabriela Oprea-llies Geoffrey Smith, et al. The emory breast imaging dataset (embed): A racially diverse, granular dataset of 3.4 million screening and diag- nostic mammographic images. Radiology: Artificial Intelligence, 5(1):e220047, 2023.
Jue Jiang, Yu-Chi Hu, Neelam Tyagi, Pengpeng Zhang, Andreas Rimner, Joseph O Deasy, and Harini Veeraraghavan. Cross-modality (ct-mri) prior augmented deep learning for robust lung tumor segmentation from small mr datasets. Medical physics, 46(10):43924444) -4094, 2019.
Lie Ju, Xin Wang, Xin Zhao, Huimin Lu, Dwarikanath Mahapatra, Paul Bonnington, and Zongyuan Ge. Synergic adversarial label learning for grading retinal diseases via knowledge distillation and multi-task learning. IEEE Journal of Biomedical and Health Informatics, 25(10):3709-372 2021.
Galvin Khara, Hari Trivedi, Mary Newell, Ravi Patel, Tobias Rijken, Peter Kecskemethy, and Ben Glocker. Generalisable deep learning method for mammographic density prediction across imaging techniques and self-reported race. Communications Medicine, 4(1):21, 2024a.
Galvin Khara, Hari Trivedi, Mary Newell, Ravi Patel, Tobias Rijken, Peter Kecskemethy, and Ben Glocker. Generalisable deep learning method for mammographic density prediction across imaging techniques and self-reported race. Communications Medicine, 4(1):21, 2024b.
Julian Hans Kleinknecht, Anca Ileana Ciurea, and Cristiana Augusta Ciortea. Pros and cons for breast cancer screening with tomosynthesis-a review of the literature. Medicine and Pharmacy Reports, 93(4):335, 2020.
Adrienne Kline, Hanyin Wang, Yikuan Li, Saya Dennis, Meghan Hutch, Zhenxing Xu, Fei Wang, Feixiong Cheng, and Yuan Luo. Multimodal machine learning in precision health: A scoping review. npj Digital Medicine, 5(1):171, 2022.
Edward Korot, Zeyu Guan, Daniel Ferraz, Siegfried Wagner, Gongyu Zhang, Xiaoxuan Liu, Livia Faes, Nikolas Pontikos, Samuel G Finlayson, Hagar Khalid, et al. Code-free deep learning for multi-modality medical image classification. Nature Machine Intelligence, 3(4):288-298, 2021.
Paul Pu Liang, Yun Cheng, Ruslan Salakhutdinov, and Louis-Philippe Morency. Multimodal fu- sion interactions: A study human and automatic quantification. Proceedings of the 25th International Conference Multimodal Interaction, pp. 425-435-5 2023.
12

---

Published as a conference paper at ICLR 2025
Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, et al. Quantifying & modeling multimodal interactions: An information decomposition framework. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Aakarsh Malhotra, Surbhi Mittal, Puspita Majumdar, Saheb Chhabra, Kartik Thakral, Mayank Vatsa, Richa Singh, Santanu Chaudhury, Ashwin Pudrod, and Anjali Agrawal. Multi-task driven explainable diagnosis of covid-19 using chest x-ray images. Pattern Recognition, 122:108243, 2022.
Zdravko Marinov, Simon Reiß, David Kersting, Jens Kleesiek, and Rainer Sticfelhagen. Mirror u- net: Marrying multimodal fission with multi-task learning for semantic segmentation in medical imaging. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 2283-2293. 2023.
Mingyuan Meng. Bingxin Gu, Lei Bi, Shaoli Song, David Dagan Feng, and Jinman Kim. Deepmts: deep multi-task learning for survival prediction patients with advanced nasopharyngeal car- cinoma using pretreatment pet/ct. IEEE Journal of Biomedical and Health Informatics, 26(9): 4497-4507, 2022.
Mingyuan Meng, Lei Bi, Michael Fulham, Dagan Feng, and Jinman Kim. transformer networks for survival prediction in head and neck cancer. In International Conference Medical Image Computing and Cosputer-Assisted Intervention, pp. 400-410. Springer, 2023.
hybrid
Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Mul- timodal contrastive learning with limoe: the language-image mixture of experts. Advances in Neural Information Processing Systems, 35:9564 9576, 2022.
Hieu T Nguyen, Ha Q Nguyen, Hieu H Pham, Khanh Lam, Linh T Le, Minh Dao, and Van Vu. Vindr-mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography. Scientific Data, 10(1):277, 2023.
Olaide N Oyelade, Eric Aghiomesi Irunokhai, and Hui Wang. A twin convolutional neural network with hybrid binary optimizer for multimodal breast cancer digital image classification. Scientific Reports, 14(1):692, 2024.
Bernd J Pichler, Martin S Judenhofer, and Christina Pfannenberg. Multimodal imaging approaches: pet/ct and pet/mri. Molecular Imaging I, pp. 109-132, 2008
Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts. arXiv preprint arXiv:2308.0095: 2023.
Wei Shao, Hang Shi, Jianxin Liu, Yingli Zuo, Liang Sun, TianSong Xia, Wanyuan Chen, Peng Wan, JianPeng Sheng, Qi Zhu, et al. Multi-instance multi-task learning for joint clinical outcome and genomic profile predictions from the histopathological images. IEEE Transactions on Medical Imaging, 2024.
Sören Richard Stahlschmidt, Benjamin Ulfenborg, and Jane Synnergren. Multimodal deep learning for biomedical data fusion: a review. Briefings in Bioinformatics, 23(2):bbab569, 2022.
Vinitra Swamy, Malika Satayeva, Jibril Frej, Thierry Bossy, Thijs Vogels, Martin Jaggi, Tanja Käser, and Mary-Anne Hartley. Multimodn- multi-task, interpretable modular networks. Advances in Neural Information Processing Systems, 36, 2024.
Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, et al. Towards generalist biomedical ai. NEJM AI, Aloa23333311 2024.
Hu Wang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, and Gustavo Carnerr. Multi- modal learning with missing modality via shared-species feature modelling. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 15878-15887, 2023.
13

---

Published as a conference paper at ICLR 2025
Amy Weisman, Ojaswita Lokre, Brayden Schott, Victor Fernandes, Robert Jeraj. Timothy Perk, Steve Cho, and Scott Perlman. Automated detection and quantification of neuroendocrine tumors on 68ga-dotatate pet/ct images using u-net ensemble method, 2022.
Junde Wu, Huihui Fang, Fei Li, Huazhu Fu, Fengbin Lin, Jiongcheng Li, Yue Huang, Qinji Yu, Sifan Song, Ximxing Xu, et al. Gamma challenge: glaucoma grading from multi- multi-modality images. Medical Image Analysis, 90:102938, 2023.
Zhongliang Xue, Ping Li, Liang Zhang, Xiaoyuan Lu, Guangming Zhu, Peiyi Shen, Syed Afaq Ali Shah, and Mohammed Bennamoun. Multi-modal co-learning for liver lesion segmentation on pet-ct images. IEEE Transactions on Medical Imaging, (0(12):3531-5444 2021.
Adam Yala, Peter G Mikhael, Fredrik Strand, Gigin Lin, Siddharth Satuluru, Thomas Kim, Imon Banerjee, Judy Gichoya, Hari Trivedi, Constance D Lehman, al. Multi-instiiutional validation of a mammography- breast cancer risk model. Journal of Clinical Oncology, 40(16):1732- 1740, 2022.
Dingkang Yang, Shuai Huang, Zhi Xu, Zhenpeng Li, Shunli Wang, Mingcheng Li, Yuzheng Wang, Yang Liu, Kun Yang, Zhaoyu Chen, et al. Aide: A vision-driven multi-view, multi-modal, multi- tasking dataset for assistive driving perception.
Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 20459-20470, 2023.
Jiancheng Yang. Multi-task learning for medical foundation models. Nature Computational Science pp. 1-2, 2024.
Wenfang Yao, Chen Liu, Kejing Yin, William K Cheung, and Jing Qin. Addressing asynchronic- ity in clinical multimodal fusion via individualized chest x-ray generation. arXiv preprint arXiv:2410. 17918, 2024a.
Wenfang Yao, Kejing Yin, William K Cheung, Jia Liu, and Jing Qin. Drfuse: Learning disentangled representation for clinical modti-modal fusion with missing modality and modal inconsistency. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 16416-16424, 2024b.
Sukwon Yun. Inyoung Choi, Jie Peng, Yangfan Wu, Jingxuan Bao, Qiyiwen Zhang, Jiayi Xin, Qi Long, and Tianlong Chen. Flex-moe: Modeling arbitrary modality combination via the flexible mixture-of-experts. arXiv preprint arXiv:2410.08245 2024.
Yilan Zhang, Yingxue Xu, Jianqi Chen, Fengying Xie, and Hao Chen. Prototypical informa- tion bottlenecking and disentangling for multimodal cancer survival prediction. arXiv preprint arXiv:2401.01646, 2024.
Yu Zhang and Qiang Yang. An overview of multi-task learning. National Science Review, 5(1): 30-43, 2018.
Yuanwang Zhang, Kaicong Sun, Yuxiao Liu, and Dinggang Shen. Transformer-based multimodal fusion for early diagnosis of alzheimer's disease using structural mri and pet. 2023 IEEE 20th International Symposium Biomedical Imaging (ISBI), pp- 1-5. IEEE, 2023.
Xiongjun Zhao, Xiang Wang, Fenglei Yu, Jiandong Shang, and Shaoliang Peng. Unimed: Mul- timodal multitask learning for medical predictions. In 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 1404. IEEE, 2022.
Yue Zhou, Houjin Chen, Yanfeng Li, Qin Liu, Xuanang Xu, Shu Wang, Pew-Thian Yap, and Ding- gang Shen. Multi-task learning for segmentation and classification of tumors in 3d automated breast ultrasound images. Medical Image Analysis, 70: 101901 2021.
Yuhang Zhou, Zihua Zhao, Siyuan Du, Haolin Li, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Exploring training on heterogeneous data with mixture of low-rank adapters. In Ruslan Salakhut- dinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference Machine Learning, vol- ume 235 of Proceedings of Machine Learning Research, pp. 62294 62306. PMLR, 21-27 Jul 2024. URL wtt.thtttp///ooottt ngs ml wttppppwws////eppppppppp hhwp://////hhuuu html
14

---

Published as a conference paper at ICLR 2025
Ke Zou, Tian Lin, Zongbo Han, Meng Wang, Xuedong Yuan, Haoyu Chen, Changqing Zhang, Xiao- jing Shen, and Huazhu Fu. Confidence-awe aware multi-modalityy learning for eye disease screening. Medical Image Analysis, 96: 96: 103214,,, 2024a.
Ke Zou, Tian Lin, Zongbo Han, Meng Wang, Xuedong Yuan, Haoyu Chen, Changqing Zhang, Xiao- jing Shen, and Huazhu Fu. Confidence-awarr multi-modality learning for eye disease screening. Medical Image Analysis, 96: 103214, 2024b.
Samantha P Zuckerman, Emily F Conant, Brad Keller, Andrew DA Maidment, Bruno Baru- faldi, Susan P Weinstein, Marie Synnestvedt, and Elizabeth S McDonald. Implementation of synthesized two-dimensional mammography in a population-baser digital breast tomosynthesis screening program. Radiology, 281(3):730- 0007333--36,. 2016.
15


Reviews:
[{"review_id": "kOyDROhYyp", "reviewer": "Reviewer_P7K2", "summary": "In this paper, the author present a new framework for training multi-modal networks, called the Multi-modal Multi-task Mixture of Experts. The framework consists of two components:\n- MSoE: Modality specific mixture of experts --> for each modality, they learn a function g that applies:\n column-wise softmax (D) on X times a learnable matrix, multiplied by X, followed by row-wise softmax (C) on the output and a linear combination to compute the prediction.\n- MToE: Modality shared modality task mixture of experts --> connects tasks to input modalities by learning a task embedding shared across experts.\n\nThey also propose a mutual information loss and evaluate the approach on four publicly available medical imaging datasets for breast cancer and OCT.", "questions": "- Can the authors discuss the scalability of the framework? What is the computational complexity?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The framework presented is original and interesting. It outperforms existing baseline models. The authors run experiments on multiple datasets and conduct an ablation study.", "weaknesses": "- The paper presentation requires improvement. For example, there is unnecessary use of ; and there is incorrect use of opening quotations \". The authors also repeatedly introduce the abbreviations - this should be done once.\n- I found it difficult to parse through Figure 2 (Can you relate it with the textual explanation of the functions?)\n- The authors only compare to a few baselines, can you incorporate more? There is a lot of literature on multimodal learning now.\n- Are the performance improvements significant? Can you conduct significance testing and provide confidence intervals?\n- The experiments are conducted on medical imaging datasets. How does this apply to other non-imaging modalities where modality competition may be more pronounced. For example, this could be applicable to MIMIC CXR (chest X-rays) and MIMIC EHR where downstream tasks are more dependent on the EHR modality.\n- The main results section in the text should also discuss the quantitative results.\n- What was your hyperparameter tuning strategy? It is unclear if these baselines have been best optimized.\n- Can you also compute AUROC and AUPRC for the classification tasks? Accuracy is not sufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ksCn2chrig", "reviewer": "Reviewer_xATa", "summary": "The paper mainly addresses two challenges in clinical tasks: patient-level and task-level dynamic fusion. For the patient-level fusion, a modality-specific MoE is employed. For the task-level fusion, a modality-task MoE with conditional MI regularization between experts and modalities given tasks is adopted. The experiments using EMBED, RSNA, VinDR, and GAMMA datasets outperform existing methods in both single-task and multi-task settings.", "questions": "- Are there any ablation studies on datasets other than EMBED?\n- Are there any theoretical explanations on why the method mitigates gradient conflict?\n- Is there any clinical interpretation of the results in Figure 5? For example, the difference in modality contribution across different tasks.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-structured and easy to follow. \n- The motivation is clearly stated and convincing. \n- The experiments show promising results over many baselines both in stand-alone and add-on manners.  \n- The paper adopted PID to make a fair comparison of synergy information.", "weaknesses": "- As far as I understand, there have been works leveraging the shared and specific information across modalities and should be included in discussions, see [1-3]. \n- Is there an ablation study for a reduced number of experts? How sensitive is this method when the number of experts decreases compared to other MoE methods? What is the procedure for choosing the number of experts? \n- Please discuss the computational cost compared to the baselines. \n\n\n[1]Wang, Hu, et al. \"Multi-modal learning with missing modality via shared-specific feature modelling.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[2]Yao, Wenfang, et al. \"DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 15. 2024.\n\n[3]Chen, Cheng, et al. \"Robust multimodal brain tumor segmentation via feature disentanglement and gated fusion.\" Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part III 22. Springer International Publishing, 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "beQvBiay8T", "reviewer": "Reviewer_dm8E", "summary": "This paper proposes a multi-modal, multi-task, mixture of experts for various medical diagnoses to address the challenges of sample-dynamic modality fusion (and modality-task dependence (selecting the right modalities for a task). Concretely, this is done by using a combination of modality-specific experts and experts shared between modalities and tasks. M4OE shows promising initial results in terms of both absolute performance and enforcing modality utilization.\n\nBased on the weaknesses and questions outlined my score indicates a rejection for now, but I generally like the motivation of the paper, especially the aspect on modality utilization. I am willing to increase my score if my concerns are addressed and questions clarified.", "questions": "- How do you determine which expert sees which task? The connection between Figure 1 and the method section is not very clear.\n- The manuscript talks a lot about sample adaptivity, but how does your experimental setup show that the model handles sample adaptivity effectively? Which aspects of sample adaptivity?\n- Figure 3c suggests that the modality utilization is forced towards the same mean in your method. What about cases where modality dominance/competition is good? For example, if I have one very noisy modality, wouldn’t it be desirable to have the modality that contains all the signal to get all the model’s attention? Isn’t this graph showing that we enforce equal utilisation of all modalities regardless of the signal? Additionally, does this finding not contradict your claim in Figure 1, which is that only some experts are used (as opposed to all experts with a more balanced contribution).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The M4OE is highly effective at enforcing modality utilization - this is a meaningful contribution that many multimodal models suffer from, although I do have some questions about this.\n- The overall performance of the model is outperforming the baseline, even if the results are missing crucial information to validate the statistical significance of the results.\n- Strong visuals that are additive to the understanding of the paper.\n- Good conceptual motivation of the paper, although I believe that the motivation would further benefit from some concrete examples of sample dynamism and clinical examples of tasks that are modality-dependent.", "weaknesses": "- Abstract: the one-liner for sample-dynamic modality fusion is unclear as the specific and shared information always varies per sample unless they are identical. To my knowledge, sample-dynamic spans a much wider field of problems like missingness, robustness to noise, which the manuscript does not consider.\n- Abstract: “Results demonstrate superiority over state-of-the-art methods” is extremely vague. Along which metric?\n- You claim an expansive space by saying that the method is “multi-modal multi-task”, but your experiments only look at multi-view settings of a single modality (images). I would encourage you to narrow the scope/claim of the paper as the paper does not consider heterogeneous modalities (images, text, tabular, etc.).\n- Experimental setup: I would encourage you to provide more detail in this section to aid reproducibility. For example, it is unclear whether cross-validation is used. No confidence intervals or standard deviation of results are reported to judge the statistical significance of the results. Additionally, no code was provided in the supplementary materials that would help with the clarification of the experimental setup.\n- Literature: missing out on the largest corpus of literature (intermediate fusion), which many latent variable models for multimodal fusion fall under, many of which are using a mix of modality-specific and shared spaces.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HempfVZAo4", "reviewer": "Reviewer_7L3d", "summary": "This paper introduces M4oE, a framework for multi-modal, multi-task learning in medical diagnosis. M4oE addresses two primary challenges in multimodal diagnosis: sample-dynamic modality fusion and modality-task dependence. The framework incorporates modality-specific modules and a modality-shared modality-task mixture of experts (MoE) to dynamically learn both unique and shared information across modalities. A conditional mutual information loss is used to optimize the framework efficiently. Experimental results on two medical diagnostic tasks demonstrate M4oE’s advantages over existing methods.", "questions": "1. Could the authors provide computational resource requirements and runtime comparisons for M4oE and baseline methods?\n\n2. In Figure 4, the similar difference between subfigures (a) and (b) and between (c) and (d) suggests both M4oE and the baseline might be capturing modality-task dependence. Could the authors clarify?\n\n3. In Figure 3, is it reasonable to assume that the diverse distribution in (c) is preferable to that in (b), given the absence of a known ground truth distribution for each modality? Using the method in Liang et al. (2024) to quantify modality interactions could provide a more rigorous evaluation.\n\n4. In Figure 1(a), under Part 3, should the label for the green triangle be $G_p$?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The M4oE framework is a novel contribution, with two innovative components—modality-specific modules and a modality-shared modality-task MoE—that allow for efficient learning of both distinct and shared information across modalities for multiple tasks.\n\n2. The analysis is comprehensive, including detailed evaluations of modality competition, modality-task dependence, and sample-level modality contributions.", "weaknesses": "1. The M4oE framework is complex, raising concerns about optimization and practical implementation. Including a discussion on computational resources and runtime would help readers assess its feasibility for real-world applications.\n\n2. The paper does not address whether M4oE can function effectively when certain modalities or tasks are unavailable—a common scenario in clinical settings. Clarifying this would strengthen the model’s applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)