Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts
Authors: Sukwon Yun
Inyoung Choi
Jie Peng
Yangfan Wu
Jingxuan Bao
Qiyiwen Zhang
Jiayi Xin
Qi Long
Tianlong Chen
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Multi-modal learning
MoE
Primary area: machine_learning_for_healthcare

Abstract: Multimodal learning has gained increasing importance across various fields, offering the ability to integrate data from diverse sources such as images, text, and personalized records, which are frequently observed in medical domains. However, in scenarios where some modalities are missing, many existing frameworks struggle to accommodate arbitrary modality combinations, often relying heavily on a single modality or complete data. This oversight of potential modality combinations limits their applicability in real-world situations. To address this challenge, we propose Flex-MoE (Flexible Mixture-of-Experts), a new framework designed to flexibly incorporate arbitrary modality combinations while maintaining robustness to missing data. The core idea of Flex-MoE is to first address missing modalities using a new missing modality bank that integrates observed modality combinations with the corresponding missing ones. This is followed by a uniquely designed Sparse MoE framework. Specifically, Flex-MoE first trains experts using samples with all modalities to inject generalized knowledge through the generalized router ($\mathcal{G}$-Router). The $\mathcal{S}$-Router then specializes in handling fewer modality combinations by assigning the top-1 gate to the expert corresponding to the observed modality combination. We evaluate Flex-MoE on the ADNI dataset, which encompasses four modalities in the Alzheimer's Disease domain, as well as on the MIMIC-IV dataset. The results demonstrate the effectiveness of Flex-MoE, highlighting its ability to model arbitrary modality combinations in diverse missing modality scenarios. Code is available at: \url{https://github.com/UNITES-Lab/flex-moe}.

Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts
Sukwon Yun', Inyoung Choi Jie Peng Yangfan Wu Jingxuan Bao?, Qiyiwen Zhang", Jiayi Xin², Qi Long", Tianlong Chen
University of North Carolina Chapel Hill University of Pennsylvania 3 University of Science and Technology of China
(swyun, tianlong)@cs. unc. edu, (inyoungc, jiayixin)0seas (pengjieb, ustc_uyf/@mail ustc..us....... usu..edu............. edu. on (jingxuan. bao, qiyiwen. (hhangg)peemmennnn
wpw.n edu, qlong@upenn edu edu
Abstract
Multimodal learning has gained increasing importance across various fields, of- fering the ability to integrate data from diverse sources such images, text, and personalized records, which are frequently observed in medical domains. However, in scenarios where some modalities are missing, many existing frameworks struggle to accommodate arbitrary modality combinations, often relying heavily on single modality or complete data. This oversight of potential modality combinations limits their applicability in real-world situations. To address this challenge, we propose Flex-MoE (Flexible Mixture-of -fxxurr-of-Expeetts.... a new framework designed flexibly incorporate arbitrary modality combinations while maintaining robustness missing data. The core idea of Flex-MoE to first address missing modalities using new missing modality bank that integrates observed modality combinations with the corresponding missing ones. This is followed by a uniquely designed Sparse MoE framework. Specifically, Flex-MoE first trains experts using samples with all modalities to inject generalized knowledge through the generalized router (G-Router). The S-Router then specializes in handling fewer modality combina- tions by assigning the top-1 gate to the expert corresponding to the observed modal- ity combination. We evaluate Flex-MoE on the ADNI dataset, which encompasses four modalities in the Alzheimer's 's Disease domain, as well as on the MIMIC-IV dataset. The results demonstrate the effectiveness of Flex- F highlighting its ability to model arbitrary modality combinations in diverse missing modality sce- narios. Code is available at: htttt:: http::://ggihib...o//UUUETS Lab/f ff//fllxmmmm
Introduction
In many fields, including healthcare, language, and vision, multimodal learning 6 75 37 has emerged as a crucial approach for integrating data from multiple sources such clinical records, imaging, and genetic data. Multimodal data enables more comprehensive analysis and decision- making, offering the potential for improved diagnosis and prediction in various applications (59 33 68). However, a prominent challenge across these domains is the missing modality scenario 776 60
where not modalities are consistently available for every instance due to diverse reasons such as individualized data collection protocols the variable availability of certain modalities.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Clinical
Biospacimen
Clinical
Biospecimen
Clinical
Blospecimen
479
2,380
1,744
Image
Genetic
Image
157
21
Genetic
127
122
Image
Genetic
1,474
1,474
1,758
1,596
(a) Existing Single-modal
Existing Works Mutti-modal
(c) A Ours
Data statistics from real-world multimodal dataset (e.g., the Alzheimer's Disease Neu- roimaging Initiative (ADNI)), where patients exhibit unique combinations of available modalities. Existing approaches focus either (a) single-modality data (b) complete multimodal data, losing the potential to leverage other combinations. Our approach incorporates all possible modality combi- nations, offering more robust solution to the missing modality scenario.
Figure 2:
Clinicol
Blospeciman
As an representative example, in Alzheimer's Disease (AD) 4 one of the most prevalent neurodegenerative disorders, handling this inherently multimodal data is crucial for accurate diagnosis (Figure T). AD datasets often include combination of clinical symptoms, imaging data (42). and genetic profiles [48]. However, in weal-world clinical settings, not all these modalities readily available for each patient. Some data, such as clinical and imaging data, may be available from routine visits, whereas other data, such genetic biospecimen information, may require additional time to collect. This leads to incomplete datasets, which poses a challenge for existing models that tend to either rely heavily single modalities only utilize complete data, thereby missing the opportunity to leverage the full potential of multimodal learning (Figure
AD?
MDA
Genetic
Figure Multimodal AD.
Single Modality and Complete Data Reliance. The reliance on single-modality data complete data across many frameworks a significant limitation real-world scenarios, where missing data is the norm rather than the exception. As seen in Figure 2 many current models either work with single-mot modality data or focus on the intersection of modalities, neglecting the potential contribution of partially available modalities. In healthcare, particularly for discases such AD, this often leads to missed opportunities in diagnosis and treatment due to the inability to fully exploit multimodal data when some modalities missing.
Oversight Modality Combinations. Beyond the challenge of missing modalities, there is also the need to model the interactions between available modalities properly. Different combinations of modalities can provide complementary information, and each combination may hold unique significance for downstream tasks. For example, in AD diagnosis, combining biospecimen data and imaging data can reveal key insights: cerebrospinal fluid biomarkers may indicate early signs of AD 21.. while functional MRI can highlight cognitive impairments 42) Hence, it is essential develop models that not only handle missing modalities but also effectively utilize the available modality combinations.
Given the general challenge of the missing modality scenario in multimodal learning, we propose novel framework, Flex (Flexible Mixture-oo of of--xperts), to flexibly incorporate arbitrary modality combinations while maintaining robustness to missing data. FMex-MoE first sort samples based on the available modalities and process them through modality encoders. For missing modalities, we introduce learnable missing modality bank, which provides learnable embeddings for missing modalities based on the observed ones. This approach ensures that the model can handle incomplete datasets effectively. Our framework also builds upon Sparse Mixture-of-ENae Ex-eerrss (SMoE) design, allowing to generalize the expert knowledge from complete data (samples with all modalities) through the G-Router, followed by a specialized S-Router for handling fewer modality combinations. Each expert becomes specialized in handling different modality combinations, ensuring that the model can effectively process any combination modalities. We demonstrate the effectiveness of Flex-MoE through comprehensive experiments several real-world datasets, including the Alzheimer's s Disease Neuroimaging Initiative (ADNI), which involves four key modalities for AD stage prediction, and the MIMIC-IV dataset. The results confirm the robustness of Flex-MoE in diverse missing modality scenarios.
The contributions of this work can be summarized as follows:

---

We introduce flexible framework that effectively incorporates arbitrary modality combina- tions and addresses the missing modality scenario across various domains.
Flex-MoE features a novel approach, including missing modality bank and generalized and specialized expert training, which ensures robustness to missing modality scenario.
Extensive experiments on real-world datasets, including ADNI and MIMIC-IV, showcase the consistent and robust performance of Flex-MoE in handling diverse modality combinations.
2 Related Works
Single Modality Approach many fields, deep learning models often rely on single modality data for tasks such as classification [35 [I], diagnosis [56] 72], prediction [2] 61 34] While effective in certain cases, these approaches fail to capture the potential synergies between different data sources, especially in contexts where multiple modalities are available. In the Alzheimer's Disease domain, many studies focus on specific modalities. For instance, image- approaches include a VGG19 model
4 that diagnoses early-stage AD from MRI scans and modified ResNet18 architecture [45] that predicts AD progression using fMRI data. Other studies focus on genomics, such DLG (36] for classifying AD patients and SWAT-CNN 26) for discovering AD-associated genetic variants. In the biospecimen modality, a deep bearning-assistien spectroscopy platform [29 diagnoses AD by analyzing blood-based amyloid-beta and metabolite biomarkers. Regarding clinical data, a deep learning model [5] outperforms earlier machine learning techniques in classifying AD patients. However, since AD data inherently multimodal, methods based on single modality are suboptimal, missing the potential to leverage interactions between different modalities.
Multimodal Approach Across multiple fields, multimodal learning has become increasingly valuable for its ability to integrate and capture dynamics within and across different modalities, providing richer and more comprehensive representations of data. Approaches such as the Tensor Fusion Network (74), Multimodal Transformer [58], and Multimodal Adaptation Gate [3 highlight the effectiveness of combining multiple data sources. Recently, sparse mixture-offeepeers--bassee methods, such as [4, 8 19], have been introduced to enhance modality interactions, though these methods still relatively unexplored in the AD domain due the complexity of handling various modality combinations. In AD research, some works have emerged to leverage multimodal data, such as [46] and [59]. which integrated a deep learning framework that combines imaging, genetic, and clinical data, achieving superior AD staging accuracy. Another multimodal model [33], incorporating longitudinal and cross-sectional data, provided more accurate AD predictions. While multimodal AD studies have shown significant progress, the challenge of missing modalities, especially in the context of how effectively cope with modality combinations, remains largely underexplored.
3 Methods
3.1 Preliminaries and Notations
Why Sparse Mixture-of Experts? Given multimodal nature, we choose to utilize Sparse Mixture- of-Experts (SMoE) (55) due to its computational efficiency and its ability to handle multimodal data by effectively alleviating the gradient conflict optimization issue between modalities [50). To brietly introduce SMoE, Traditional Mixture-of-Fe offture-of--Epeets (MoE) models 23
28 9 770 evolved by incorporat- ing sparsity into their structure, optimizing computational efficiency and model performance. SMoE selectively activates only the most relevant experts for a given task, reducing overhead and improving scalability. This innovation is particularly beneficial in handling complex, figgh-dimensioall datasets across diverse applications. It has been widely used vision 54
40 16
18
62 69
49 and
language processing 335 30
78
79 25 fields, dynamically assigning different parts of the net-
work to specific tasks in classification tasks in digital number recognition (20) and medical signal processing (3). In this work, we further explore the use of SMoE to model arbitrary modality combinations and address the
or data modalities [32 Research has shown its effectiveness
missing modality scenario.
Notation. Formally, the SMoE consists of multiple experts, denoted as fi, ......, f where
[F] represents the total number of experts, and a router, R, which is responsible for the routing mechanism and sparsely selects the top-k experts. For given embedding token x, the router R
3

---

engages the top-k experts based on the highest scores obtained from softmax function with learnable gating function, g(-) (usually one or two layer MLP), and output R(x),, where denotes the expert
index. This process can be described as follows:
[
y = Σ R(x), fi(x),
R(x) = Top- K (softtma((((x)) k),
(I)
[v, if v is in the top k:
TopK(v, k) = ( otherwise.
3.2 Our approach: Flex-MoE
In this section, we present our novel algorithm, Flex-MoE, specifically designed to flexibly address the challenge of missing modalities in the multimodal domain. We start by sorting the samples based on their number of observed modalities. Following a modality-specifie encoder, we supplement the embeddings for missing parts via missing modality bank completion. This effectively manages missing modalities by learning embedding banks that capture the information specific to observed modality combinations. Next, Transformer coupled with an SMoE layer is employed. We introduce an expert generalization and specialization step to optimize modality utilization by fully leveraging samples with complete modalities and obtaining modality combination-specte samples with fewer modalities. A comprehensive illustration of Flex-MoE is provided in Figure 3 Throughout the details in the following section, while work is exemplified through the AD domain for predicting AD stages using four representative modalities-image clinical, biospecimen, and genetic-it important to note that Flex-MoB can be generalized any other multimodal domain.
knowledge through
(a) Foox Pramework
(c) Expert Generalization (Pall Modalities)
EIE
Bulkes
4C4
6G3
Tenwera
Samee Soki Laver
Moselty
Spedialuniceee 6O
CN
G-Racte
(b) Missing Modality Bank Completion
(d) Expert Specialization (Vew Modalities)
Mixsing Modillly
Climial Bnoaky
Seepadiva boodey
Genate
Gnoudyy
S.Ruee
The comprehensive illustration of our proposed methodology, Flex-MoE. (a) Overall framework of Flex-MoE. Given samples with diverse modality combinations, first sort the samples based on their number of available modalities in descending order, and then pass through the modality-specific encoder. (b) Each encoder is only trained with their available samples. For the missing embeddings, we introduce a missing modality bank containing learnable embeddings given the observed modality combination with their corresponding missing modality index. Equipped with this embedding, FMex-MoE passes through the Transformer where the FFN layer is replaced with Sparse MoE layer. Here, (c) full modality samples take charge of training generalized experts in balanced manner via G-router, then (d) the remaining few modality combinations further specialize the expert knowledge with S-Router, which fixes the top-1 1 gate as the corresponding observed modality combination expert. In this figure, top-2 selection of experts is illustrated as an example.
Figure 3:
3.2.1 Missing Modality Bank Completion
Given set of samples with their own modalities, it straightforward to pass them through modality- specific encoders, such 3D-CNN for MRI images. However, we are dealing with missing

---

modality scenario in multimodal data, where specific modalities are often missing based on their observed modality combinations. Thus, common to use padded or imputed inputs for the corresponding missing modalities in multimodal setting. This approach becomes troublesome when considering interactions between modalities. For instance, given batch with samples, some samples might have image, genetic, and clinical modalities, while others might be missing image and genetic modalities. In such cases, the image encoder and genetic encoder would take zero-padded or hypothesized imputed inputs derived from the missing samples, which are synthetic and of lower quality than the observed ones. This negatively affects the training of modality- sppecitt-speccifi encoders. Additionally, heavy imputation for each modality in multimodal setting increases time complexity, which is not desirable in clinical settings.
Given this situation, we propose training each encoder solely with observed samples to fully leverage the potential of the encoder by using only observed inputs. Unlike existing approaches, our design principle considers modality combinations to ensure robust and flexible training and effective handling of missing modalities. As illustrated in Figure [ each patient has diverse symptoms and personalized treatments, leading to variations in available modalities. For example, patients might lack image and genomic modalities (i.e., possessing only biospecimen and clinical modalities) due to various reasons such patient conditions, resource limitations, or specific clinical settings ST. Imputing these missing modalities must be handled within this context, rather than applying global learnable representative embedding for each modality without considering the observed environment.
Therefore, we propose a learnable missing modality bank. Given the number of modality combinations without fully observed scenarios, the total cases would be, given a modality set M (I, 1 (I-!) 21 1. The resulting missing modality bank can be defined as B E R22 By using this bank, the concatenated embedding of all modalities for patient i, h: |e ef 1 ef] would be represented as follows:
Encoder" (i)
if modality m observed in sample otherwise
e,"h = B Mmm.
Vm E
(2)
where c.th E Rd denotes the embedding from the corresponding modality m for patient i, and d is the hidden dimension. Here, the main idea of the missing modality bank completion is to supplement missing modalities from a predefined bank, ensuring robust data integration with observed ones. For example, if a patient lacks clinical data but has imaging, biospecimen, and genetic data, the observed modalities pass through their specific encoders. The missing clinical embedding is supplemented from the missing modality bank, indexed by the observed modalities (e.g.. B M. BIz.g.B).ol c).. By doing so, the encoder for each modality can be trained without encountering non-observed, incomplete input features. Then, move to the Transformer layer, where the FFN layer is replaced by an SMoE layer, a common approach in the SMoE domain 24 38 10).
3.2.2 Expert Generalization & Specialization
While adopting the SMoE backbone, it is important to note that our environment differs from concurrent SMoE studies, especially in terms of multimodal learning with missing modalities. In this context, choosing the most relevant tokens is challenging for experts, since the significance, i.e., the quality of input information, varies with the missing modalities. This significantly motivates us to take distinct approach from concurrent SMoE studies, where the input token is derived from fully observed scenarios. To address the unique challenges of the missing modality scenario, we propose modality combination-specine MoF. design. Specifically, we assign expert indices based on all possible modality combinations. For example, 'IGCB' is assigned as 0, 'IGC' as I, .... up to 'B' as 14. The remaining experts act as buffers, allowing the Router to select the most relevant top-k experts and activate them automatically. This approach leaves room for ficxibility and maintains the initial intuition of the MoE design.
Generalization It now becomes clear why the samples used for training Flex-Mot are sorted in descending order. Inspired by curriculum learning 63), where easy samples are presented first and more challenging samples appear later, we regard the level of difficulty as the number of missing modalities. We first train our SMoE layer with easy samples, where all modalities are fully observed. Using this intersection as a gold standard, we initially train all the experts in the MoE model. The procedure essentially follows the vanilla SMoE design as described in Equation I but with one key
5

---

difference: the input tokens consist only of inputs where all modalities are fully observed. Hence, we refer this router the Generalized Router, G-Router. This approach leverages the completeness of information in these samples, which should be fully utilized before specializing the experts in their respective areas. To ensure balanced activation of the experts initially, which will later specialize, we incorporate the load and importance balancing loss [55] which will later be exemplified in Equation
Specialization Once the experts are initially trained using fully observed samples, we aim to specialize each expert, which the key advantage of the MoE design. We leverage the remaining samples, which encompass diverse modality combination configurations. Each modality combination requires its own specialized expertise. For instance, samples with Image, Biospecimen, and Genetic data will have a corresponding expert index activated through the top-1 gating mechanism to fully utilize the specialized knowledge of that expert (i.e., expert 'IBG' in Figure To effectively specialize the modality combination-specinf experts, we propose a Specialized Router design, S-Router, which encompasses following technical novelties. First, to facilitate targeted expert selection when an input token is provided, we avoid manually replacing the selected routing policy with preferred choice post-hoc manner, which would stop the continuous gradient flow. Instead, we innovatively introduce a cross-entropy loss between the top- I expert selection and the targeted expert indices for each token by the S S-RRuter. Formally, this can be described as follows:
72
Coe
where MC(x,) denotes the modality combination index of given token x, in a total of n tokens. max (xx(S-Router(xx)) denotes the maximum prediction probability of the corresponding activated expert index, which corresponds the probability of the top-1 expert index. By comparing these two, the router is trained to activate the corresponding expert index for a given input token with a certain modality combination. Thus, the specialized knowledge inherent in specific modality combinations is naturally contained within the target expert.
Moreover, when calculating load and importance balancing loss [55], we specifically compute the loss for the remaining top-k-1 experts, as the top-1 selection is manually handled and thus considered biased rather than balanced. We aim for the selection of the remaining k-1 experts to occur in balanced manner, allowing interaction with other related modality combinations. Formarlly, can be expressed as follows:
Lxulance CV²
Σ importance,
CV2
Σ load,
(4)
N
where importance,
8(gie 0), We E Ctop-1 Σ Jio: load, Σ
(x) is the standard deviation of x, μ(x) is the mean of x, Die is the gate value for sample i with expert index e as discussed in Equation [ and 8(- >> 0) is an indicator function that is when the inner value is greater than O. E \ Crop-1 denotes the set of expert indices excluding the top-1 selected expert index. This ensures that the resulting MoE model not only retains global knowledge but also incorporates specialized expert knowledge tailored to specific modality combinations. During the inference phase, the specified expert index for a particular modality combination can be activated alongside other experts, enabling predictions to consider both the specific modality combination and intersections with other modalities.
where (v²
(a(w) [( ((())
Finally, the output embeddings of the Sparse MoE layer pass through 1-layer MLP prediction head to predict one of the three stages AD, i.e., Dementia, CN, or MCI. To further facilitate curriculum-learning approach, we first use warm-up epochs with sorted samples, followed by shuffled samples for the remaining epochs. This strategy enhances generalizability across diverse input samples, enabling better handling of variability during the inference phase.

---

4 Experiments
4.1 Experimental Setting
ADNI Dataset Alzheimer's Disease Neuroimaging Initiative (ADNI) is a landmark multimodal AD dataset that tracks disease progression and pathological changes, comprising of comprehensive imaging, genetic, clinical, and biospecimen data ((64), [67). The imaging data in ADNI includes magnetic resonance imaging (MR)) and positron emission tomogrpahy (PET). The genetic data includes a variety of genetic information, including genotyping data such as APOE genotyping and single nucleotide polymorphisms. The clinical data includes demographies, physical examinations, and cognitive assessments. Biospecimens such as blood, urine, and cerebrospinal fluid are also collected. ADNI has established standardized multi-center protocols and provides open access to qualified researchers, making it gold-standard resource in the field ([65), [66]). Before integrating all modalities, to address the initial missing data within each modality, we applied simple mean imputation (39] for each column. For more detailed data table with preprocessing steps for each modality, please refer to Appendix[A.]
MIMIC-IV Dataset We use the Medical Information Mart for Intensive Care IV (MIMIC-IV) database (27]. which contains de-identified health data for patients who were admitted to either the emergency department or stayed in critical care units of the Beth Israel Deaconess Medical Center in Boston, Massachusetts22. MIMIC-IV excludes patients under 18 years of age. We take a subset of the MIMIC-IV data, where each patient has at least more than 1 visit in the dataset as this subset corresponds to patients who likely have more serious health conditions. For each datapoint, we extract ICD-9 codes, clinical text, and labs and vital values. Using this data, we perform binary classification on one-year mortality, which foresees whether or not this patient will pass away in a year. We drop visits that occur the same time the patient's 's death. In order to align the experimental setup with the ADNI data, which does not contain temporal data, we take the last visit for each patient.
Baselines We compare the performance of Flex-MoE with various state-of-the-art baselines across modality-specific, e.g., image or genetic, and multimodal approaches. For the image-only modality, we first experimented with 3D MRI scans by utilizing a 3D CNN [17] and an architecture that combines 3D CNN and 3D CLSTM (68) To decrease computational complexity, we also extracted 2D slices from the 3D volumes. For 2D MRI scans, implemented the VGG architecture with pre-trained weights and applied layer-wist wise transfer learning (43), as well as a modified ResNet-15 18 employed a ResNet-34 based architecture to handle the high-dimensional genetic data [36.. In ADNI dataset, we further implemented domain speicfic baselines, such as auto-encoder and 3D CNN- -baed architecture that incorporates imaging, genetic, and clinical data [59] and a GRU-based architecture that considers imaging, genetic, clinical, and biospecimen data 33 Moreover, we include ShaeSpec [60]. which utilizes a spectral attention mechanism to emphasize important features across modalities, and mmFormer (76). which is based on transformer-basee multimodal fusion with an attention mechanism. For multimodal approaches in both ADNI and MIMIC-IV, we incorporate the recent FuseMOE [19] model, which directly integrates multimodal data through a mixture of experts strategy, as the most straightforward baseline. Additionally, we compare the following methods: MulT [57]. which captures cross-mes cross-modal interactions through cross-attention mechanisms; MAG [52) which fuses multimodal features by mapping them to an adaptation vector; TF 73), which combines multimodal embedding sub-networks and a tensor fusion layer; and LIMoE 44 which addresses training stability in multimodal learning using entropy regularization based on contrastive learning.
network (45) For the genetic-only approach,
Experimental Settings. To ensure fair comparison with other baselines, we used the best hyper- parameter settings provided in the original papers. If not available, we tuned the learning rate in le-3, 1c-4, 1c-5. the hidden dimension 64, 128, 256, and the batch size in 8, 16. For our proposed method, we searched the number of experts in 16, 32, and Top-k in 2, 3, 4. We set the coefficient of the sum of additional losses (importance and load balancing) combined with our cross-entropy loss to 0.01, scaling it within the task classification loss. For the dataset split, we chose 70% for training, with the remaining 30% split evenly between validation and test sets (15% each). to note that, to share the same inference space, where single and multimodal baselines should both be able to predict, we opted to choose the intersection as the test and validation sets. This means that during the training phase, the dataset can be incomplete. For the multi-modal baselines, if they had the ability to impute or interact with other modalities, we leveraged their methods. Otherwise, we used zero-padding to facilitate batch-wisc training. For single-mode modal and multi-moda basclines
important
7

---

that solely work on the intersection region, we filtered that data and used it during training. All experiments were conducted using NVIDIA A100 GPUs. Each experiment was run three times with different seeds to ensure reproducibility, and the results were averaged. The optimal hyperparameter settings for Flex-MoE can be found in Appendix A.2
Table I: Performance on ADNI dataset with ACC metric across different models and modality combinations, given the Image (I, RE)) Genetic (g. & ) Clinical (C, ), and Biospecimen (B, modalities. .MC denotes observed modality combination.
Modalitee
Dataset: ADNI MMtrier ACC
JMC
88
Shaßppr
mml ormer
Malf
FruMooF
Flax--oot
I.G
51.96
54.53
50.14
52.19
50.95
51.45
52.57
51.17
52,47
53.87
P.C.
91
52.85
$3.55
09.58 4::43
16
46.03
40.94
15.76
18.84
60.60
G.8
45.0
52.66
58.54
63.45
42.20 42:20
19.40
40.52222.0
16.88
52.91
63.59
0,0
46.63 10.32
63.68
10:05
59.00
12.53
47,71
11:43
39.65 12.30
44.54
40.15
43.95 .0.30
57.91
60.50
Z.5.C
55.12 10:30
54.72 Jn.23
49.30 2317
06,49 11:3
$4.05 11:00
60.97 L0.35
61.34 L0.11
53.50 -2:35
60.97 -1:32
63.21 11133
I.9.8
56.12 13:45
55.28 13:65
$2.45
10.33
07.15
15.43
$4.44
15:30
53.03 13.25
$4.15 13.35
53.97 -1:35
53.55
1100
62.28 1275
7.C.8
48.70 a1-01
49.85 01:00
64.85
6.C.6
11.00
13.33
06.38
1424
43.33
43.32 16.76
37.25
13.30
40.39
34.64
T,0.C,8
$8.92
Table 2: Performance MIMIC-IV dataset with ACC metric across different models and modality
combinations, given the Lab and Vital values (C, (C, ) Clinical Notes (N, ), and ICD-9 Codes (C, modalities. .MC denotes observed modality combination.
Modalities
Dataset: MIMIC-IV Metric: ACC
MC
TF
MulT
MAG
LIMoE
FuseMoE
Flex-MoB
N.N
60.05
57.96 27.2
62.72
12.36
63.80
60.50 13.82
76.14
10.73
C,C
64.13 #3.29
62.47
=2001
60.13
#1.97
64.89
63.31 #3.21
75.15
#0.55
N.C
60.97 #2.36
62.23
=2.1
59.41 #4.15
64.27
#4.05
64.77 =3.05
74.96
+1.00
C.N.C
63.11 12.17
64.62
10.44
62.87
12.50
61.61
12.37
63.90 1172
76.81
1.090
4.2 Primary Results
In Table andd Table 2, we provide a comprehensive comparison of Flex-MoE with various multi- modal baselines. We have the following observations: (1) Overall, Flex-MoE performs effectively in diverse multimodal settings, fully harnessing its potential as more modalities become available. This is supported by the large margin of improvement (7.6% and 11.07% over the best performing baselines, MAG and the most recent work FuseMoE, respectively, in full modality settings in Table D]. (2) Although the recently proposed FuseMoE [19] suggested its ability to handle missing scenarios, the lack of effective modality combination creates bottleneck in such AD domain, even performing worse when smaller number of modalities is used (FuseMoE performs better with three modalities than with full modalities), which is not optimal given the diverse missing modality scenarios. (3) the reliance on intersection data and the lack of consideration for how missing modalities relate to observed modality combinations have been overlooked. (4) Overall, the performance gain derived from Flex-MoE can be attributed its unique ability to cope with diverse modality combinations through missing modality bank, and its capability to fully harness the knowledge of samples via a generalization followed by specialization step for experts. For additional results on different metrics, such as Macro-F1 and AUC, please refer to Appendix A.3
Despite its specific characteristics in the AD domain 33 59
4.3 Effectiveness of Modality Combination Consideration
To validate the effectiveness of the two essential modules of Flex-MoE the missing modality bank and the unique SMoE design-under a missing modality scenario, we evaluate them followingly.
First, to evaluate the effectiveness of the missing modality bank introduced in Figure 3 (b), we assess whether it captures relevant embedding information given an observed modality combination. Specifically, we validate this by examining the inter-relationship between modalities, focusing
8

---

the consistency of how the missing modality bank handles missing information. In Figure 4 we measure the cosine similarity between observed modality combinations. The key observation that (1) with more overlapping modality combination, it tends to share more similar embedding information. This is evident in the left side of Figure [4] where the full modality scenario (ICBG) shows higher similarity with ICB and CBG (0.56 and 0.58, respectively) compared to IC and CB (0.46 and 0.45). The clinical modality (C) is most similar across combinations, which aligns with the dataset characteristic that clinical data is present in all input combinations, shown in Figure 2 On the right side of Figure 4 the similarity between missing modalities is shown. When (2) modality G is missing, it is more similar to the cases where C and B missing compared to I, suggesting that certain missing modalities share more commonality in how they are handled by the model. This insight underscores the importance of careful consideration when modeling missing modalities and demonstrates how the missing modality bank effectively captures necessary embedding information in terms of modality combination.
Furthermore, in Figure | we show the acti- vation ratio of input modality combinations across each expert index, i.e., possible modal- ity combinations. We observe two key find- ings: (1) Thanks to expert generalization us- ing full modality samples (BCGI), general- ized knowledge is distributed across all ex- perts. This allows each expert to leverage commonly shared knowledge while activating the most relevant inputs for the downstream task. (2) Expert specialization enables each expert acquire specialized knowledge. For instance, in the case of the BCG expert, the two most activated input tokens were BCGI and its corresponding token, BCG. Similarly, for the BCI and CI experts, they not only pos- sess general knowledge from BCGI but also hold their own specialized knowledge from BCI and CI, respectively, to effectively handle these inputs. This demonstrates that when certain modality combination is provided, the top-1 expert is successfully selected, allow- ing it to supplement its specialized and neces- sary information. Overall, these routing exper- contains
Observed Modallity Combinstion
Missiing Madality
IC80
0.55
0:46
0:36
0,50
0.45
3M
0:49
0.52
10
0.49
0.51
0.60
0.64
0.56
0.52
0:45
0.47
0.54
0.50
0.65
0.01
CBG
0.38
0.46
0.6
0.47
0.42
0.45
0.52
0.54
ICB0
10
CBG
Figure 4: Cosine similarity between observed modal- ity combination and missing modality, correspond- ing to row and column in missing modality bank.
Input Modalliy Combination
Activation Ratio per Expert
102
82-
Percortage
65-
23
iments demonstrate that both globally generalized and locally expert- specific knowledge, achieved by leveraging samples with both full and fewer modalities.
Expert Index
Figure 5: Modality combination activation ratio.
4.4 Comprehensive Evaluation
Ablation Study. In this section, we investigate the crucial components that contribute most positively to the performance gain of Flex-MoE. From Table we observe that (1) when both expert specialization and generalization are absent, the performance drop is most severe. Additionally, (2) the performance decline in the embedding bank negatively affects overall performance, indicating that the missing modality bank combined with expert generalization and specialization is crucial for handling missing modality scenarios. Furthermore, (3) the sorting based on descending order appear most effective as the expert generalization occurs first withi the full modality samples.
Sensitivity Study. In Figure g we also varied the hyperparameters used in this study. We examined the number of experts, number of SMoE layers and top-k selection. We found that (1) employing many experts does not always guraantee higher performance compared to its increase in complexity, showing using experts apeear to suitable choice to equip fine-grained specialized knowledge. (2) Using sing a single layer of the SMoE was most effective, as stacking more layers or adding a Transformer block caused an overload to parameter learning. Additionally, (3) compared to the commonly used t gating network in concurrent SMoE studies, we found that top-4 selection was

---

Table Ablation study of Plex-Moo
#.rSMElam Spp
ACC
Fl
ACC 0
ArC
Flex-MoE
66.11
64.73
w/o ES
62.75
60.79
w/o (ES EG)
62.49
60.07
Figure 6: Sensitivity analysis of Flex-MoE. The hyper-
w/o embedding bank
63.87
62.48
parameters include the number of experts, the number
w/o sorting random
62.65
60.70
of SMoE layers and Top-k expert selection. For the experiment, ADNI dataset with full modalities is used.
w/o sorting ascending
63.87
62.22
the most effective. This is because manually assigning the top-1 expert index to the target modality combination leaves more room for better harmonization with the SMoE design.
Complexity Study. In Table 4 further verify the benefits of utilizing the SMoE design in terms of mean time per iteration, GFLOPs, and the number of parameters compared to the baselines. We observed the following: (1) Compared recent baseline FuseMoF,, Flex-MoE achieves notable efficiency gains (e.g., 22.74%, 1.15%, and 19% gain in mean time, GFLOPs, and # of parameters, respectively.) while delivering higher performance.
(2) Although TF appears to be lightweight design in the I, G and I, G, C settings, it trades off computational efficiency with significantly lower performance compared to Flex-MoE: MoE. (3) Notably, as the number of modalities increases, existing models tend to become more complex in terms of GFLOPs and the number of parameters to manage the additional complexity. However, Flex-MoE remains robust and efficient, maintaining higher performance due to its effective use of sparsely activated experts, brought by the SMoE framework.
Modality
Metric
MulT
MAG
LIMOE
FuseMoE
Flex-MoB
Mean Time (8) (1)
12.40
12.85
11.64
12.65
18.68
12.73
GFLOPs (L)
$9.05
59.24
$9,06
$9.24
59.74
$9.06
I,9
of Parameters
33,370,898
37,343,683
36,454.595
37.344,707
264,680,387
36,516,807
Accuracy (f)
59.94 10.40
60.32 20.95
59.94 #1.01
59.29
40.95
60.41 20.87
61.08 20.78
Mean Time (L)
13.80
23.28
14.55
14.64
18.68
14.53
I,9.C
GFLOPs (L)
59.05
59.59
59.06
59.32
59.74
59.06
# of Parameters (L)
34,424,162
88,985
36,504,643
37, 37,960.643
340,923 (20,9299475
36,685,51
Accuracy (f)
$4.06 11.98
60.97 10.05
61.34 10.61
53.50 12:25
60.97 11.32
63.21 11.73
Mean Time (s) ($)
15.83
38.70
16.04
17.96
20.71
16.00
GFLOPs (1)
59.39
60.12
59.06
59.41
$9.76
59.07
I,9.C,B
# of Parameters (1)
119,483,922
46.409.667
36,504,643
38,638,531
340,929,475
36.916,167
Accuracy (t)
57.24 +0.35
58.82 20.82
61.44 11
55.18
#2.42
59.52
66.11 #1.14
Table 4: Complexity comparsion of mean time, GFLOPs, and # of parameters in ADNI dataset.
Conclusion
While multimodal learning brings new opportunities and challenges across various domains, including medical fields, existing approaches struggle to handle arbitrary modality combinations, especially in missing modality scenarios, often relying on single modalities or complete datasets. In this work, we propose a flexible multimodal learning framework, Flex-MoE, capable of managing arbitrary subsets of available modalities. By carefully considering modality combination, it leverages a learnable embedding bank to capture missing modality information and utilizes unique SMoE design to enhance expert generalization and specialization. Extensive experiments on the representative ADNI and MIMIC-IV datasets validate its effectiveness in handling diverse modality combinations. Future work includes extending the framework to explore the scaling laws of available modalities, which in turn presents numerous modality combinations, offering significant room for further improvement.
Societal Impact and Limitation: The proposed algorithm has the potential to significantly improve early diagnosis and treatment outcomes for patients, reducing the burden healthcare systems. However, its effectiveness can be limited by the availability of comprehensive and high-quality patient data, and there may be challenges in integrating this tool into existing clinical workflows.
This work is supported by RF1-AG063481, R01-AG071 174, Gemma Academic Program, and OpenAl Researcher Access Program.
Acknowledgement
10

---

References
[1] A. Abbas and Y. Andreopoulos. Biased mixtures of experts: Enabling computer vision inference under data transfer limitations. IEEE Transactions Image Processing, 29:7656- 2020.
[2] K. Ahmed, M. H. Baig, and L. Torresani. Network of experts for large-scale image categorization.
In European Conference on Computer Vision, pages 516-532. Springer, 2016.
[3] R. Aoki, F. Tung, and G. L. Oliveira. Heterogeneous multi task learning with expert diversity, CoRR, abs/2106. 10595, 2021. URL Utt.httpa://arxi..rrr
2106 10595
[4] A. Association et al. 2016 alzheimer's disease facts and figures. Alzheimer's 's & Dementia, 12 (4):459-509) 2016.
[5] Avila-Jiménez, V. Cantón-Habas, M. del Pilar Gurrera-Gonzáie M. Rich-Ruiz, and S. Ven- tura. A deep learning model for alzheimer's ' disease diagnosis based on patient clinical records. Computers Biology and Medicine, 169:107814 14, 2024.
[6] Baltrušaitis, C. Ahuja, and L.-P. Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):223 443, 2018.
[7] Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pages 41 41-48, 2009.
[8] B. Cao, Y. Sun, P. Zhu, and Q. Hu. Multi-modal gated mixture of local-to-global experts for dynamic image fusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23555-23564, 2023.
[9] K. Chen, L. Xu, and H. Chi. Improved learning algorithms for mixture of experts multiclass classification. Neural networks, 1 12(9): 1229-12522222 1999.
[10] T. Chen, Z. Zhang, A. Jaiswal, S. Liu, and Z. Wang. Sparse dense and self slimmabieelle transformers, 2023.
the new dropout: Scaling
[11] Z. Chen, Y. Shen, M. Ding, Z. Chen, H. Zhao, E. G. Learned-Miller, and C. Gan. Mod-squad: Designing mixtures of experts as modular multi-task learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11828-12 11837, June 2023.
[12] J. M. Choi, M. Christman, and R. Sengupta. Personalized video relighting with an at-home light stage, 2024. URL
[13] Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional 2018
transformers for language understanding. arXiv preprint
[14] J. Doshi, G. Erus, Y. Ou, S. M. Resnick, R. C. Gur, R. E. Gur. T. D. Satterthwaite, S. Furth, C. Davatzikos, A. N. Initiative, et Muse: Multi-atlas region segmentation utilizing ensembles of registration algorithms and parameters, and locally optimal atlas selection. Neuroimage, 127: 186-195, 2016.
[15] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL htt.hhtpsss/oppeeviieee http://ppnnevieeeeeeeeeeoereeoee
[16] D. Eigen, M. Ranzato, and I. Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.
[17] Esmaeilzadeh, I. Belivanis, K. M. Pohl, and E. Adeli. End-to-end alzheimer's disease diagnosis and biomarker identification. In Machine Learning in Medicul Imaging: 9th Inter- national Workshop, MI.I 2018, Held Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings 9, pages 337-345 Springer, 2018
11

---

[18] S. Gross, M. Ranzato, and A. Szlam. Hard mixtures of experts for large scale weakly supervised vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 6865-6874 2017.
[19] X. Han, H. Nguyen, C. Harris, N. Ho, and S. Saria. Fusemoe: Mixture-of-expert transformers for fleximodal fusion. arXiv preprint arXiv:2402.032266 2024.
[20] H. Hazimeh, Z. Zhao, A. Chowdhery, M. Sathiamoorthy, Y. Chen, R. Mazumder, L. Hong, and E. H. Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021. NeurIPS 2021, December 6-14, 2021, virtual. pages
2021. URL.tttpa hatttss:/ppooeeedingg wewpiiinng.n.nuiiiee neurips... cetrripsccceeeeeee patpppppppp//aaaaapeeappaa
f5ac21cd0ef 1888 10155511eeb5a aeeaaaebbaaaaa Abstract. html
[21] C. R. Jack and D. M. Holtzman. Biomarker modeling of alzheimer's disease. Neuron, 80(6): 1347-1358, 2013.
[22] C. R. Jack Jr. D. A. Bennett, K. Blennow, M. C. Carrillo, B. Dunn, S. B. Haeberlein, D. M. Holtzman, W. Jagust, F. Jessen, J. Karlawish, et al. Nia-aa research framework: toward biological definition of alzheimer's 's disease. Alzhemmer's & Dementia, 14(4):535-562, 2018.
[23] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
[24] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud,
L. Saulnier, M.-A. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao,
T. Gervet, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mixtral of experts, 2024.
[25] H. Jiang, K. Zhan, J. Qu, Y. Wu, Z. Fei, X. Zhang, Chen, Dou, X. Qiu, Z. Guo, et al. To- wards more effective and economic pparsely-activann model. arXiv preprint arXiv:2110.07431, 2021.
[26] T. Jo, K. Nho, P. Bice, A. J. Saykin, and A. D. N. Initiative. Deep learning- based identification of genetic variants: application alzheimer's disease classification. Briefings in Bioinformatics, 23(2)bbbac022, 2022.
[27] A. E. Johnson, T. J. Pollard, S. J. Berkowitz, and et al. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data, (317):1-8, 2019. doi: 10. 10.1038/41597-019.032222 941597-019-0322-0
URL https://doi..nno //g////o..r//....////// /000 1038/
[28] M. I. Jordan and R. A. Jacobs. Hicrarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181-214, 1994.
[29] M. Kim, S. Huh, H. J. Park, S. H. Cho, M.-Y. Lee, S. Jo, and Y. S. Jung. Surface-functionalized sers platform for deep bearning-assistee diagnosis of alzheimer's disease. Biosensors and Bioelectronics, page 116128, 2024.
[30] Y. J. Kim, A. A. Awan, A. Muzio, A. F. Cruz-Salinas, L. Lu, A. Hendy, S. Rajbhandari, Y. He, and H. H. Awadalla. Scalable and efficient moe training for multitask multilingual models. CoRR, abs/2109. 10465, 2021. URL https:/ts:///rtt /rrxiv.ivvv , org//bs//1100 2109. 10465
[31] T. N. Kipf and M. Welling, Semi-superviised classification with graph convolutional networks, 2017. URL WRLLhttps: http:///axivv..ro org/abs//600 1000b//9900909000777
[32] S. Kudugunta, Y. Huang, A. Bapna, M. Krikun, D. Lepikhin, M. Luong, and O. Firat. Beyond distillation: Task-level mixtere-of-experts for efficient inference. In M. Moens, X. Huang, L. Specia, and S. W. Yih, editors, Findings of the Association for Computational Linguistics; EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pages 3577-3599, Association for Computational Linguisties, 2021. doi: 10. 18653/v1/2021. findings-emnll... URL URLhttts : www.oo.g///o..... 11w.:/1..11111111111 findings emnlp 304
12

---

[33] G. Lee, K. Nho, B. Kang, K.-A Sohn, and D. Kim. Predicting alzheimer's 's disease progression using multi-modal deep learning approach. Scientific reports, 9(1):1952, 2019.
[34] Lee, S. Yun, Y. Kim, Chen, M. Kellis, and C. Park. Single-cell ma sequencing data imputation using bi-level feature propagation. Briefings in Bioinformatics, 25(3):bbae209, 2024.
[35] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In 9th Inter- national Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. Jttphttts:///ppeereree
XHHmbb
[36] L. Y. Huang, Y. Han, and J. Jiang. Use of deep learning genomics to discriminate alzheimer's disease and healthy controls. In 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 5788-579) IEEE, 2021.
[37] P. P. Liang, Y. Lyu, X. Fan, Z. Wu, Y. Cheng, J. Wu, L. Chen, P. Wu, M. A. Lee, Y. Zhu, R. Salakhutdinov, and L. Morency. Multibench: Multiscale benchmarks for multimodal representation learning. In NeurIPS Datasets and Benchmarks, 2021.
[38] B. Lin, Z. Tang, Y. Ye, J. Cui, B. Zhu, P. Jin, J. Huang, J. Zhang, M. Ning, and L. Yuan. Moe-Ilava: Mixture of experts for large vision- vision-language models, 2024.
[39] R. J. Little and D. B. Rubin. Statistical analysis with missing data, volume 793. John Wiley & Sons, 2019.
[40] Y. Lou, F. Xue, Z. Zheng, and Y. You. Cross-token modeling with conditional computation. 2021.
arXiv preprint
[41] J. Ma, Z. Zhao, X. Yi, J. Chen, L. Hong, and E. H. Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts In Y. Guo and F. Farooq, editors. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19.23, 2018, pages 1930- 1939. ACM, 2018. doi: 10.1145/3219819:322000
//LLtttps://doi. http::////ooo.orr////// 10. 111111////1111 3220007 3220007
[42] F. Márquez and M. A. Yassa. Neuroimaging biomarkers for alzheimer's ' disease. Molecular neurodegeneration, 14(1):21, 2019.
[43] A. Mehmood, S. Yang, Z. Feng, M. Wang, A. S. Ahmad, R. Khan, M. Maqsood, and M. Yaqub. transfer learning approach for early diagnosis of alzheimer's 's disease on mri images. Neuro- science, 460:43-52, 2021.
[44] B. Mustafa, C. Riquelme, J. Puigcerver, R. Jenatton, and N. Houlsby. Multimodal contrastive learning with limoe: the language-iman image mixture of experts. CoRR, 0ttp/22066022770 2022. doi: 11.48550axxi..22260077777
https://///oi org/10 48550/arXiv wwwp:xiii.........
[45] M. Odusami, R. Maskeliänas, R. Damaševičius, and T. Krilavičius. Analysis of features of alzheimer's disease: Detection of early stage from functional brain changes in magnetic resonance images using finetuned resnet18 network. Diagnostics, 11(6):1071, 2021.
[46] M. Odusami, R. Maskeliunas, R. Damaševičius, and S. Misra. Machine learning with multi- modal neuroimaging data to classify stages of alzheimer's disease: systematic review and meta- anal-is Cognitive Neurodynamics, pages 1-20, 2023.
[47] Y. Ou, A. Sotiras, N. Paragios, and C. Davatzikos. Dramms: Deformable registration via attribute matching and mutual-saliency weighting. Medical image analysis, 15(4):622-639) 2011.
[48] A. Papassotiropoollss, M. Fountoulakis, T. Dunckley, D. A. Stephan, and E. M. Reiman. Genet- ics, transcriptomics, and proteomics of alzheimer's 's discase. Journal of Clinical Psychiatry, 67 (4):652, 2006.
[49] Pavlitskaya, C. Hubschneider, M. Weber, R. Moritz, F. Huger, P. Schlicht, and M. Zollner. Using mixture of expert models to gain insights into semantic segmentation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 342-343, 2020.
13

---

[50] J. Peng, K. Zhou, R. Zhou, T. Hartvigsen, Y. Zhang, Z. Wang, and T. Chen. Sparse moe as new treatment: Addresssing forgetting, fitting, learning issues in multi-modal multi-task learning. 2024. URL URL htt.htttps::/ppererviii
net/ heet/foruumtc-ebbRyypeeeu
[51] R. C. Petersen, P. S. Aisen, L. A. Beckett, M. C. Donohue, A. C. Gamst, D. J. Harvey, C. Jack Jr, W. J. Jagust, L. M. Shaw, A. W. Toga, et al. Alzheimer's 's discase neuroimaging initiative (adni) clinical characterization. Neurology, 74(3):201-209, 2010.
[52] W. Rahman, M. K. Hasan, S. Lee, A. Bagher Zadeh, C. Mao, L..PP Morency, and E. Hoque. Integrating multimodal information in large pretrained transformers. In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, editors, Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 2359-2369, Online, July 2020. Associa- tion for Computational Linguistics. doi: 1116663//122222.cc-mmmiiia //aclanthology. htgpe//olgooooooooooooooo wwwgg2220.cco acl main. 214
URL https
[53] W. Rahman, M. K. Hasan, S. Lee, A. Zadeh, C. Mao, L.-P. Morency, and E. Hoque. Integrating multimodal information in large pretrained transformers. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2020, page 2359. NIH Public Access, 2020.
[54] C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. S. Pinto, D. Key- sers, and N. Houlsby. Scaling vision with sparse mixture of experts. In M. Ran- zato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Ad- vances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 8583-8595, 2021. https://///
her.ssnnurrs.p. cc..ccccpppe hwpppaaarr//// ppaper/2011/hsh//
[55] N. Shazeer, A. Mirhoseini, Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outra- geously large neural networks: The sparsely-gated mixxture-of-expeee layer. arXiv preprint arXiv:1701.06533. 2017.
[56] X. Tang, J. Zhang, Y. He, X. Zhang, Z. Lin, S. Partarrieu, E. B. Hanna, Z. Ren, H. Shen, Yang, et al. Explainable multi-task learning for multi-modality biological data analysis. Nature Communications, 14(1):2546, 2023.
[57] Y.-H. H. Tsai, S. Bai, P. P. Liang. J. Z. Kolter, L.-P. Morency, and R. Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In A. Korhonen, D. Traum, and I.. Marquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 6558 6569, Florence, Italy, July 2019. Association for Computational Linguistics, doi: 10.18653/v 1ttp:///PP99/ 1/1111 1656. URL https:///
1656
[58] Y.-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, and Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for computational linguisties. Meeting, volume 2019, page 6558. NIH Public Access, 2019.
[59] Venugopalan, L. Tong, R. Hassanzadeh, and M. D. Wang. Multimodal deep learning models for early detection of alzheimer's disease stage. Scientific reports, 11(1):3254, 2021.
[60] H. Wang, Y. Chen, C. Ma, J. Avery, L. Hull, and G. Cameiro. Multi-modal learning with missing modality via shared- shared-specifer specific feature modelling, 2024. URL httpt:///axxx org/ abs/2307 14126.
[61] Wang, A. Ma, Y. Chang, J. Gong, Y. Jiang, R. Qi, C. Wang, H. Fu, Q. Ma. and D. Xu. segnn is a novel graph neural network framework for single-cell rna-seq analyses. Nature communications, 18(1) 2021.
[62] X. Wang, F. Yu, L. Dunlap, Y.-A. Ma, R. Wang A. Mirhoseini, T. Darrell, and J. E. Gonzalez. Deep mixture of experts via shallow embedding. In Uncertainty in artificial intelligence, pages 552-562. PMLR, 2020.
14

---

[63] X. Wang, Y. Chen, and W. Zhu. A survey on curriculum learning. IEEE transactions on pattern analysis and machine intelligence, 44(9):4555 4576, 2021.
[64] M. W. Weiner, P. S. Aisen, C. R. Jack Jr, W. J. Jagust, J. Q. Trojanowski, L. Shaw, A. J. Saykin, J. C. Morris, N. Cairns, L. A. Beckett, et al. The alzheimer's s disease neuroimaging initiative: progress report and future plans. Alzheimer's & Dementia, 6(3):202-211, 2010.
[65] M. W. Weiner, D. Veitch, P. S. Aisen, L. A. Beckett, N. J. Cairns, R. C. Green, D. Harvey,
C. R. Jack, W. Jagust, E. Liu, et al. The alzheimer's disease neuroimaging initiative: a review of papers published since its inception. Alzheimer's 's & Dementia, 9() 2013.
[66] M. W. Weiner, D. P. Veitch, P. S. Aisen, L. A. Beckett, N. J. Cairns, J. Cedarbaum, M. C. Donohue, R. C. Green, D. Harvey, C. R. Jack Jr, et al. Impact of the alzheimer's 's disease neuroimaging initiative, 2004 2014. Alzheimer's & Dementia, 11(7):855-884 2015.
[67] M. W. Weiner, D. P. Veitch, P. S. Aisen, L. A. Beckett, J. Cairns, R. C. Green, D. Harvey, C. R. Jack Jr. W. Jagust, C. Morris, al. The alzheimer's 's disease neuroimaging initiative 3: Continued innovation for clinical trial improvement. Alzheimer's 's & Dementia, 1355)5611-57, 2017.
[68] Z. Xia, G. Yue, Y. Xu, C. Feng, M. Yang, T. Wang, and B. Lei. A novel end-to-end hybrid network for alzheimer's disease detection using 3d enn and 3d clstm. In 2020 IEEE 17th international symposium on biomedical imaging (ISBI), pages 1-4. IEEE, 2020.
[69] B. Yang. Bender, Q. V. Le, and Ngiam. Condconv: Conditionally parameterized con- volutions for efficient inference. Advances in Neural Information Processing Systems, 32, 2019.
[70] S. E. Yuksel, J. N. Wilson, and P. D. Gader. Twenty years of mixture of experts. IEEE Transactions on Neural Networks and Learning Systems, 2377):1177 2012. doi: 10. 1099 TNNLS
[71] S. Yun, K. Kim, K. Yoon, and C. Park. Lte4g: Long-tail experts for graph neural networks. In Proceedings of the 31st ACM International Conference on Information and Knowledge Management, CIKM '22. ACM, Oct. 2022. doi: 10.1145//5118888355733 URL http
7/dx. orooo/////// 10. 1145/3511808 3557381
[72] S. Yun, J. Peng, A. E. Trevino, C. Park, and T. Chen. Mew: Multiplexed immunofluorescencce image analysis through an efficient multiplex network, 2024. URL/attps///axii org/ /bs/
2407.17857.
[73] A. Zadeh, M. Chen, S. Poria, E. Cambria, and L.-P. Morency. Tensor fusion network for multimodal sentiment analysis. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1103- 1114, Copenhagen, Denmark, Sept. 2017. Association for Computational Linguistics. doi: 10. 10.18653/w1/D17-11155 URL htl.itttss//acaaahoolloo htw.e//aaaahooooooooo////----1-- org/D17-1115
[74] A. Zadeh, M. Chen, S. Poria, E. Cambria, and L.-P. Morency, Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv: 07250, 2017.
[75] Y. Zhang and D.-Y. Yeung. Multi-task learning in heterogeneous feature spaces. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pages 574-579, 2011.
[76] Y. Zhang, N. He, J. Yang, Y. Li, D. Wei, Y. Huang, Y. Zhang, Z. He, and Y. Zheng. mm- former: Multimodal medical transformer for incomplete multimodal learning of brain tumor segmentation, 2022. URL hothhttps://// //tpxv///ivvv///a www.xiiiiorggoo org/ ab//////////2//////// 02425)
[77] Z. Zhang, Y. Lin, Z. Liu, P. Li, M. Sun, and J. Zhou. Moefication: Conditional computation of transformer models for efficient inference. arXiv preprint arXiv:2110.01786, 2021.
[78] Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. Dai, Z. Chen, Q. Le, and J. Laudon. Mixture-of-experts with expert choice routing. arXiv preprint arXiv:2202.09360 02922..90 2022.
[79] S. Zuo, X. Liu, J. Jiao, Y. J. Kim, H. Hassan, R. Zhang, J. Gao, and T. Zhao. Taming sparsely activated transformer with stochastic experts. In International Conference on Learning Representations, 2022. URL httpss://ppereviiee www.://.nnnnnoonenteeeeeee Btt.:rmmmmmmmmm02288888880 B72HXs80q4
15

---

A Appendix
A.1
Detailed Data Preprocessing in ADNI
Name
Data
Colmun Examples
Missin Rate
Imzaee
Procesed Tl-wzichol IMRI
(10353, 91, 109, 91)
NIA
N6A
ADNI cheow DJ. fewwed, OSTLONI
(187, N67)66)
1133445011 'OSA-si 16587990
1.45
ADNL 3 Forwanl.Bi
(432, 567586)
Genomuss
ADNL_OO LOO__RRAA_AAAOA OWWA_ssL.oii
(361, 567386)
nJ82004761", 'n386134241'
ADNIS PLINK Pinal
(57), 967886
03131992 'NOWSNIST
ADNIX PLINK FINAL, 2nd
(325, 567)
200510-977 '118613644"
0.46
MEDHIST 2004A
(3053.80)
MISOURCEE; 'MHPSYCHH, 'MHENEURL; MEEHEAA
13.29
NEUROEXM BEBBOEXXXLLMMM220A......
(3873.25)
NXTREMOR' 'NXCONSCT" NNNNFRIE: 'XXMOTRR
13.71
Chinicll
PIDEMOG L9Muy 2024,0V
(5073.77)
PIWURK. "PINIIT: :PIRIYR: 'PTHOME'
06,08
RECCMEDS, RECCMEESS_DMMay2222.oo
(56622.29)
"CMEVNUM" 'OMIGIN"
35.40
VITALS, 204.
(1538122))
VSHEIGHT, VVSHTUNIT. 'VSBPSYS', "VSBPDIA'
20,88
Bisopacinen
APOERES 2004
(2737,
APTESTDT: AIRRNNI APGINE, 'APVOLUME
UPENNHIOMK
2004.2 (M74,
ARRTAII,
Table 5: Summary of data files with their respective shapes, column examples, and missing rates.
Image Modality We first performed magnetic field intensity inhomogeneity correction to ensure that the MRI images are uniform and reliable. Then, we used a method called MUSE (Multiatlas Region Segmentation Utilizing Ensembles of Registration Algorithms and Parameters) to segment the gray matter tissue, which was the focus of our study 14] This process involves using multiple atlases and selecting the most optimal ones to accurately extract region-of- aegiion-of-interee of-interest values from the segmented gray matter tissue maps. Next, voxelwise regional volumetric maps for each tissue volume were generated by spatially aligning skull-stripped images to template residing in the Montreal Neurological Institute (MNI) space using a registration method [47]
Genetic Modality We collected SNP (single nucleotide polymorphisms) data from the ADNI 1, GO/2, and studies. First, SNP data from the different phases of these studies were aligned the same reference build using Liftover /ttps:///ifftt
broadinstitute org/ Specifically, all SNP data were converted to NCBI build 38 (UCSC hg38). After liftover, we merged the studies into a unified dataset. Next, linkage disequilibrium (LD) pruning, with parameters (50, 5, 0.1), was applied filter out SNPs that were highly correlated with others. Here, the parameters represent the window size (50), the step size (5), and the r-squared threshold 10. The SNP data contains values of 0, 1, 2.
files provided by ADNI. The file UPENNBIOMK ROCHE ELECSYS 09May2024 csv was used for Total Tau and Phosphorylated Tau data. The file APOERES 09Mayy024.css was used for ApoE genotype data. For numerical data, we applied a MinMax scaler to scale the values to a range of -1 to 1. For categorical data, we used one-hot encoding. For the missing values, we imputed the mean value for numerical columns and the mode for categorical columns.
Biospecimen Modality We extract biospecimen data from the following
Clinical Modality We extracted clinical data from the following files provided by ADNI: MEDHIST MEDHIITTO0May2000 (00.0y9924.24sy NEUROEXM NEUROEXM_0MMay2222 09Mayy024.css, PTDEMOG M0M000GG0MMMy222corr REC- CMEDS 09May2024.cs) VITALS 09May2024.csv. During the preprocessing of this clinical data, we excluded the columns "PTCOGBEG," 'PTADDX;" and 'PTADBEG' as they contain information directly related Alzheimer's Disease diagnosis. For numerical data, we applied MinMax scaler to scale the values to a range of 11 to 1. For categorical data, we used one-hot encoding. For the missing values, we imputed the mean value for numerical columns and the mode for categorical columns.
16

---

A.2 Hyperparameter Setting
Table 6: The hyperparameter setup for Flex -MoE
ADNI
MIMIC-IV
I. G.G.C.B
C.N.C
Leurning rate
0.0001
0.0001
Experts
32
# of SMoE layers
Top-K
3
Training Epochs
50
50
Warm-up Epuchs
Hidden dimension
128
128
Batch Size
8
# of Attention Heads
4
A.3 More Primary Results
Table 7: Performance ADNI dataset with Macro F1 metric across different models and modality combinations, given the Image (I, (I,8)), Genetic (G, ), Clinical (C, ), and Biospecimen (B,
modalities. MC denotes observed modality combination.
Modalitiss
Tuibest: ADNI Metric: Marro F1
MMC
520
33
ShaaSpex
mnnFormer
TF
MulT
MAG
LIMeB
FissMoFF
Plex-Mot
12:03
G.e
50.54
39.45
17.85
29:35
29.57
27.49
6.8
31.21
35,63
2982
12.41
28:29
20:91
20.91
61.65
0,8
14.28
63.46
55.46
31.00
29:57
35:22
37.20
30:36
29.31
21
Z.G.C
46.59 25.
53.86
23.54
$8.59 22.13
26,88 11:30
56.31 10.00
01 10.31
01 21.34
51.35 21.35
61.30 21.07
61:98 2104
I.0.8
55.46 42:45
$4.82 64.14
51.50 21.11
28.19
44:3
$3.70 62.01
$2.45
+1.35
$2.54
-1.5
$2.80 m1.92
52.75 m181
59.45
-131
Z.C.B
25.53 14.03
61.34
12.03
$1.50
27.87
1118
$2.64 13:00
50.14
10.95
$2.02
-2:22
$2.41
-1.11
50.61 -0.07
61.60
8.C.8
26.00
25.09 km
40.40 20.00
pm
21
2013
Z.O.C.8
20:94
Table 8: Performance on ADNI dataset with AUC metric across different models and modality
combinations, given the Image (I, R.) Genetic (g. ) Clinical (C, ) and Biospecimen (B, modalities. MCC denotes observed modality combination.
Datase: ADNI
.MC
BR
668
ShaSpp
mmn ormmer
Plax--oft
72.85
11.37
71.65
11.86
7.8
57.02
68.56
68:44
71
12.43-2144
12:43
21
70.59-11
77.88
D.C.
69.24
66.46
71.38
65.50
47.57 47.57 21:0
61.17
22.00
51.14-01
49.23-1.x
78.34
6.B
48.91 15:97
09.08 13.55
75.24 12:11
64.89 11:26
51.32 12:30
53.53 10.30
51.36-1.17
51.82
51.82
79.24
0,8
38.41 25.55
79.53 20.34
78.75
20.32
03.25 11.00
49.82 12/0
04.35 12.90
30.20:1.0
48.29 -3.11
43.82
79.65
Z.G.C
69.07 13.39
76.05 10.95
56.70 14.55
66.35
13:66
74.24 Le02
71.87 10.5
72.77 11.21
70.98 1.35
71.34 .053
79.55 1.0
I.G88
21.31
60.79
21:31
63.91
71.70-1-
Z.C.8
01.31
66.09
72.04 s00
01.33
R.C,8
63.49
24
51.75
2,6.C,8
69.44
73.93
71.09
71,16-1.1
Table 9: Performance on MIMIC-IV dataset with Macro Fl metric across different models and
modality combinations, given the Lab and Vital values (C, M, Clinical Notes (N, B), and ICD-9
Codes (C. modalities. MC denotes observed modality combination.
Modalities
Dataset: MIMIC-IV Metric: Macro F1
MC
TF
MulT
MAG
LIMoE
FuseMoE
Flex MoE
E.N
50.81 #0.47
50.61
#2.77
$3,46
26
55.19 #1.62
52.79
#1.32
$1.29
=1.53
C,C
55.09 11.20
56.33
41.00
54.07
20.98
57.32
10.52
54.78
20.01
53.85
=1.43
N.C
54.37 40.41
55.33
#1.04
54.15
11.79
54.59
40.65
55.54
20.00
53.02
=3.99
L,N.C
54.19 10.38
58.43 10.29
55.04 #1.41
55.79 10.94
55.38
11.06
53.19 21.25
Table 10: Performance on MIMIC-IV dataset with AUC metric across different models and modality
combinations, given the Lab and Vital values (C, A), Clinical Notes (N, ), and ICD-9 Codes (C, modalities. MC denotes observed modality combination.
Modalities
Dataset: MIMIC-IV Metric: AUC
MC
TF
MulT
MAG
LIMoE
FuseMoE
Flex-NoB
N,N
56.31 #1.00
57.10
10.78
62.64 11.81
58.33 10.30
64.39 20.28
C,C
60.43 #0.76
65.12
#2.19
60.75
65.14 40.34
62.52
20.39
66.27
=0.17
N.C
61.37 #1.03
62.18
20.77
61.99
20.20
61.34 40.41
61.74
20.31
64.27 =0.57
C,N.C
60.66 20.05
67.35 1018
61.29 20.32
65.18 10.00
61.67 20.15
69.87 20.11
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state the challenges of the problem at hand and the contributions that we make in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations and future directions of the work in the Conclusion
section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer: [NA]
Justification: Our work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide details about the experiment implementation in Section 4.1 Experi- mental Setting. We provide further details about the data processing for the experiments in the Appendix. Thus, it is possible to replicate the main experimental results of the paper. We also provide the anonymized code.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
19

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
The data is available online https: loni wsc. edu about/ paper also provides access to the code, which is reproducible.
Justification:
The
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
httppp.cc/pubbbcguuuceeee Codeeubmii ssi crrionPPoiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details about the experimental setting, including data splits, hyperparame- ters etc., included in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For each experiment, we report the mean accuracy and Fl score, as well as the corresponding standard deviation values in Section 4. Thus, we provide appropriate information about the statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the compute resources necessary to reproduce the experiments in Section 4.1 Experimental Settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httts::/ cc http:/pp......bllccctttccccstted
Answer: [Yes]
Justification:
All content in this paper abide by the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
We discuss Societal Impact and Limitation subsection under Section Conclusion.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not use any data or models that have a high risk of misuse such as pretrained language models, image generators, or scraped datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use the ADNI dataset and provide the proper citation. All materials related to the paper will adhere the copyright policies of NeurlPS.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
22

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The details of our proposed method are outlined in Sections and 4. Addition- ally, we release the source code of our work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We do not include crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
IRB approval was not necessary for this project.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "lFLk75XvUb", "reviewer": "Reviewer_BwQM", "summary": "This paper introduces Flex-MoE, a novel multimodal learning framework for Alzheimer's Disease that handles missing modalities using a Sparse Mixture-of-Experts design and demonstrates its efficacy on the ADNI dataset.", "questions": "The most biggest question for me is pointed in weaknesses Q2: The interpretation of \\( B_{M \\setminus m, m} \\) in Equation (2) is unclear; it should be better explained in details. Whether it involves a learnable module that outputs missing modality features based on available modality features, or via other method?", "limitations": "Please refer to the weaknesses section", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The idea of Flex-MoE is clear and straightforward, effectively addressing the challenge of integrating and handling missing modalities in Alzheimer's Disease research. \n2. The expression is smooth and clear, making the complex concepts and methodologies easily understandable.", "weaknesses": "1. The paper emphasizes that the Sparse Mixture-of-Experts (SMoE) selectively activates only the most relevant experts to improve scalability, but does not provide experimental results demonstrating computational efficiency or scalability.\n\n2. The interpretation of \\( B_{M \\setminus m, m} \\) in Equation (2) is unclear; it should be better explained in details. Whether it involves a learnable module that outputs missing modality features based on available modality features, or via other method?\n\n3. The approach to handling missing modalities differs from the method in the paper \"Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling,\" which uses averaging for missing modalities. A comparison with this approach would be beneficial. The paper provides code https://github.com/billhhh/ShaSpec.\n\n4. The paper overlooks some existing missing modality baselines, such as mmFormer and ShaSpec.\n\nSome minor weaknesses:\n1. The abstract states that \"few recent studies attempt to integrate multiple modalities,\" but there are actually many methods for handling missing modalities (as noted in point 4 above), which are not compared experimentally in this paper.\n\n2. The font size in Table 1 is too small and needs to be redrawn for better readability.\n\n3. In line 180, the bold formatting for \"It\" is incorrect.\n\n4. In Table 2(b), the value 61.08 in the first row is not the best result, which should be corrected.\n\n5. Table 2 should ideally use a visualization approach similar to that in the ShaSpec paper to represent different modality combinations more clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer **BwQM** for their careful review and for mentioning that Flex-MoE is `straightforward`, `effectively addresses missing modalities in AD`, and `easily understandable`. For the remaining concerns, we provide details below:\n\n---\n\n**[W 1: Computational efficiency and scalability]**   \nThroughout the paper, we explained the rationale behind the SMoE design principle for handling missing modality scenarios by assigning modality combinations to each expert index. As the reviewer mentioned, utilizing the SMoE design indeed benefits computational efficiency and scalability. To further verify its effectiveness, we compared the training time, GFLOPs, and number of parameters used in training at this link: [PDF](https://openreview.net/attachment?id=KkGeR9PVFe&name=pdf).\n\nWe observe that Flex-MoE not only performs best in the I, G, C, B modality configuration, as shown in Table 2 of the manuscript, but also achieves efficiency in terms of training time and GFLOPs. Additionally, it demonstrates scalability with fewer parameters compared to existing baselines, including the state-of-the-art model, FuseMoE [1].\n\n[1] https://arxiv.org/abs/2402.03226\n\n---\n\n**[W 2 & Q 1: Interpretation of $\\mathbf{B} _ {\\mathcal{M} \\setminus m}$ in Equation (2)]**  \nThe main idea of the missing modality bank completion is to supplement missing modalities from a learnable embedding bank. If a sample contains missing modality data, we retrieve the missing information (i.e., embedding) from this bank, which includes all possible modality combinations and their respective missing modalities. Otherwise, for the observed input, the data directly passes through the modality-specific encoder.\n\nFor example, if a patient lacks clinical data (i.e., $m = \\mathcal{C}$ where $m$ denotes the missing modality, thus $\\mathcal{M} \\setminus m = \\{ \\mathcal{I}, \\mathcal{B}, \\mathcal{G} \\}$) but has imaging, biospecimen, and genetic data, the observed modalities pass through their respective encoders. The missing clinical embedding is supplemented from the missing modality bank, indexed by the observed modalities (i.e., $\\mathbf{B} _ {\\mathcal{M} \\setminus m} = \\mathbf{B} _ {\\{\\mathcal{I}, \\mathcal{B}, \\mathcal{G}\\}}$). This approach ensures that encoders only process observed features, preventing them from being influenced by incomplete or naively imputed data. As this missing modality bank is learnable throughout the iterations, it eventually possesses the knowledge relevant to the downstream task, such as AD stage prediction.\n\nEmpirically, as shown in Figure 4 of the manuscript, the clinical modality (C) shows more significant similarities with the full modality combination when predicting AD diagnosis compared to other modalities (I, G, B). Additionally, patients missing clinical (C) and genetic (G) modalities exhibit more similarities compared to other missing modality combinations, reinforcing our claim that missing modality combinations provide unique diagnostic contexts in AD.\n\n---\n\n**[W 3 & W 4 & M-W 5: Comparison with other baselines in ShaSpec style]**  \nWe appreciate highlighting these two relevant baselines in our study. To briefly summarize each paper: ShaSpec employs a strategy using auxiliary tasks based on distribution alignment and domain classification, along with a residual feature fusion procedure, to learn shared and specific features for handling the missing modality issue. During the rebuttal period, we employed ShaSpec, and the performance comparison on the ADNI dataset with additional metric, AUC can be found here: [PDF](https://openreview.net/attachment?id=KkGeR9PVFe&name=pdf).  \nInterestingly, while ShaSpec has a specific module to handle missing modalities, it falls short of Flex-MoE. We believe this is due to forcing distribution alignment without carefully considering the context of each modality combination. For instance, when the clinical modality is missing, ShaSpec leverages other modalities to generate the clinical feature, but this may overlook the unique contextual meaning of the missing clinical data. In contrast, Flex-MoE retrieves a learnable embedding from the missing modality bank to supplement the clinical embedding without forcing observed modalities to contribute, allowing the clinical embedding more flexibility to learn downstream task-specific information.\n\nMoreover, mmFormer is an end-to-end framework initially designed for different MRI modalities, consisting of hybrid modality-specific encoders, a modality-correlated encoder, and a convolutional decoder. This approach differs from ours, which considers images as one modality and extends the perspective to genetic, biospecimen, and clinical data. When implementing mmFormer for our task, we followed its proposed architecture for processing image data and replaced their encoder with our modality-specific encoders used in Flex-MoE. The results can be found here: [PDF](https://openreview.net/attachment?id=KkGeR9PVFe&name=pdf).\n\nIn summary, mmFormer lacks the ability to handle diverse modalities beyond images, which have different contexts, making it significantly less effective than Flex-MoE. Additionally, by supplementing missing modality embeddings as global modality-invariant features, mmFormer fails to handle the specific context of modality combinations and corresponding missing modalities. This justifies the necessity of incorporating modality combination information while handling missing modalities, as proposed in Flex-MoE.\n\n---\n\n**[M-W 1,2,3: Improvements]**  \nThank you for the constructive feedback. We will incorporate the discussed baselines, change the font size in Table 1 for better readability, and carefully revise the bold font.\n\n---\n\n**[M-W 4: Best result in Table 2 (b)]**  \nWe have checked the bold formatting in Table 2 (b) and found no errors. Flex-MoE with 61.08 performs best in that row. Could we kindly ask reviewer **BwQM** to re-clarify the point? We are ready to provide more details and refine accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZLDEZXzRYA", "reviewer": "Reviewer_qdnn", "summary": "The paper presents a multimodal learning framework, Flex-MoE (Flexible Mixture-of-Experts), designed to integrate diverse modalities in Alzheimer's Disease (AD) research using a Sparse Mixture-of-Experts design. Flex-MoE sorts samples based on the number of available modalities and processes them through modality-specific encoders. The framework trains experts with full modality samples and uses an S-Router to adapt the knowledge for fewer modality combinations.", "questions": "1. Why is the model specifically designed for Alzheimer's disease, and how does it differ from a general model for missing modalities?\n2. How were the labels (dementia, CN, MCI) decided? What is the prediction timeline, and were patients at different stages treated as separate samples? What time period of data was used, and was the clinical data time series data?\n3. What is the imbalance rate of the datasets? Including metrics like AUC can provide a more complete understanding of the model’s performance. How was the threshold chosen for determining the classification boundary to obtain the F1 score?\n4. What specific computational resources are required for training and inference using Flex-MoE?\n5. How would the model perform on an additional external dataset?", "limitations": "1. The evaluation and practical application of the model are challenging to assess. The necessity of training different models for each modality combination is not well justified.\n2. The lack of external dataset validation limits the generalizability of the model's performance.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The research topic addressing missing modalities in Alzheimer’s disease is crucial and highly relevant.\n2. Source code is provided, facilitating the replication of experiments.", "weaknesses": "1. The model's specificity to Alzheimer's disease is unclear, as it appears to be a general approach for handling missing modalities.\n2. The experimental setup lacks clarity. Details on how labels (e.g., dementia, CN, MCI) were decided, the prediction timeline, and the handling of patients at different stages are not provided. Additionally, the use of clinical data and whether it is time series data is not explained.\n3. The evaluation is unconvincing. The imbalance rate of the datasets is not discussed. Only accuracy and F1 score are used for evaluation, but additional metrics like AUC could provide a more comprehensive understanding of the model’s performance. The process for selecting the threshold for classification to obtain the F1 score is also unclear.\n4. The computational resources required for training and inference using Flex-MoE are not specified, especially given the need to train the model for various modality combinations.\n5. No external dataset is included to validate the model's performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer **qdnn** for the comprehensive review of our paper. Specifically, we appreciate the reviewer's recognition of our key contribution in addressing missing modalities in AD as `crucial` and `highly relevant`. For the concerns raised, we provide the following responses:\n\n---\n\n**[W 1 & Q 1: model’s specificity to AD]**  \nAs the reviewer mentioned, our approach can be generalized, similar to common phenomena in various biomedical papers (e.g., Transformer in 3D MRI [1], Graph Neural Network in the single-cell domain [2,3]). However, we emphasize the importance of a careful application that considers domain-specific characteristics, especially in response to recent improvements in the ML field. In this paper, we focused on the AD domain for two main reasons:\n\n- **Underexplored Characteristic**: Despite the comprehensive multi-modal nature of AD (e.g., clinical, biospecimen, image, genetic) [3], it remains underexplored compared to domains with fewer modalities (e.g., image, text) [4, 5]. Empirically, as shown in Table 2 of the manuscript, the direct application of multimodal ML, e.g., LIMoE [5], underperforms compared to Flex-MoE in handling 3 or more modalities and in overall performance. This supports our motivation to address the diverse multimodal nature prevalent in the AD domain and the specific missing modality combinations.\n\n- **Unique Characteristics of Missing Modalities**: In the AD domain, the missing modality problem provides a unique context for each patient’s diagnosis. For instance, patients with biospecimen and image modalities but missing clinical and genetic data present unique characteristics [6]. Empirically, as shown in Figure 4 of the manuscript, clinical modality (C) shares more significant similarities with the full modality combination when predicting AD diagnosis compared to other modalities (I, G, B). Additionally, patients missing clinical (C) and genetic (G) modalities show more similarities compared to other missing modality combinations, reinforcing our claim that missing modality combinations provide unique diagnostic contexts in AD.\n\nIn summary, our study aims to advance clinical diagnosis in the AD domain by effectively integrating multimodal data, especially in the growing era of AI4Science (Biology, Medicine, Health).\n\n[1] https://www.nature.com/articles/s41598-024-59578-3  \n[2] https://www.nature.com/articles/s41467-021-22197-x  \n[3] https://www.nature.com/articles/s41467-023-36559-0  \n[4] https://arxiv.org/abs/2309.15857  \n[5] https://arxiv.org/abs/2206.02770  \n[6] https://pubmed.ncbi.nlm.nih.gov/24360540/\n\n---\n\n**[W 2, 3 & Q 2: Clarity in Experimental Setup]**  \nTo validate the effectiveness of Flex-MoE, we focused on the gold-standard ADNI dataset. Following existing studies [1,2], we performed an AD stage prediction task to predict specific labels for Dementia, CN, and MCI. Specifically, we used the diagnostic summary file 'DXSUM_PDXCONV_22Apr2024.csv' and the 'DIAGNOSIS' column, which underwent clinical and neuropsychological evaluation as reported by ADNI [3]. The label statistics in our study were: CN (1030 patients, 43.3%), MCI (860 patients, 36.1%), Dementia (490 patients, 20.6%). Despite concerns about imbalance, we observed a relatively balanced ratio of 490/1030 = 0.47. The F1-score was calculated using the scikit-learn package [4], which automatically computes the score without manual selection of threshold.\n\nRegarding timeline information, timestamp data is not readily available across genetic, biospecimen, and clinical modalities. However, for the image modality, we selected the most recent image to ensure relevance and mapped the patient ID with other modalities. Thus, our method remains static, but dynamic analysis with time information is a promising direction for future work.\n\nWe will add this information to the final version for clarity. Detailed statistics and preprocessing steps for each modality are provided in Section 4.1 and Appendix A of the manuscript, addressing the reviewer's concerns about the clinical modality.\n\n[1] https://www.nature.com/articles/s41598-020-74399-w  \n[2] https://www.nature.com/articles/s41598-018-37769-z  \n[3] https://adni.loni.usc.edu/wp-content/uploads/2008/07/inst_about_data.pdf  \n[4] https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html\n\n---\n\n**[W 4 & Q 4: Computational Resources]**  \nWe appreciate the reviewer's concerns about computational resources. We used NVIDIA A100 GPUs for training and inference but highlight the efficiency of adopting the Sparse MoE design, where only the selected top-k experts are utilized. We performed a computational comparison experiment in terms of Mean Time, FLOPs, and the number of activated parameters, which can be found here: [PDF](https://openreview.net/attachment?id=KkGeR9PVFe&name=pdf).  \nThe result demonstrate the efficiency of the SMoE design, showcasing the potential for various real-world datasets.\n\n---\n\n**[W 3, 5 & Q 4, 5 & L 1, 2: More Comprehensive Experiment]**  \nTo achieve greater generalizability, we added the MIMIC dataset [1], the AUC as an additional metric, and two more baselines (ShaSpec and mmFormer) during the rebuttal period. We tested all possible modality combinations, and the results can be found here: [PDF](https://openreview.net/attachment?id=KkGeR9PVFe&name=pdf).  \nOur experiments show that Flex-MoE outperforms existing baselines across various metrics and modality combinations, thanks to its careful consideration of modality combinations. Additionally, to justify the necessity of training different experts, Figure 5 shows that each expert index contains both global context knowledge and specialized modality combination knowledge, providing flexibility in responding to various modality combinations. Figure 4 further illustrates the interpretability of how different observed and missing modality combinations share similarities in predicting AD status.\n\n[1] https://physionet.org/content/mimiciv/3.0/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6UjZ2Wrdua", "reviewer": "Reviewer_6yyS", "summary": "The paper aims to deal with the task of AD patient classification with missing modalities, and proposes a framework, Flex-MoE.\nFlex-MoE also utilizes a learnable missing modality combination bank to complete the missing embeddings of missing modalities.\nThe training strategy and the gating mechanism of Flex-MoE are also well-designed to handle this task.\nThe experiments verify that Flex-MoE is able to achieve SoTA performance in this task on the shown modality combinations.", "questions": "Please see the section of weaknesses. \nI may raise or lower my score after the rebuttal.", "limitations": "The authors have claimed the limitations and broader impacts of the proposed method.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The task explored by this study is practical in real-world scenarios. Addressing the issue of missing modalities in the AD domain is worth the attention of the community.\n- The introduction section is well-written and easy to follow. (suggestion: using .pdf or .svg format for the figures is recommended.)\n- The motivations for the proposed techniques in Flex-MoE, which you have claimed in your paper, are rational and decent.\n- The ablation and sensitivity study is sufficient, showcasing the effectiveness of the components in Flex-MoE.", "weaknesses": "- The expression from Line 159 to Line 167 is a little bit confusing. More explanation (or even illustration) is needed.\n- Is the usage of the missing modality bank just a simple look-up operation? I am concerned that it seems to be naive and there may be a more smart approach to achieve this.\n- There is not $R_{(x_j)}$ of Line 199 in your equations. You need to correct this.\n- Why the S-router can be trained to activate the corresponding expert index by Eq. (3)?\n- It would be better if you could use some equations to assist to express the content of Line  204-215.\n- This paper also leaks sufficient discussion of some related works, such as other tasks of biomedical images [1,2], in Sec.2 or appendix.\n- The modality combinations of the experiments in Table 2 is few. It would be better to provide the experiments based on more combinations to showcase the performance of Flex-MoE.\n\n[1] Zhou, D., Gu, C., Xu, J., Liu, F., Wang, Q., Chen, G., & Heng, P. A. (2023). RepMode: learning to re-parameterize diverse experts for subcellular structure prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3312-3322).\n\n[2] Jiang, Y., & Shen, Y. (2024). M$^4$oE: A Foundation Model for Medical Multimodal Image Segmentation with Mixture of Experts. arXiv preprint arXiv:2405.09446.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer **6yyS** for mentioning our work as `practical`, `addressing missing modalities is worth attention to the AD community` and `rationale and decent techniques`. For the concerns, we provide responses below:\n\n--- \n\n**[W 1, 3: Expression from Line 159 - 167, 199]**  \nThe main idea of the missing modality bank completion is to supplement missing modalities from a predefined bank, ensuring robust data integration with observed ones. For example, if a patient lacks clinical data but has imaging, biospecimen, and genetic data, the observed modalities pass through their specific encoders. The missing clinical embedding is supplemented from the missing modality bank, indexed by the observed modalities (e.g., {Imaging, Biospecimen, Genetic}, Clinical). This approach prevents reliance on incomplete or naively imputed data, as encoders only process observed modalities. Once embeddings are standardized in dimension, they proceed to the Sparse MoE layer within the Transformer architecture. For line 199,  it should be $\\text{max}(\\mathcal{S}\\text{-Router}(x_j))$. To improve readability, we will include such details in the final version.\n\n---\n\n**[W 2: Usage of Missing Modality Bank]**  \nAs the reviewer mentioned, this can be seen as a look-up operation, but the way we construct and leverage this look-up table is what makes our work novel and distinctive. The rationale behind designing the missing modality bank is to differentiate the context of patient groups based on their unique observed modality combinations and corresponding missing modalities. In the AD domain, the missing modality problem provides a unique context for each patient’s diagnosis. For example, patients with biospecimen and image data but missing clinical and genetic data exhibit unique characteristics [1]. Motivated by this, we designed the missing modality bank with learnable embeddings indexed by observed modality combinations and their corresponding missing modality. This allows existing encoders to process only the observed features without being distracted by missing samples, equipping the model to handle various observed modality contexts flexibly. Empirically, as shown in Figure 4 of the manuscript, the clinical modality (C) shows more significant similarities with the full modality combination when predicting AD diagnosis compared to other modalities (I, G, B). Additionally, patients missing clinical (C) and genetic (G) modalities exhibit more similarities compared to other missing modality combinations, reinforcing our claim that missing modality combinations provide unique diagnostic contexts in AD.\n\n[1] https://pubmed.ncbi.nlm.nih.gov/24360540/\n\n---\n\n**[W 4: Equation to assist line 204 - 215]**  \nHere, we aim to utilize load balancing loss [1] especially targeted for the remaining experts (i.e., experts except top-1 selected expert) to ensure experts for each modality combination to be balanced activated. Formally it can be expressed as, \n\n1. **Load Balancing Loss**   \n   $\\mathcal{L}_{\\text{balance}} = \\text{CV}^2(\\sum{_j^N} \\text{importance}{_j}) + \\text{CV}^2(\\sum{_j^N} \\text{load}{_j}) $ \n- where $N$ is the number of samples\n\n2. **Coefficient of Variation Squared (CV²)**  \n   $ \\text{CV}^2(x) = \\left( \\frac{\\sigma(x)}{\\mu(x)} \\right)^2 $\n-    where $\\sigma(x)$ is the standard deviation of $x$ and $\\mu(x)$ is the mean of $x$.\n\n3. **Importance and Load**   \n  $\\text{importance}{_e} = \\sum{_j^N} g _ {je},  \\forall e \\in E \\setminus \\{e _ {\\text{top-1}}\\}  $\n - where $e$ is expert index and $g{_ie}$ is the gate value for sample $j$ with expert $e$    \n $\\text{load}{_e} = \\sum{_j^N} \\delta(g _ {je} > 0),  \\forall e \\in E \\setminus \\{e _ {\\text{top-1}}\\}  $\n - where $\\delta(g _ {je} > 0)$ is an indicator function that is 1 if the gate value $g _ {ie}$ is greater than 0, indicating that the expert $e$ is selected for sample $j$.\n\nFor the prediction head, we can simply denote it as: $\\mathbf{Y} = \\mathbf{Z} \\cdot \\mathbf{W}$ where\n$\\mathbf{Y} \\in \\mathbb{R}^{N \\times |\\mathcal{C}|} $ denote the predicted probability for each sample on each class, $\\mathbf{Z} \\in \\mathbb{R}^{N \\times (D*|\\mathcal{M}|)}$ denote the concatenated embedding of each modality, and $ \\mathbf{W} \\in\\mathbb{R}^{(D*|\\mathcal{M}|) \\times |\\mathcal{C}|}$ denote the weight matrix that transforms the concatenated embedding into final prediction space.\n\nFor the final version, we will incorporate above equations to provide better understanding.\n\n[1] https://arxiv.org/abs/1701.06538\n\n---\n\n**[W 5: Discussion of related works]**  \nTo briefly summarize, RepMode predicts 3D fluorescent images of subcellular structures from 3D transmitted-light images, addressing challenges of partial labeling and multi-scale variations. M4oE uses modality-specific experts and a gating mechanism to enhance medical image segmentation across various modalities. However, RepMode focuses on subcellular structure prediction, and M4oE focuses on medical image segmentation, differing from our work, which aims to accurately classify the stage of AD disease, particularly in the multimodal AD domain. While these works leverage image modalities, they lack the ability to cope with various other modalities, such as genetic and clinical. Moreover, although M4oE uses modality-specific experts, it does not consider the characteristic of modality combination information in practical missing modality scenarios, which is our main focus.\n\n---\n\n**[W 6: More modality combination experiments]**  \nTo achieve greater generalizability, besides more modality combinations, we added the MIMIC dataset [1], the AUC score as an additional metric, and two more baselines (ShaSpec and mmFormer) during the rebuttal period. The results can be found here: [PDF](https://openreview.net/attachment?id=KkGeR9PVFe&name=pdf).\n\nWe still observe Flex-MoE outperforms existing baselines across various modality combinations.\n\n[1] https://physionet.org/content/mimiciv/3.0/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
In this study, we propose **Flex-MoE**, a novel framework designed to address the issue of missing modalities in the AD domain, where existing studies often **(1)** rely on single modality and complete data, and **(2)** overlook modality combinations. As a remedy, Flex-MoE includes a **missing modality bank completion** step, followed by **expert generalization and specialization** steps, and is equipped with a novel router design.

We express our profound gratitude to all reviewers for their time and effort in evaluating our manuscript. We particularly appreciate the positive feedback on the `clear and straightforward idea`, `effectively addressing missing modalities in AD domain which is crucial and highly relevant`, `worth the attention in the community`, `motivations for Flex-MoE are rational and decent`. Besides, all the constructive feedback has been invaluable in enhancing our work. There were common concerns raised by some reviewers, which we address below, alongside more detailed responses to each reviewer.

---

1. More comprehensive experiments - Reviewers **BwQM**, **qdnn**, **6yyS**
- To achieve greater generalizability, we added the (1) MIMIC dataset [1], the (2) AUC score as an additional metric, and (3) two more baselines (ShaSpec and mmFormer) during the rebuttal period. We tested (4) all possible modality combinations and the comprehensive result are shown in below [PDF](https://openreview.net/attachment?id=KkGeR9PVFe&name=pdf). 
- The results show that in most cases, Flex-MoE consistently outperforms existing baselines across various metrics and modality combinations, thanks to its careful consideration of modality combinations. Notably, as the number of available modalities increases (e.g., 2, 3, 4), the performance of Flex-MoE significantly improves. In contrast, other baselines achieve their best performance when utilizing a subset of modalities. This demonstrates Flex-MoE's potential to effectively advance and address the challenges in the multi-modal AD domain.
- **Details on MIMIC dataset preprocessing**: For the MIMIC dataset, we use the Medical Information Mart for Intensive Care IV (MIMIC-IV) database, which contains de-identified health data for patients who were admitted to either the emergency department or stayed in critical care units of the Beth Israel Deaconess Medical Center in Boston, Massachusetts24. MIMIC-IV excludes patients under 18 years of age. We take a subset of the MIMIC-IV data, where each patient has at least more than 1 visit in the dataset as this subset corresponds to patients who likely have more serious health conditions. For each datapoint, we extract ICD-9 codes, clinical text, and labs and vital values. Using this data, we perform binary classification on one-year mortality, which foresees whether or not this patient will pass away in a year. We drop visits that occur at the same time as the patient's death. In order to align the experimental setup with the ADNI data, which does not contain temporal data, we take the last visit for each patient.

---

2. Computational Resources - Reviewers **BwQM**, **qdnn** 
- Regarding the usage of SMoE design, we further performed a computational comparison experiment in terms of Mean Time, FLOPs, and the number of activated parameters, where the result can be found in below [PDF](https://openreview.net/attachment?id=KkGeR9PVFe&name=pdf). 
- The results demonstrate the efficiency of the Sparse MoE design, showcasing the potential for improved performance on various real-world datasets.

---

3. Interpretation of Equation (2) - Reviewers **BwQM**, **6yyS**
- The main idea of the missing modality bank completion is to supplement missing modalities from a predefined bank, ensuring robust data integration with observed ones. For example, if a patient lacks clinical data but has imaging, biospecimen, and genetic data, the observed modalities pass through their specific encoders. The missing clinical embedding is supplemented from the missing modality bank, indexed by the observed modalities (e.g., {Imaging, Biospecimen, Genetic}, Clinical). This approach prevents reliance on incomplete or naively imputed data, as encoders only process observed modalities. 

---

If you have any remaining concerns, we are more than happy to discuss them until the end of the discussion period. Please do not hesitate to ask any questions. Once again, we deeply appreciate all the careful reviews and the time from the reviewers.

Best,  
Authors

Meta-review: This paper addresses multimodal learning with missing modalities with Flex-MoE, which is a sparse mixture of experts method.  The approach is tested for Alzheimer’s disease classification using the ADNI dataset, and results show SOTA performance.  The reviewers identified several strengths, such as: 1) idea is clear, 2) paper is well written, 3) research topic is relevant, 4) motivation is well explained, and 5) the ablation study is comprehensive. The reviews also identified a few issues, which include: 1) missing more comprehensive experiments in other tasks, 2) missing discussion on complexity, 3) equation (2) needs clarification, and 4) missing comparison with other methods (e.g., mmformer, shaspec). The rebuttal addressed all points above and the final scores of the paper are 5,7,7 (average score of 6.33, with an average confidence of 3.67). I recommend this paper to be accepted.
Decision: Accept (spotlight)