Title: FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion
Venue: NeurIPS 2024
Accepted: yes
Keywords: Multimodal Fusion; Mixture-of-Experts; MoE Theory; Missingness and Irregularity
Abstract: As machine learning models in critical fields increasingly grapple with multimodal data, they face the dual challenges of handling a wide array of modalities, often incomplete due to missing elements, and the temporal irregularity and sparsity of collected samples. Successfully leveraging this complex data, while overcoming the scarcity of high-quality training samples, is key to improving these models' predictive performance. We introduce ``FuseMoE'', a mixture-of-experts framework incorporated with an innovative gating function. Designed to integrate a diverse number of modalities, FuseMoE is effective in managing scenarios with missing modalities and irregularly sampled data trajectories. Theoretically, our unique gating function contributes to enhanced convergence rates, leading to better performance in multiple downstream tasks. The practical utility of FuseMoE in the real world is validated by a diverse set of challenging prediction tasks.

FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion

Abstract

As machine learning models in critical fields increasingly grapple with multimodal data, they face the dual challenges of handling a wide array of modalities, often incomplete due to missing elements, and the temporal irregularity and sparsity of collected samples. Successfully leveraging this complex data, while overcoming the scarcity of high-quality training samples, is key to improving these models' predictive performance. We introduce 'FuseMoE', a mixture-of-experts framework incorporated with an innovative gating function. Designed to integrate a diverse number of modalities, FuseMoE is effective in managing scenarios with missing modalities and irregularly sampled data trajectories. Theoretically, our unique gating function contributes to enhanced convergence rates, leading to better performance in multiple downstream tasks. The practical utility of FuseMoE in the real world is validated by a diverse set of challenging prediction tasks.

1 Introduction

Multimodal fusion is a critical and extensively studied problem in many significant domains [78, 94, 90, 9], such as sentiment analysis [26, 54], image and video captioning [42, 41], and medical prediction [33, 83]. Previous research has shown that embracing multimodality can improve predictive performance by capturing complementary information across modalities, outperforming single-modality approaches in similar tasks [71, 32]. However, an ongoing challenge lies in the creation of scalable frameworks for fusing multimodal data under a variety of conditions, and in creating reliable models that consistently surpass their single-modal counterparts.

Handling a variable number of input modalities remains an open challenge in multimodal fusion, due to challenges with scalability and lack of unified approaches for addressing missing modalities. Many existing multimodal fusion methods are designed for only two modalities [25, 102, 101], rely on costly pairwise comparisons between modalities [90], or employ simple concatenation approaches [83], rendering them unable to scale to settings with a large number of input modalities or adequately capture inter-modal interactions. Similarly, existing works are either unable to handle missing modalities entirely [101, 98] or use imputation approaches [89, 51, 83] of varying sophistication. The former methods restrict usage to cases where all modalities are completely observed, significantly diminishing their utility in settings where this is often not the case (such as in clinical applications); the latter can lead to suboptimal performance due to the inherent limitations of imputed data. In addition, the complex and irregular temporal dynamics present in multimodal data have often been overlooked [101, 88], with existing methods often ignoring irregularity entirely [83] or relying on positional embedding schemes [90] that may not be appropriate when modalities display a varying degree of temporal irregularity. Consequently, there is a pressing need for more advanced and scalable multimodal fusion techniques that can efficiently handle a broader set of modalities, effectively manage missing and irregular data, and capture the nuanced inter-modal relationships necessary for robust and accurate prediction. We use the term FlexiModal Data to capture several of these key aspects, which haven't been well-addressed by prior works:

'Flexi' suggests flexibility, indicating the possibility of having any combination of modalities, even with arbitrary missingness or irregularity.

FlexiModal data is most evident in clinical scenarios, where extensive monitoring results in the accumulation of comprehensive electronic health records (EHRs) for each patient. A typical EHR encompasses diverse data types, including tabular (e.g., age, demographics, gender), images (Xrays, magnetic resonance imaging, and photographs), clinical notes, physiological time series (ECG and EEG), and vital signs (blood chemistry, heart rate). In this setting, we observe a variety of modalities, sampled with varying irregularity and a high degree of missingness and sparsity.

Contributions In this paper, we introduce a novel mixture-of-experts (MoE) framework, which we call FuseMoE , specifically designed to enhance the multimodal fusion of FlexiModal data. FuseMoE incorporates sparsely gated MoE layers in its fusion component, which are adept at managing distinct tasks and learning optimal modality partitioning. In addition, FuseMoE surpasses previous cross-attention-based methods in scalability, accommodating an unlimited array of input modalities. Furthermore, FuseMoE routes each modality to designated experts that specialize in those specific data types. This allows FuseMoE to effectively handle scenarios with missing modalities by dynamically adjusting the influence of experts primarily responsible for the absent data, while still utilizing the available modalities. Lastly, FuseMoE integrates a novel Laplace gating function, which is theoretically proven to ensure better convergence rates compared to traditional Softmax functions, thereby enhancing predictive performance. We have conducted comprehensive empirical evaluations of FuseMoE across a range of application scenarios to validate its effectiveness.

…

…

Per-

Modality

Encoders

+

Multi-

Time

Attention

Vital Signs

ECG

Clinical

Notes

Chest-X-

Rays

(missing)

11:38 AM

2:55 PM

5:49 PM

10:14 PM

1/1/2023

2/15/2023

3/10/2023

4/20/2023

…

Multi-

Head

Self-

Attention

Sparse

Gate/

Router

FFN 1

FFN 2

FFN 3

FFN 4

FFN 5

FFN 6

FFN N

MoE

Fusion

Layer

×𝐿

+

Dense

Encoder

Layer

Task-

Specific

Loss

+

Load

Balance

Loss

Add

+

Normalize

2 FuseMoE: Enhance Predictive Performance for FlexiModal Data

In this section, we delve into the fundamental components of FuseMoE , illustrated in Figure 1. We focus on two critical elements: the modality and irregularity encoder, and the MoE fusion layer.

2.1 Sparse MoE Backbone

The main components of a sparse MoE layer are a network G as a sparse gate and an expert network E . [79] proposed a TopK gating function that takes as an input a token representation x ∈ R D and then routes it to the TopK experts out of the set { E i } S i =1 . The gating network parameter W ∈ R D × N produces logits h s ( x ) = Top K( x · W ) , which are normalized via Softmax:

Gating Network Design The gating network's advantage lies in its capacity to be concurrently trained with FFNs, facilitating the learning of an optimal sparse combination of experts. Essentially, by evaluating the similarity between the input token and the experts, the gating network/router optimally matches the input partition with the most suitable experts. In many cases, variations in the routing mechanism can greatly influence performance across diverse applications [48]. The Softmax gating is the most widely adopted across domains [77, 79]. We introduce a novel Laplace gating function that offers enhanced convergence guarantees and delivers superior predictive performance, particularly in FlexiModal applications. The function is formulated as follows:

Each expert network ( E i : R D → R D ) contains a feed-forward layer (FFN) and its parameters are independent of other models. The final output of the expert network y is the linearly weighted combination of each expert's output on the token by the gate's output: y = ∑ S i =1 G ( x ) i E i ( x ) .

The Laplace gating function, characterized by its Euclidean term exp( -∥ W -x ∥ 2 ) , is less prone to converge towards extreme weight distributions due to the bounded nature of this term. In subsequent sections, we will illustrate how this gating function facilitates faster parameter estimation rates compared to Softmax gating. Moreover, our empirical findings indicate that the Laplace gating exhibits enhanced performance in managing FlexiModal data.

2.2 Modality and Irregularity Encoder

To encode the irregularity of sampling in each modality, we utilize a discretized multi-time attention (mTAND)module [82], which leverages a time attention mechanism [43, 92] to discretize irregularly sampled observations into discrete intervals. Specifically, given a set of l k continuous time points,

Router

R1

R2

R3

R1

R2

R3

FFN 1

FFN 2

FFN 3

FFN 4

FFN 1

FFN 2

FFN 3

FFN 4

FFN 1

FFN 2

FFN 3

FFN 4

TS

TXT IMG

TS

TXT

IMG

TS

TXT

IMG

𝑝 = 0.45

𝑝 = 0.35

(a) Joint experts & router

(b) Per-modality routers

(c) Disjoint experts & routers

t ∈ R l k , corresponding to the k th dimensionality of a given modality, we employ H embedding functions ϕ h ( τ ) to embed each τ k ∈ t k in a d h dimensional vector space (detailed definition and examples can be found in Appendix B and E). The i th dimension of the h th embedding is defined as

where { w i , ϕ i } d h i =1 are learnable parameters. By performing this for each continuous time point in t k , we create a d h dimensional representation of each time point in H different embedding spaces. We then leverage these embeddings to discretize the irregularly sampled observations into discretized bins. Specifically, we seek to discretize x k (with l k corresponding observation times t k ) into γ regularly sampled intervals γ . We do this via an attention mechanism, which, for each embedding function ϕ h ( τ ) , takes γ as queries, t k as keys, and x k as values and produces ˆ x k,h ∈ R γ embeddings for each sequence. Formally,

where Q h and K h are learnable parameters. This formulation allows us to discretize univariate observations x k into γ regularly-sampled bins. To model irregularity across a multivariate set of observations for a given modality with d m dimensions, we repeat this process for each dimension of the input. This allows us to obtain an interpolation matrix ˆ X h = [ˆ x 1 ,h , ˆ x 2 ,h , ..., ˆ x d m ,h ] ∈ R γ × d m for each of the h embedding functions. We then concatenate the interpolation matrices across all H embedding functions (i.e., I = [ ˆ X 1 , ˆ X 2 , ..., ˆ X h ] ∈ R γ × ( H · d m ) ) and employ a linear projection to achieve a final, discretized embedding for each modality, Z ∈ R γ × d e , where d e denotes the desired dimensionality of each modality's representation. The discretization procedure offers a standardized approach to managing irregularly sampled time series across various input types; however, it can inevitably result in information loss. On the other hand, relying solely on the mTAND module may yield suboptimal performance due to the potentially varying sampling rates of different variables [31], especially in scenarios where the sample sizes are small. To mitigate this, we combine discretized outputs with continuous representations learned through the mTAND module.

Encoding Multiple Modalities The process described above allows us to discretize an arbitrarily long irregular, multivariate sequence into a regularly sampled, discretized embedding with length γ and dimensionality d e . We repeat this for each of the M modalities, to create M embeddings, { Z j } j M =1 , which are then combined to generate predictions.

2.3 MoE Fusion Layer

Router Design Study Upon obtaining embeddings from each of the j modalities, we propose multiple complementary approaches for processing multimodal inputs. Figure 2 illustrates a range of router design options. The most straightforward strategy involves employing a common router that handles the concatenated embeddings of all j modalities, without imposing any gating constraints.

As the complexity increases with additional modalities, we consider more sophisticated alternatives: deploying separate routers for each modality's embedding and assigning these embeddings to a shared pool of experts. This allows for distinct processing while maintaining a unified expert framework. Additionally, we further segregate these common expert pools, allowing each router to direct its respective embedding to dedicated experts skilled in handling such specific inputs. These varied router design choices offer users enhanced flexibility, enabling more fine-grained control of both inter-modal and intra-modal relationships. Details of the respective advantages and challenges of these router design mechanisms can be found in Appendix C.

We implement an entropy regularization loss to ensure balanced and stable expert utilization, a concept supported by various previous studies [58, 57, 22]. It maximizes the mutual information between modalities and experts and serves as an auxiliary loss function in addition to task-specific loss. Given a total of M modalities, and denoting H as the entropy, we define the loss function E as

where ˆ p m j ( E ) is the distribution over the experts { E i } S i =1 for the j th modality. This distribution can be approximated by ˆ p m j ( E ) = 1 l j ∑ l j i =1 p m j ( E | x m j i ) , where l j is the number of observations of the j th modality. Intuitively, we actively encourage the input embeddings to diminish the uncertainty in selecting experts. By incorporating the loss E , we aim to stabilize the experts' preferences within each modality, while promoting a diverse range of expert selections across different modalities.

Missing Modalities In scenarios where certain modalities are missing throughout the data trajectories, we substitute the original embedding Z missing with a learnable embedding Z , acting as a generic 'missing indicator'. This strategy is facilitated by employing per-modality routers, which, in conjunction with entropy regularization, guide Z predominantly toward a specific group of lessutilized experts. The new embeddings Z are dynamically adjusted throughout the model training process to minimize the task-specific loss and the entropy regularization loss. As a result, the router will assign lower weights to the experts responsible for processing these embeddings.

3 Theoretical Contribution

In this section, we provide a theoretical guarantee of the benefits of the Laplace gating over the standard Softmax gating in MoE. In particular, we conduct a convergence analysis for maximum likelihood estimation (MLE) under the Lapace gating Gaussian MoE, and demonstrate that the MLE under this model has better convergence behaviors than that under the softmax gating Gaussian MoE.

Problem Setup. Since the convergence analysis of MLE under the Top-K sparse gating MoE has been studied in [63], we will focus on examining the Laplace gating solely in the sequel. Assume that ( X 1 , Y 1 ) , . . . , ( X n , Y n ) ∈ R d × R are i.i.d. samples drawn from the Laplace gating Gaussian MoE of order k ∗ whose conditional density function p G ∗ ( Y | X ) is

where we define for any vectors v = ( v i ) k ∗ i =1 that softmax( v i ) := exp( v i ) ∑ k ∗ j =1 exp( v j ) . Above, f ( ·| µ, ν ) denotes a univariate Gaussian density function with mean µ and variance ν . For ease of the presentation, we denote G ∗ := ∑ k ∗ i =1 exp( β ∗ i ) δ ( W ∗ i ,a ∗ i ,b ∗ i ,ν ∗ i ) as a true but unknown mixing measure associated with unknown parameters ( β ∗ i , W ∗ i , a ∗ i , b ∗ i , ν ∗ i ) for i ∈ { 1 , 2 , . . . , k ∗ } . In the paper, we specifically consider two settings of the true number of experts k ∗ : (i) Exact-specified setting : when k ∗ is known; (ii) Over-specified setting : when k ∗ is unknown, and we over-specify the model in equation 4 by a Laplace gating MoE model with k > k ∗ experts. However, due to the space limit, we present only the latter setting, and defer the former setting to Appendix J.

Maximum Likelihood Estimation. We use the maximum likelihood method to estimate the unknown mixing measure G ∗ [91]. In particular, the MLE is given by

where G k (Θ) := { G = ∑ k ′ i =1 exp( β i ) δ ( W i ,a i ,b i ,ν i ) : 1 ≤ k ′ ≤ k, ( W i , a i , b i , ν i ) ∈ Θ } denotes the set of all mixing measures with at most k components. Given the MLE defined in equation 5, we are ready to present the main results. Before that, let us introduce some necessary notations for our analysis.

Notations. We denote [ n ] := { 1 , 2 , . . . , n } for any n ∈ N . For any vector v ∈ R d , ∥ v ∥ stands for its 2 -norm value. Additionally, the notation | S | indicates the cardinality of a given set S , while δ denotes the Dirac delta measure. Finally, for any two probability densities p, q dominated by the Lebesgue measure µ , we denote V ( p, q ) = 1 2 ∫ | p -q | dµ as their Total Variation distance.

Firstly, we demonstrate in Theorem 3.1 that the convergence rate of density estimation under the Laplace gating Gaussian MoE is parametric on the sample size n .

Theorem 3.1 (Density estimation) . The density estimation p ̂ G n ( Y | X ) converges to the true density p G ∗ ( Y | X ) under the Total Variation distance at the following rate:

Proof of Theorem 3.1 is in Appendix K.2. The parametric rate O ( √ log( n ) /n ) of the conditional density function p ̂ G n indicates that if there exists a loss function among parameters D such that E X [ V ( p ̂ G n ( ·| X ) , p G ∗ ( ·| X ))] ≳ D ( ̂ G n , G ∗ ) , then we will achieve the parameter and expert estimation rates via the bound D ( G n , G ∗ ) = O ( √ log( n ) /n ) .

̂ Voronoi Loss. Following the above implication, we now define a loss function among parameters based on a notion of Voronoi cells as in [55]. Given some mixing measure G , we distribute its components θ i := ( W i , a i , b i , ν i ) to the following Voronoi cells, which are generated by the components θ ∗ j := ( W ∗ j , a ∗ j , b ∗ j , ν ∗ j ) of the true mixing measure G ∗ :

̸

for any 1 ≤ j ≤ k ∗ . Note that, the cardinality of the V oronoi cell A j is exactly the number of fitted components approximating θ ∗ j . For ease of the presentation, let us denote Φ ij ( ρ 1 , ρ 2 , ρ 3 , ρ 4 ) := ∥ W i -W ∗ j ∥ ρ 1 + ∥ a i -a ∗ j ∥ ρ 2 + | b i -b ∗ j | ρ 3 + | ν i -ν ∗ j | ρ 4 , for any ( ρ 1 , ρ 2 , ρ 3 , ρ 4 ) ∈ R 4 . Then, the Voronoi loss function D 2 ( G,G ∗ ) used for our analysis under the over-specified setting is given by:

The notation ¯ r ( |A j | ) stands for the minimum value of r ∈ N such that the following system of polynomial equations does not have any non-trivial solutions for the unknown variables { ( q 1 i , q 2 i , q 3 i ) } |A j | i =1 :

A solution to the above system is regarded as non-trivial if at least among variables q 1 i is different from zero, whereas all the variables q 3 i are non-zero. It is worth noting that the function ¯ r ( · ) was previously studied in [28] to characterize the convergence behavior of parameter estimation under the location-scale Gaussian mixture models. [28] also gave some specific values of that function, namely ¯ r (2) = 4 and ¯ r (3) = 6 . Meanwhile, they claimed that it was non-trivial to determine the

10

3

10

4

10

5

Sample size

n

10

-

3

Loss

D

1

(

̂

G

n

, G

∗

)

0

.

2

n

-

0

.

49

D

1

(

̂

G

n

, G

∗

)

0

.

2

n

-

0

.

49

(a) Exact-specified setting

10

3

10

4

10

5

Sample size

n

10

-

3

10

-

2

Loss

D

2

(

̂

G

n

, G

∗

)

0

.

2

n

-

0

.

47

D

2

(

̂

G

n

, G

∗

)

0

.

2

n

-

0

.

47

(b) Over-specified setting

value of ¯ r ( m ) when m ≥ 4 , and further techniques should be developed for that purpose. Since Gaussian MoE models are generalization of the Gaussian mixture models, we also involve the function ¯ r ( · ) in our convergence analysis. Now, we provide in the following theorem the convergence rate of parameter estimation under the over-specified setting of the Laplace gating Gaussian MoE model (see also Figure 3 for the empirical convergence rates justifying the theoretical rates in Theorem 3.2).

O

O

O

O

O

Consequently, we obtain that D 2 ( ̂ G n , G ∗ ) = O ( √ log( n ) /n ) . Proof of Theorem 3.2 is in Appendix K.3. The results of Theorem 3.2 together with the formulation of the loss function D 2 in equation 7 reveal that (see also Table 2):

(i) The parameters W ∗ i , a ∗ i , b ∗ i , ν ∗ i which are fitted by exactly one component, i.e. |A n i | := |A i ( ̂ G n ) | = 1 , enjoy the same estimation rate of order O ( n -1 / 2 ) (up to some logarithmic factor), which match those in [64].

(ii) The rates for estimating the parameters W ∗ i , a ∗ i , b ∗ i , ν ∗ i which are fitted by more than one component, i.e. |A n i | > 1 , are no longer homogeneous. On the one hand, the estimation rates for the parameters b ∗ i and ν ∗ i are of orders O ( n -1 / 2¯ r ( |A n i | ) ) and O ( n -1 / ¯ r ( |A n i | ) ) , respectively, both of which are determined by the function ¯ r ( · ) and vary with the number of fitted components |A n i | . Those rates are comparable to their counterparts in [64]. On the other hand, the estimation rates for the gating parameters W ∗ i and the expert parameters a ∗ i are all of order O ( n -1 / 4 ) , which remains constant with respect to the number of fitted components. Meanwhile, those rates in [64] depend on a different system of polynomial equations from that in equation 8, which are significantly slower.

Advantage of Laplace Gating on FlexiModal Setting In the standard Softmax gating [64], the similarity score is computed as the inner product of a token's hidden representation and an expert

±

±

±

±

±

±

±

±

(a)

(b)

(c)

CIFAR-10Dataset

PAM Dataset

MIMIC-IVDataset

SoftmaxGating

85.0

Non-missing

(%)

85

Laplace Gating

85

80

82.5

80.0

Missing

Disjoint

Per-mod

80

75

77.5

75.0

A

75

70

72.5

70.0

ViT-MoE

ViT-Small-MoE

65

SeFT

mTAND

IP-Net Softmax FuseMoE

48-IHM

LOS

25-PHE

embedding. However, this approach can lead to representation collapse [13, 69], where a subset of experts dominates the decision-making process, resulting in the redundancy of other experts. This issue likely contributes to the slow rates of estimating expert parameters a ∗ i in this setting (see Table 2). By contrast, the Laplace gating function partially alleviates this problem by computing the similarity score as the L 2 -distance between token representations and expert embeddings. This approach does not inherently favor any expert based on magnitude, unlike inner product which can be biased towards experts with larger norms. The Laplace gating ensures that all experts have a more balanced opportunity to be selected based on how close they are to the token representation. Therefore, Laplace gating is beneficial when dealing with heterogeneous inputs, such as multimodal data, where its feature distributions can be very different across modalities. This is because it can handle these differences without being overly sensitive to the scale and variance of the input features. In addition, it can gracefully degrade in the presence of missing data, rather than causing abrupt changes in gating probabilities that might occur with inner product-based measures. The improved estimation rates for expert parameters a ∗ i under the Laplace gating Gaussian MoE, along with our empirical results on multiple large-scale datasets, substantiate these insights.

4 Experiments

Overview We demonstrate that FuseMoE can provide accurate and efficient predictions when applied to the FlexiModal setting. We tested FuseMoE on a diverse set of benchmarks, including MIMIC-III [40] and MIMIC-IV [36], CMU-MOSI and MOSEI [97], the Physical Activity Monitoring (PAM) dataset [75], and CIFAR-10 [46]. Compared to CMU-MOSI and MOSEI, the MIMIC ecosystem exhibits irregular and missing modality patterns and includes distinct modalities unlike PAM and CIFAR-10. Evaluating FuseMoE across these diverse datasets provides various empirical insights into critical aspects of our model's performance. Comprehensive details on the datasets, metrics, parameters, and additional results are thoroughly presented in the Appendices.

4.1 Main Results

CMU-MOSI and MOSEI Datasets We first apply our method to the CMU-MOSI and MOSEI datasets [97], which utilize visual, acoustic, and textual data for sentiment analysis and emotion recognition tasks. Our methodology employs pre-trained T5 [73] for text encoding, librosa [56] for audio feature extraction, and EfficientNet [84] for video feature encoding. Table 3 details the per-

formance of various router design mechanisms within our MoE architecture, utilizing the Laplace gating function, compared against representative baselines. The baselines include (1) the early fusion method, Tensor Fusion Network (TFN) [96]; (2) the Multimodal Transformer (MulT), which fuses modalities by modeling their interactions [90]; (3) the Multimodal Adaptation Gate (MAG), which focuses on the consistency and differences across modalities [74]; and (4) multimodal fusion using standard MoE with the Softmax gating function. Results indicate that employing an MoE backbone-regardless of the gating function chosen or whether utilizing per-modality routers or a joint experts & router configuration-significantly enhances performance on the multimodal task. This improvement is attributed to the MoE's ability to effectively allocate specific components to handle distinct input modalities, thus better addressing both inter- and intra-modal relationships.

CIFAR-10 Dataset Subsequently, we evaluate our method using the Vision-MoE framework [77] on the CIFAR-10 classification task [46], with results illustrated in Figure 4(a). In this experiment, we selectively replace the FFN layers with an even number in the Vision Transformer (ViT) models with MoE layers. These results, along with Table 3 on the CMU-MOSI and MOSEI datasets comparing Softmax-gating MoE, indicate that the Laplace gating function surpasses the standard Softmax gating function in performance. This outcome is consistent with our theoretical claims.

MIMIC-IV and PAM Datasets We then conduct comprehensive evaluations of FuseMoE on MIMIC-IV [36], and the Physical Activity Monitoring (PAM) dataset [75]. These datasets feature multiple input modalities, each characterized by varying degrees of irregular sampling or significant levels of missingness . Our tasks of interest for MIMIC datasets include the 48-hour in-hospital mortality prediction (48-IHM), 25-type phenotype classification (25-PHE), and length-of-stay (LOS) prediction. In addition to the previously mentioned baselines, we have incorporated the HAIM method [83], a data pipeline specifically designed for integrating multimodal data from the MIMICIV dataset. We also include the cross-attention combined with irregular sequences modeling approach (MISTS) [101]. Table 4 shows the outcomes of combining irregular vital signs and clinical notes from the MIMIC-IV dataset. In addition to the commonly used Softmax gating function, we also evaluated the Gaussian gating function [93] as a comparative benchmark. The FuseMoE -based methods surpass baselines in most scenarios, often by a non-trivial margin. Furthermore, we observe that HAIM shows considerable efficacy in extracting features from time series, resulting in a strong performance in the 48-IHM and LOS tasks, which are heavily reliant on such data. However, its performance appears more moderate on the 25-PHE task. The PAM dataset captures daily living activities through 17 sensors, with data from each sensor treated as a separate modality. These modalities are individually processed through time-series and irregularity encoders before being integrated into the FuseMoE framework. Our baselines include the Transformer [92], GRU-D [10], SeFT [31], a mTAND-only configuration, and IP-Net [81]. We use the Laplace gating and its joint experts & router structure in these experiments. The results in Figure 4(b) have again shown the efficacy of integrating the irregularity encoder with the MoE fusion layer.

4.2 Ablation Studies

Scalability of FuseMoE with Increasing Modalities Table 5 presents the revised outcomes of the MIMIC-IV dataset after integrating CXR and ECG of corresponding patients, employing the per-modality router and the entropy loss E within FuseMoE . This setup was chosen as it slightly outperformed the joint router with an increase in modalities. Relative to their two-modality versions, FuseMoE has effectively harnessed additional information (notably from CXR), resulting in

±

±

±

±

±

±

±

±

a significant enhancement in performance. Conversely, the addition of new modalities did not benefit the HAIM method, possibly due to its reliance on vital signs and clinical notes without adequately addressing the dynamics between different modalities. Furthermore, HAIM's notably high F1 scores on the 25-PHE task can be attributed to XGBoost's proficiency in managing missing minority classes. Note that, except for HAIM, other baselines were not designed to be agnostic to the quantity and variety of input modalities. Therefore, adapting them to manage extra and missing modalities requires considerable model changes, which might compromise their performance.

Missing Modalities Figure 4(c) illustrates the effectiveness of utilizing per-modality routers and the entropy loss E in addressing missing modalities. Initially, we compare the performance of FuseMoE on patients with fully available modalities against those with missing components, employing a joint router mechanism with the importance loss function [79], to ensure load balancing. The inclusion of datasets with missing modalities, while expanding the sample size, resulted in a decrease in performance due to the compromised data quality. However, a performance enhancement was observed upon integrating per-modality or disjoint routers with E . Notably, the outcomes for the 48-IHM and LOS tasks with missing modalities surpassed those obtained from datasets without any missingness. This is because the per-modality approach can better separate the present and missing modalities, reducing the influence of experts responsible for processing the absent inputs. Therefore, this leads to a more efficient exploitation of a broader array of samples.

5 Discussions and Limitations

In this paper, we introduced FuseMoE , a model adept at managing multimodal data characterized by random missingness or irregularity-a crucial yet relatively unexplored challenge. FuseMoE integrates MoE fusion layers with modal embeddings and offers multiple router configurations to adeptly handle multimodal inputs across different complexity levels. FuseMoE also employs an innovative Laplace gating function, which provides better theoretical results. Through empirical evaluation, FuseMoE has demonstrated superior performance across diverse scenarios. However, our current approach to encoding irregularities may potentially lead to over-parameterization when the input size is small. In our future work, we aim to identify simpler and more efficient methods to handle the irregularities of input samples while preserving the model's overall performance.

References

Vibhu Agarwal, Tanya Podchiyska, Juan M Banda, Veena Goel, Tiffany I Leung, Evan P Minty, Timothy E Sweeney, Elsie Gyang, and Nigam H Shah. Learning statistical models of

phenotypes using noisy labeled training data. Journal of the American Medical Informatics Association , 23(6):1166-1173, 2016.

Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. Publicly available clinical bert embeddings. arXiv preprint arXiv:1904.03323 , 2019.

Aryan Arbabi, David R Adams, Sanja Fidler, Michael Brudno, et al. Identifying clinical terms in medical text using ontology-guided machine learning. JMIR medical informatics , 7(2):e12596, 2019.

Zachi I Attia, Suraj Kapa, Francisco Lopez-Jimenez, Paul M McKie, Dorothy J Ladewig, Gaurav Satam, Patricia A Pellikka, Maurice Enriquez-Sarano, Peter A Noseworthy, Thomas M Munger, et al. Screening for cardiac contractile dysfunction using an artificial intelligence-enabled electrocardiogram. Nature medicine , 25(1):70-74, 2019.

Aya Awad, Mohamed Bader-El-Den, James McNicholas, and Jim Briggs. Early hospital mortality prediction of intensive care unit patients using an ensemble learning approach. International journal of medical informatics , 108:185-195, 2017.

Dimitris Bertsimas, Jean Pauphilet, Jennifer Stevens, and Manu Tandon. Predicting inpatient flow at a major hospital using interpretable analytics. Manufacturing & Service Operations Management , 24(6):2809-2824, 2022.

Serhat S Bucak, Rong Jin, and Anil K Jain. Multiple kernel learning for visual object recognition: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence , 36(7):1354-1369, 2013.

Rhonda R Butler. Icd-10 general equivalence mappings: Bridging the translation gap from icd-9. Journal of AHIMA , 78(9):84-86, 2007.

Bing Cao, Yiming Sun, Pengfei Zhu, and Qinghua Hu. Multi-modal gated mixture of localto-global experts for dynamic image fusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 23555-23564, 2023.

Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports , 8(1):6085, 2018.

JunKai Chen, Zenghai Chen, Zheru Chi, and Hong Fu. Emotion recognition in the wild with feature fusion and multiple kernel learning. In Proceedings of the 16th International Conference on Multimodal Interaction , pages 508-513, 2014.

Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining , pages 785-794, 2016.

Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. On the representation collapse of sparse mixture of experts. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Joseph Paul Cohen, Joseph D Viviano, Paul Bertin, Paul Morrison, Parsa Torabian, Matteo Guarrera, Matthew P Lungren, Akshay Chaudhari, Rupert Brooks, Mohammad Hashir, et al. Torchxrayvision: A library of chest x-ray datasets and models. In International Conference on Medical Imaging with Deep Learning , pages 231-249. PMLR, 2022.

A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum Likelihood from Incomplete Data Via the EM Algorithm. Journal of the Royal Statistical Society: Series B (Methodological) , 39(1):1-22, September 1977.

Iman Deznabi, Mohit Iyyer, and Madalina Fiterau. Predicting in-hospital mortality by combining clinical notes with time-series data. In Findings of the association for computational linguistics: ACL-IJCNLP 2021 , pages 4026-4031, 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

A Elixhauser. Clinical classifications software (ccs) 2009. http://www. hcug-us. ahrq. gov/toolssoft-ware/ccs/ccs. jsp , 2009.

William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research , 23(1):5232-5270, 2022.

Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Wei Xu. Are you talking to a machine? dataset and methods for multilingual image question. Advances in neural information processing systems , 28, 2015.

Ashutosh Garg, Vladimir Pavlovic, and James M Rehg. Boosted learning in dynamic bayesian networks for multimodal speaker detection. Proceedings of the IEEE , 91(9):1355-1369, 2003.

Aude Genevay. Entropy-regularized optimal transport for machine learning . PhD thesis, Paris Sciences et Lettres (ComUE), 2019.

Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. circulation , 101(23):e215-e220, 2000.

Brian Gow, Tom Pollard, Larry A Nathanson, Alistair Johnson, Benjamin Moody, Chrystinne Fernandes, Nathaniel Greenbaum, Seth Berkowitz, Dana Moukheiber, Parastou Eslami, et al. Mimic-iv-ecg-diagnostic electrocardiogram matched subset. 2022.

Wei Han, Hui Chen, Alexander Gelbukh, Amir Zadeh, Louis-philippe Morency, and Soujanya Poria. Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis. In Proceedings of the 2021 International Conference on Multimodal Interaction , pages 6-15, 2021.

Wei Han, Hui Chen, and Soujanya Poria. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. arXiv preprint arXiv:2109.00412 , 2021.

Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific data , 6(1):96, 2019.

N. Ho and X. Nguyen. Convergence rates of parameter estimation for some weakly identifiable finite mixtures. Annals of Statistics , 44:2726-2755, 2016.

Nhat Ho, Chiao-Yu Yang, and Michael I. Jordan. Convergence rates for Gaussian mixtures of experts. Journal of Machine Learning Research , 23(323):1-81, 2022.

Sepp Hochreiter and J¨ urgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.

Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt. Set functions for time series. In International Conference on Machine Learning , pages 4353-4363. PMLR, 2020.

Shih-Cheng Huang, Anuj Pareek, Saeed Seyyedi, Imon Banerjee, and Matthew P Lungren. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ digital medicine , 3(1):136, 2020.

Shih-Cheng Huang, Anuj Pareek, Roham Zamanian, Imon Banerjee, and Matthew P Lungren. Multimodal fusion with deep neural networks for leveraging ct imaging and electronic health record: a case-study in pulmonary embolism detection. Scientific reports , 10(1):22147, 2020.

Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pages 590-597, 2019.

Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation , 3(1):79-87, 1991.

Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimic-iv. PhysioNet. Available online at: https://physionet. org/content/mimiciv/1.0/(accessed August 23, 2021) , 2020.

Alistair Johnson, Matt Lungren, Yifan Peng, Zhiyong Lu, Roger Mark, Seth Berkowitz, and Steven Horng. Mimic-cxr-jpg-chest radiographs with structured labels. PhysioNet , 2019.

Alistair Johnson, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimiciv-note: Deidentified free-text clinical notes, 2023.

Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data , 6(1):317, 2019.

Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data , 3(1):1-9, 2016.

Justin Johnson, Andrej Karpathy, and Li Fei-Fei. Densecap: Fully convolutional localization networks for dense captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4565-4574, 2016.

Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3128-3137, 2015.

Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus Brubaker. Time2vec: Learning a vector representation of time. arXiv preprint arXiv:1907.05321 , 2019.

Britt E Keuning, Thomas Kaufmann, Renske Wiersema, Anders Granholm, Ville Pettil¨ a, Morten Hylander Møller, Christian Fynbo Christiansen, Jos´ e Castela Forte, Harold Snieder, Frederik Keus, et al. Mortality prediction models in the adult critically ill: A scoping review. Acta Anaesthesiologica Scandinavica , 64(4):424-442, 2020.

Swaraj Khadanga, Karan Aggarwal, Shafiq Joty, and Jaideep Srivastava. Using clinical notes with time series data for icu management. arXiv preprint arXiv:1909.09702 , 2019.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Yann LeCun, L´ eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Bo Li, Yifei Shen, Jingkang Yang, Yezhen Wang, Jiawei Ren, Tong Che, Jun Zhang, and Ziwei Liu. Sparse mixture-of-experts are domain generalizable learners. arXiv preprint arXiv:2206.04046 , 2022.

Yikuan Li, Ramsey M Wehbe, Faraz S Ahmad, Hanyin Wang, and Yuan Luo. Clinicallongformer and clinical-bigbird: Transformers for long clinical sequences. arXiv preprint arXiv:2201.11838 , 2022.

Ke Lin, Yonghua Hu, and Guilan Kong. Predicting in-hospital mortality of patients with acute kidney injury in the icu using random forest model. International journal of medical informatics , 125:55-61, 2019.

Jinghui Liu, Daniel Capurro, Anthony Nguyen, and Karin Verspoor. Attention-based multimodal fusion with contrast for robust clinical prediction in the face of missing modalities. Journal of Biomedical Informatics , 145:104466, 2023.

Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Efficient low-rank multimodal fusion with modality-specific factors. arXiv preprint arXiv:1806.00064 , 2018.

Karla R Lovaasen and Jennifer Schwerdtfeger. ICD-9-CM Coding: Theory and Practice with ICD-10, 2013/2014 Edition-E-Book . Elsevier Health Sciences, 2012.

Navonil Majumder, Devamanyu Hazarika, Alexander Gelbukh, Erik Cambria, and Soujanya Poria. Multimodal sentiment analysis using hierarchical fusion with context modeling. Knowledge-based systems , 161:124-133, 2018.

T. Manole and N. Ho. Refined convergence rates for maximum likelihood estimation under finite mixture models. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 14979-15006. PMLR, 17-23 Jul 2022.

Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in python. In SciPy , pages 18-24, 2015.

Clara Meister, Elizabeth Salesky, and Ryan Cotterell. Generalized entropy regularization or: There's nothing special about label smoothing. arXiv preprint arXiv:2005.00820 , 2020.

Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Multimodal contrastive learning with limoe: the language-image mixture of experts. Advances in Neural Information Processing Systems , 35:9564-9576, 2022.

Ara V Nefian, Luhong Liang, Xiaobo Pi, Liu Xiaoxiang, Crusoe Mao, and Kevin Murphy. A coupled hmm for audio-visual speech recognition. In 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing , volume 2, pages II-2013. IEEE, 2002.

Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multimodal deep learning. In Proceedings of the 28th international conference on machine learning (ICML-11) , pages 689-696, 2011.

Huy Nguyen, Pedram Akbarian, and Nhat Ho. Is temperature sample efficient for softmax Gaussian mixture of experts? In Proceedings of the ICML , 2024.

Huy Nguyen, Pedram Akbarian, TrungTin Nguyen, and Nhat Ho. A general theory for softmax gating multinomial logistic mixture of experts. In Proceedings of the ICML , 2024.

Huy Nguyen, Pedram Akbarian, Fanqi Yan, and Nhat Ho. Statistical perspective of top-k sparse softmax gating mixture of experts. In International Conference on Learning Representations , 2024.

Huy Nguyen, TrungTin Nguyen, and Nhat Ho. Demystifying softmax gating function in Gaussian mixture of experts. In Advances in Neural Information Processing Systems , 2023.

Huy Nguyen, TrungTin Nguyen, Khai Nguyen, and Nhat Ho. Towards convergence rates for parameter estimation in Gaussian-gated mixture of experts. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , 2024.

Xiaonan Nie, Xupeng Miao, Shijie Cao, Lingxiao Ma, Qibin Liu, Jilong Xue, Youshan Miao, Yi Liu, Zhi Yang, and Bin Cui. Evomoe: An evolutional mixture-of-experts training framework via dense-to-sparse gate. arXiv preprint arXiv:2112.14397 , 2021.

Behnaz Nojavanasghari, Deepak Gopinath, Jayanth Koushik, Tadas Baltruˇ saitis, and LouisPhilippe Morency. Deep multimodal fusion for persuasiveness prediction. In Proceedings of the 18th ACM International Conference on Multimodal Interaction , pages 284-288, 2016.

Raghavendra Pappagari, Piotr Zelasko, Jes´ us Villalba, Yishay Carmiel, and Najim Dehak. Hierarchical transformers for long document classification. In 2019 IEEE automatic speech recognition and understanding workshop (ASRU) , pages 838-844. IEEE, 2019.

Quang Pham, Giang Do, Huy Nguyen, TrungTin Nguyen, Chenghao Liu, Mina Sartipi, Binh T. Nguyen, Savitha Ramasamy, Xiaoli Li, Steven Hoi, and Nhat Ho. Competesmoe - effective training of sparse mixture of experts via competition. arXiv preprint arXiv:2402.02526 , 2024.

Soujanya Poria, Erik Cambria, and Alexander Gelbukh. Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis. In Proceedings of the 2015 conference on empirical methods in natural language processing , pages 2539-2544, 2015.

Gerasimos Potamianos, Chalapathy Neti, Guillaume Gravier, Ashutosh Garg, and Andrew W Senior. Recent advances in the automatic recognition of audiovisual speech. Proceedings of the IEEE , 91(9):1306-1326, 2003.

Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts. arXiv preprint arXiv:2308.00951 , 2023.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1-67, 2020.

Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, Amir Zadeh, Chengfeng Mao, LouisPhilippe Morency, and Ehsan Hoque. Integrating multimodal information in large pretrained transformers. In Proceedings of the conference. Association for Computational Linguistics. Meeting , volume 2020, page 2359. NIH Public Access, 2020.

Attila Reiss and Didier Stricker. Introducing a new benchmarked dataset for activity monitoring. In 2012 16th international symposium on wearable computers , pages 108-109. IEEE, 2012.

Stephan Reiter, Bjorn Schuller, and Gerhard Rigoll. Hidden conditional random fields for meeting segmentation. In 2007 IEEE International Conference on Multimedia and Expo , pages 639-642. IEEE, 2007.

Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, Andr´ e Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems , 34:8583-8595, 2021.

Thanveer Shaik, Xiaohui Tao, Lin Li, Haoran Xie, and Juan D Vel´ asquez. A survey of multimodal information fusion for smart healthcare: Mapping the journey from data to wisdom. Information Fusion , page 102040, 2023.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538 , 2017.

Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scaling vision-language models with sparse mixture of experts. arXiv preprint arXiv:2303.07226 , 2023.

Satya Narayan Shukla and Benjamin M Marlin. Interpolation-prediction networks for irregularly sampled time series. arXiv preprint arXiv:1909.07782 , 2019.

Satya Narayan Shukla and Benjamin M Marlin. Multi-time attention networks for irregularly sampled time series. arXiv preprint arXiv:2101.10318 , 2021.

Luis R Soenksen, Yu Ma, Cynthia Zeng, Leonard Boussioux, Kimberly Villalobos Carballo, Liangyuan Na, Holly M Wiberg, Michael L Li, Ignacio Fuentes, and Dimitris Bertsimas. Integrated multimodal artificial intelligence framework for healthcare applications. NPJ digital medicine , 5(1):149, 2022.

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pages 6105-6114. PMLR, 2019.

H. Teicher. On the mixture of distributions. Annals of Statistics , 31:55-73, 1960.

H. Teicher. Identifiability of mixtures. Annals of Statistics , 32:244-248, 1961.

H. Teicher. Identifiability of finite mixtures. Ann. Math. Statist. , 32:1265-1269, 1963.

Sindhu Tipirneni and Chandan K Reddy. Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series. ACM Transactions on Knowledge Discovery from Data (TKDD) , 16(6):1-17, 2022.

Luan Tran, Xiaoming Liu, Jiayu Zhou, and Rong Jin. Missing modalities imputation via cascaded residual autoencoder. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1405-1414, 2017.

Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguistics. Meeting , volume 2019, page 6558. NIH Public Access, 2019.

S. van de Geer. Empirical Processes in M-estimation . Cambridge University Press, 2000.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Lei Xu, Michael Jordan, and Geoffrey E Hinton. An alternative model for mixtures of experts. Advances in neural information processing systems , 7, 1994.

Bo Yang, Tao Mei, Xian-Sheng Hua, Linjun Yang, Shi-Qiang Yang, and Mingjing Li. Online video recommendation based on multimodal fusion and relevance feedback. In Proceedings of the 6th ACM international conference on Image and video retrieval , pages 73-80, 2007.

Bo Yang and Lijun Wu. How to leverage multimodal ehr data for better medical predictions? arXiv preprint arXiv:2110.15763 , 2021.

Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv:1707.07250 , 2017.

Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. Multi-attention recurrent network for human communication comprehension. In Thirty-Second AAAI Conference on Artificial Intelligence , 2018.

Xunlin Zhan, Yangxin Wu, Xiao Dong, Yunchao Wei, Minlong Lu, Yichi Zhang, Hang Xu, and Xiaodan Liang. Product1m: Towards weakly supervised instance-level product retrieval via cross-modal pretraining. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11782-11791, 2021.

Dongyu Zhang, Jidapa Thadajarassiri, Cansu Sen, and Elke Rundensteiner. Time-aware transformer-based network for clinical notes series prediction. In Machine learning for healthcare conference , pages 566-588. PMLR, 2020.

Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph-guided network for irregularly sampled multivariate time series. arXiv preprint arXiv:2110.05357 , 2021.

Xinlu Zhang, Shiyang Li, Zhiyu Chen, Xifeng Yan, and Linda Ruth Petzold. Improving medical predictions by irregular multimodal electronic health records modeling. In International Conference on Machine Learning , pages 41300-41313. PMLR, 2023.

Tongxue Zhou, Su Ruan, and St´ ephane Canu. A review: Deep learning for medical image segmentation using multi-modality fusion. Array , 3:100004, 2019.

Yanqi Zhou, Nan Du, Yanping Huang, Daiyi Peng, Chang Lan, Da Huang, Siamak Shakeri, David So, Andrew M Dai, Yifeng Lu, et al. Brainformers: Trading simplicity for efficiency. In International Conference on Machine Learning , pages 42531-42542. PMLR, 2023.

Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems , 35:7103-7114, 2022.

Contents

F.1

MISTS .

Appendix for

'FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion'

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

26

A Related Works

Multimodal Fusion Initial approaches to multimodal fusion incorporated techniques such as kernel-based methods [7, 11, 70], graphical models [59, 21, 76], and neural networks [60, 20, 67]. With the diverse evolution of deep learning models, numerous advanced methods have now been employed in the fusion of multimodal data. In the realm of sentiment analysis, [96, 52] employ a low-rank Tensor Fusion method that leverages both language and video content. Attentiongating mechanisms are used by [74, 95] to generate displacement vectors through cross-modal self-attention, which are then added to the input vectors from the primary modality. [90] takes an alternative approach by integrating multiple layers of cross-modal attention blocks in a word-level vision/language/audio alignment task.

In the context of clinical prediction, [45, 16] adopt a late fusion approach to combining vital sign and text data by concatenating embeddings from pre-trained feature extractors. [83] developed a generalizable data preprocessing and modeling pipeline for EHR encompassing four data modalities, albeit through a direct concatenation of existing feature embeddings for each modality followed by an XGBoost classifier [12]. Recently, [101] expanded on the work of [90] by introducing a discretized multi-time attention (mTAND) module [82] to encode temporal irregularities in time series and text data. Their fusion approach involves layering sets of self- and cross-modal attention blocks. However, this approach is limited to just two modalities and is not easily extendable to include additional modal components or handle missing modalities. To the best of our knowledge, existing works are tailored to application-specific settings that necessitate the computation of pairwise cross-modal relationships, which are not scalable to more general settings with arbitrary modalities. Moreover, these studies typically do not account for scenarios where modalities are missing, or rely on imputation approaches based on observed data.

Mixture-of-Experts MoE [35, 93] has gained significant popularity for managing complex tasks since its introduction three decades ago. Unlike traditional models that reuse the same parameters for all inputs, MoE selects distinct parameters for each specific input. This results in a sparsely activated layer, enabling a substantial scaling of model capacity without a corresponding increase in computational cost. Recent studies have demonstrated the effectiveness of integrating MoE with cutting-edge models across a diverse range of tasks [79, 19, 103]. These works have also tackled key challenges such as accuracy and training instability [66, 104, 72]. Given its ability to assign input partitions to specialized experts, MoE naturally lends itself to multimodal applications. This approach has been explored in fields such as vision-language modeling [58, 80] and dynamic image fusion [9]. However, the application of MoE in complex real-world settings, such as those involving FlexiModal Data, remains largely unexplored. This gap presents an opportunity to leverage MoE's potential in handling its intricate and multifaceted nature such as multimodal EHR, where reliable multimodal integration is crucial.

MoE Theory While MoE has been widely employed to scale up large models, its theoretical foundations have remained nascent. Recently, [64] provided convergence rates for both density and parameter estimation of Softmax gating Gaussian MoE. They connected these rates to the solvability of systems of polynomial equations under Voronoi-based loss functions. Later, [63] extended these theories to the top-K sparse softmax gating Gaussian MoE. Their theories further characterize the effect of the sparsity of gating functions on the behaviors of parameter estimation and verify the benefits of using top-1 sparse softmax gating MoE in practice. Other theoretical results for MoE include estimation rates of parameters and experts for multinomial logistic MoE [62], for dense-tosparse gating MoE [61], for Gaussian gating MoE [65], and for input-independent gating MoE [29].

B FlexiModal Data and Tasks of Interest

Definition of FlexiModal Data We provide a generic definition for the FlexiModal Data as we used throughout the paper. Let D = { ( x m 1 i , t m 1 i ) , ( x m 2 i , t m 2 i ) , . . . , ( x m j i , t m j i ) , y i } N i =1 to be the FlexiModal dataset with N units, where x m j i represents the input sequence from the i th unit of the j th modality, t m j i denotes the corresponding time points, and y i is the task-specific outcome. Take multimodal EHR as an example, each j th modality, which may vary from time-series data like heart

rate, blood pressure, and glucose levels to high-dimensional inputs such as clinical notes and X-rays, contains l m j observations. Figure 5 is a more specific illustration of the FlexiModal example.

B.1 MIMIC-IV and MIMIC-III Datasets

Tasks In the ICU, where rapid and informed decisions are crucial, accurate mortality prediction is essential to provide clinicians with advanced warnings of patient deterioration, aiding in critical decision-making processes [5]. Similarly, the prediction of patient length-of-stay is indispensable for optimizing treatment plans, resource allocation, and discharge processes [6]. Further, phenotyping of critical care conditions is highly relevant to comorbidity detection and risk adjustment and presents a more challenging task than binary classification, due to the heterogeneous presentation of conditions and the larger number of prediction tasks [101]. We concentrate on three critical care tasks as highlighted in [44], performing extensive empirical analysis on each building block of the proposed framework.

48-IHM In this binary classification task, we predict in-hospital mortality based on the first 48 of the ICU stay for patients who stayed in the ICU for at least 48 hours.

LOS We formulate our length-of-stay task similar to that of 48-IHM: for patients who spent at least 48 hours in the ICU, we predict ICU discharge without expiration within the following 48 hours.

25-PHE In this multilabel classification problem, we attempt to predict one of 25 acute care conditions [18, 53] (e.g., congestive heart failure, pneumonia, shock, etc.) at the end each each patient's ICU stay. Because the original task was designed for diagnoses based on ICD-9 codes, but MIMIC-IV includes both ICD-9 and ICD-10 codes, we map patients with diagnoses coded using ICD-10 using the conversion database provided by [8].

48-IHM

LOS

PHE

death

discharge

HR

Time

BP

25

series

Glucose

critical

care

Note

O

0

conditions

CXR

△

△

△

△

ECG

☆

☆

☆

☆

Time (hrs) 0

48

EOS

0

48

96

EOS

0

EOS

We implement an in-hospital mortality prediction ( 48-IHM ) task to evaluate our method's ability to predict short-term patient deterioration. Similarly, an accurate determination of patient discharge times is crucial for optimizing patient outcomes and hospital resource allocation [6], which motivates our length-of-stay ( LOS ) task. We frame 48-IHM and LOS as binary classification problems and use a 48-hour observation window (for patients who spent at least 48 hours in the ICU) to predict in-hospital mortality (48-IHM) and discharge (without expiration) within the 48 hours following the observation window (LOS). Lastly, identifying the presence of specific acute care conditions in patient records is essential for various clinical objectives, including the construction of cohorts for clinical studies and the detection of comorbidities [1]. Traditional methods, often reliant on manual chart reviews or simple billing code-based definitions, are increasingly being supplemented by machine learning techniques [27]; automating this process requires high-fidelity classifications, motivating our 25-type phenotype classification ( 25-PHE ) task. In this multilabel classification problem, we attempt to predict one of 25 acute care conditions using data from the entire ICU stay.

Evaluation In our initial analysis, we focused on patients with no missing modalities, resulting in a dataset comprised of 8,770 ICU stays for the 48-IHM and LOS tasks, and 14,541 stays for the 25-PHE task. For our analyses with missing observations, we include a total of 35,129 stays for 48-IHM and LOS, and 71.173 for 25-PHE. To evaluate the single-label tasks, 48-IHM and LOS, we employ the F1-score and AUROC as our primary metrics. In line with previous studies [101, 50, 3], we use macro-averaged F1-score and AUROC to assess the 25-PHE task.

Dataset Information We leveraged data from MIMIC-IV [36], a comprehensive database with records from nearly 300 k patients admitted to a medical center from 2008 to 2019, focusing on the subset of 73,181 ICU stays. We were able to link core ICU records (containing lab results and vital signs) to corresponding chest X-rays [39], radiological notes [38], and electrocardiogram (ECG) data [24] taking place during a given ICU stay. We allocated 70 percent of the data for model training, with the remaining 30 percent evenly split between validation and testing.

Missingness Rates The total number of samples for each of our three tasks (i.e., those in which at least one vital sign was recorded in the specified observation window), along with the total number of observations per-modality, are shown in Table 6.

Table 6: We present the total number of ICU stays in each task, taking into account observations with missing modalities. The total number of stays with at least one observation of the corresponding modality are shown in the three right-most columns.

B.2 MOSI and MOSEI Datasets

Task We focus on the multimodal sentiment analysis (MSA) task which aims to predict sentiment polarity ∈ { positive, negative, and neutral } and sentiment intensity, which is a real number ranging from -3 to +3 under a multimodal setting.

Evaluation Following previous work such as [26], we adopt mean absolute error (MAE), Pearson correlation (Corr), binary classification accuracy, F1 score computed for non-negative/negative class as evaluation metrics.

Dataset Information The CMU-MOSI dataset contains 1284/229/686 train/validation/test samples, and the CMU-MOSEI dataset contains 16326/1871/4659 train/validation/test samples. They are the largest dataset of multimodal sentiment analysis and emotion recognition to date. The datasets contain utterance videos from numerous online YouTube speakers, which are transcribed and properly punctuated, leading to multimodal input consisting of video frames, text, and audio signals.

B.3 PAM Dataset

Task Physical Activity Monitoring (PAM) dataset measures the daily living activities of 9 subjects with 3 inertial measurement units. PAM is labeled into 8 classes where each class represents an activity of daily living.

Evaluation We choose common classification accuracy as the evaluation metric for this task.

Dataset Information The processed PAM dataset contains 5,333 segments (samples) of sensory signals. Each sample is measured by 17 sensors and contains 600 continuous observations with the sampling frequency 100 Hz. PAM does not include static attributes and the samples are approximately balanced across all 8 categories.

B.4 CIFAR-10 Dataset

CIFAR-10 [46] is an established computer-vision dataset used for object recognition. It consists of 60,000 32x32 color images containing one of 10 object classes ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'), with 6000 images per class.

C Mechanisms of Different Router Designs

C.1 Joint Experts & Routers

In this approach, a concatenated embedding of all modalities is created, and this combined input is directed to selected experts by the router. This method allows the model to capture interactions between modalities at the input level, as the concatenated embedding provides a unified representation that includes all modalities. The router and experts work with this comprehensive view, enabling the model to learn correlations and interactions directly from the fused data. However, this approach might not fully capture modality-specific nuances since the characteristics of each modality are blended into a single representation.

Advantages

Captures inter-modal relationships by considering all modalities together.

Simplifies the routing mechanism by treating the concatenated embedding as a single input.

Challenges

May overlook modality-specific features due to the blending of all modalities into one representation.

Could be less efficient if some modalities are irrelevant for certain tasks or experts.

C.2 Modality-Specific Router

Each modality's embedding is independently assigned to a shared pool of experts by modalityspecific routers. This setup allows the model to maintain the distinctiveness of each modality while still leveraging a common pool of expertise. By doing so, it can better capture modality-specific nuances and how they contribute independently to the overall task. However, this approach might be less effective in capturing complex inter-modal interactions since the initial routing is done independently for each modality.

Advantages

Preserves modality-specific information by routing each modality independently.

Flexible in directing modalities to the most relevant experts, potentially improving efficiency.

Captures interactions between modalities to some extent, but may not be as effective as joint routing approaches.

Challenge needs additional coordination between independent routes to leverage cross-modal insights.

C.3 Disjoint Experts & Routers

In this configuration, modality-specific routers assign each modality's embedding to separate pools of experts, with each pool uniquely tailored to process a specific modality type. This method maximizes the ability of the model to capture and exploit modality-specific features and relationships, as each pool of experts is optimized for a particular type of data. However, this setup might limit the model's ability to learn from the interactions between modalities, as each is processed in isolation.

Advantages

Allows for highly specialized processing of each modality, potentially improving performance on modality-specific tasks.

Modality-specific experts can develop deeper insights into the characteristics and patterns within their designated data type.

Challenges

Inter-modal relationships might be underutilized due to the segregated processing of each modality.

Requires additional coordination or subsequent integration stages to combine insights from different modality-specific experts.

Each router type offers unique benefits and faces specific challenges in capturing the subtle relationships between modalities. The choice among them depends on the specific requirements of the application, including the importance of preserving modality-specific information versus capturing inter-modal interactions, and the computational efficiency of managing multiple experts and routers. For example, we found that modality-specific routers are more effective in ameliorating the effect of missing modality in our experiments.

D Data Preprocessing

D.1 MIMIC-IV

In the preprocessing stage, we focused on 30 pertinent lab and chart events from each patient's ICU record for vital sign measurements. For chest X-rays, we utilized a pre-trained DenseNet-121 model [14], which was fine-tuned on the CheXpert dataset [34], to extract 1024-dimensional image embeddings. For radiological notes, we obtained 768-dimensional embeddings using the BioClinicalBERT model [2]. ECG signals were processed using a convolutional autoencoder, adapted from [4], to generate a 256-dimensional embedding for each ECG.

Time series We selected 30 time series events for inclusion, following [83]. Nine of these were vital signs: heart rate, mean/systolic/diastolic blood pressure, respiratory rate, oxygen saturation, and Glascow Coma Scale (GCS) verbal, eye, and motor response. We also included 21 lab values: potassium, sodium, chloride, creatinine, urea nitrogram, bicarbonate, anion gap, hemoglobin, hematocrit, magnesium, platlet count, phosphate, white blood cell count, total calcium, MCH, red blood cell count, MCHC, MCV, RDW, platlet count, neutrophil count, and vancomycin. We standard scale each time series value to have mean 0 and standard deviation 1 , based on the values in the training set.

Chest X-rays To incorporate a medical imaging modality into our analyses, we use the MIMICCXR-JPG [37] module available from Physionet [23], which includes 377,110 JPG format images derived from the DICOM-based MIMIC-CXR database [39]. Following [83], for each image, we resize each JPG image to 224 × 224 pixels and then extract embeddings from the last layer of the Densenet121 model. We identify X-rays taken while the patient was in the ICU by first matching subject IDs in MIMIC-CXR-JPG with the core MIMIC-IV database, then limiting these matched X-rays to those with a chart time occuring between an ICU admission and discharge.

Radiological notes To incorporate text data, we use the MIMIC-IV-Note module [38], which contains 2,321,355 deidentified radiology reports for 237,427 patients that can be matched with patients in the main MIMIC-IV via a similar approach to chest X-rays. We note that we were unable to obtain intermediate clinical notes (i.e., notes made by clinicians throughout a patient stay), as those have not yet been publicly released. We extract note embeddings using Bio-Clinical BERT [2].

Electrocardiograms (ECGs) To include ECGs as an additional modality in our models, we utilize the MIMIC-IV-ECG [24] module, which includes approximately 800,000 ECGs (10 seconds, sampled at 500 Hz) collected from nearly 160,000 unique patients. To transform the ECGs so that they are suitable for input to our model, we adopt a convolutional autoencoder approach, adapted from [4], that compresses each ECG into a 256-dimensional vector. Specifically, each diagnostic ECG contains a 5000 × 12 dimensional vector (5000 time points × 12 ECG leads). To prepare the ECG for input to the autoencoder, we only include the first 4096 time points. We then train the autoencoder to compress the ECG into a 256-dimensional latent vector, and then reconstruct the original ECG using upsampling layers, using mean squared error as our loss function. The architecture is shown in Figure 6. We train the autoencoder with 90% of the ECGs available in the MIMIC-IV-ECG projection and use the rest for validation. We selected a batch size of 2048, and reduced the learning rate by a factor of 0 . 5 if the validation loss had plateaued for 3 epochs. Training stopped if the validation loss had not decreased for 6 epochs. For our encoder, we use filter numbers of [16 , 16 , 32 , 32 , 64 , 64] , kernel widths of [5 , 5 , 5 , 3 , 3 , 3] and a dropout rate of 0 . 1 . For the decoder, we use the same filter numbers and kernel widths in reverse, and maintain a dropout rate of 0 . 1 .

Input ECG

Reconstructed ECG

Temporal conv

Dropout

Batchnorm

Batchnorm

x6

x6

Dropout

Convolution

Max pooling

Upsampling

Reshape

Dense layer

Dense layer

Dropout (0.5)

Dropout (0.5)

Latent vector

(256-dim)

D.2 PAM Dataset

We follow the preprocessing procedure from [100] as published from their official GitHub repository 1 .

E Modeling Irregularity

E.1 Unified Temporal Discretization Embeddings

Unlike the embeddings in chest X-rays, clinical notes, and ECGs, vitals/lab/time-series values present temporal irregularity across dimensions . That is, for the former three modalities, each dimension of the corresponding is observed at each irregular time point τ . By contrast, the sampling for vitals/labs is irregular in both within and across dimensions. For example, we might observe heart rate values sampled at times τ HR = { 0 , 0 . 2 , 0 . 8 , 1 . 2 , 2 . 8 } and glucose values sampled at time τ Glu = { 0 . 1 , 0 . 7 , 3 . 4 } . Given this unique challenge present in vitals/labs, we adapt the Unified Temporal Discretization Embedding (UTDE) approach described in [101], which combines the mTAND approach described in Section 2.2 with a simpler imputation-based discretization scheme. Specifically, given a set of t observations x ∈ R t observed at irregular times τ ∈ R t , we a simple imputation scheme to discretize x into target bins γ (e.g., γ = { 0 , 1 , 2 , ..., γ } . Specifically, given bin value γ i ∈ γ , we apply the following rules:

If there exists a previously observed value of x (i.e., ∃ τ ∈ τ st. τ ≤ γ i ), we set the imputed value of x at time γ , ˆ x γ i , to the closest previously observed value.

If no previously observed value exists, we set the value of ˆ x γ i to the global mean of x .

We do this for each possible vitals/lab, to generate a matrix of imputation embeddings I ∈ R γ × d vitals , were d vitals is the number of vitals/labs. We then input this embedding into a 1D causal convolutional layer with stride 1 to obtain our final imputation embeddings with hidden dimension d h , E Imp ∈ R γ × d h .

E.2 Unifying imputation and mTAND embeddings

We combined simple imputation and mTAND embeddings via a gating function g . Following [101], we let E mTAND ∈ R γ × d h denote the mTAND embeddings for vitals/labs derived from the process described in Section 2.2 and let E Imp ∈ R γ × d h denote the simple imputations from the process described above. We use each of these discretization embeddings to derive a final set of embeddings for vitals/labs E vitals via a one-layer MLP gating function f . Specifically, we let g = f ( E Imp ⊕ E mTAND ) , where ⊕ denotes the concatenation operator. We then calculate E vitals as

where ⊙ denotes point-wise multiplication.

F Baseline Comparison

F.1 MISTS

This approach, from [101], casts time series and clinical notes as multivariate, irregularly-sampled time series (MISTS) and uses layers of self- and cross-attention to fuse modalities. The method uses a Time2Vec [43] encoding scheme to represent the irregularity of observation times. We use the same hyperparameters as in the original paper (e.g., 3 self- and cross-attention blocks, 128 -dimensional time embedding, etc.).

F.2 MulT

This model from [90] relies on multiple stacks of pairwise and bidirectional cross-modal attention blocks (without a self-attention mechanism) to attend to low-level features. The results of crossmodal attention are then sent to modality-specific transformers, concatenated, and used to make predictions.

F.3 MAG

This method introduces the Multimodal Adaptation Gate (MAG) as an extension to BERT and XLNet, allowing these pre-trained models to incorporate visual and acoustic data during fine-tuning.

By generating a modality-conditioned shift in their internal representations, MAG enables enhanced sentiment analysis performance on multimodal datasets, achieving human-level accuracy in the field [74].

F.4 TFN

The proposed Tensor Fusion Network approach (TFN) integrates three core components: Modality Embedding Subnetworks for generating rich embeddings from unimodal inputs, a Tensor Fusion Layer for capturing all levels of modality interactions through a 3-fold Cartesian product, and a Sentiment Inference Subnetwork tailored to perform sentiment analysis based on the fusion layer's output [96].

F.5 HAIM

The multimodal fusion approach detailed by [83] extracts a single set of features for each ICU stay, and uses this to predict the outcome of interest (in-hospital mortality, etc.). For vitals/lab values, the authors extract a set of 11 generic time series features: signal length, maximum, minimum, mean, median, SD, variance, number of peaks, and average time-series slope and piece-wise change over time of these metrics. This is done independently for each of the 30 events, leading to 30 × 11 = 330 vital/lab features per ICU stay. To provide a fair comparison with our method, we only include the most recent five notes and 128 vitals measurements in calculating embeddings. We only include entries for which all modalities are observed. For note/X-ray/ECG embeddings, we compute the mean embedding across all observations occurring during the specified time frame (i.e., the first 48 hours of 48-IHM and LOS, the entire stay for PHE). As with our method, we standardize scale values based on the training set. [83] uses an XGBoost [12] classifier to predict the outcomes of interest. We follow the hyperparameter optimization approach described in the paper. Specifically, we conduct a grid search across the following sets of hyperparameters: max depth = { 5 , 6 , 7 , 8 } , number of estimators = { 200 , 300 } , learning rate = { 0 . 3 , 0 . 1 , 0 . 05 } . Hyperparameters are selected based on the maximum AU-ROC from five-fold cross-validation.

F.6 Implementation

We integrate F.1 through F.4 into our workflow using the implementation provided by [101]. For F.5, we adapt the time series (e.g., series variance, mean, etc.) feature extraction and model fitting code from the repository released by the corresponding paper. The original paper doesn't use ECG waveforms, so we adopt a similar approach to ECG embeddings as with image and note embeddings, and take the mean value of the latent vector across all included observations.

G Computational Resources and Hyper-Parameters

G.1 Computational Resources

We train models using a Lambda Workstation with four A550 GPUs with 24 GB of memory. We are able to train models using a single GPU. An analysis of computation time and memory requirements is shown in Figure 12.

G.2 Hyper-Parameters

The set of parameters we used for experiments can be found in Table 8.

H Additional Results

H.1 FlexiModal Experiments

We present additional results comparing FuseMoE to baselines using the MIMIC-III dataset, which includes only vital signs and clinical notes (Table 9), and the MIMIC-IV dataset, featuring vital signs and CXR (Table 10). All experiments utilize the 'joint experts and router' configuration. In these settings, FuseMoE demonstrates noticeable advantages.

±

±

±

±

H.2 Ablation Study on FuseMoE Building Blocks

In Figure 7, we evaluate the impact of various irregularity encoders on the performance of the FuseMoE framework. Our baseline approaches include the following methods:

employing only the imputation (discretization) module from the time-series irregularity encoder, as detailed in Appendix E

utilizing solely the mTAND module [82] within the time-series irregularity encoder

implementing the SeFT method [31] as an irregularity encoder

adopting the RAINDROP method [100] as an irregularity encoder

±

±

±

±

±

±

±

±

±

±

±

±

(a)

(b)

ComparisonofDifferentIrregularityEncoders

90

Imputation-only

85

mTAND-only

SeFT

80

RAINDROP

FuseMoE

AUROC

75

70

65

60

55

50

48-IHM

LOS

25-PHE

ComparisonofDifferentIrregularityEncoders

80

Imputation-only

mTAND-only

70

SeFT

RAINDROP

60

FuseMoE

Score

50

工

40

30

20

10

48-IHM

LOS

25-PHE

In Figure 8, we evaluate the impact of various time-series encoders on the performance of the FuseMoE framework. The original FuseMoE framework feeds time-series embeddings obtained from the irregularity encoder into the Transformer [92] and extracts the last hidden states of the Transformer output to pass through fully connected layers to make predictions. Our baseline approaches include CNN [47] and LSTM [30] to encode time-series embeddings from the irregularity encoder.

(a)

(b)

Comparison ofDifferent TimeSeriesEncoders

90

CNN

LSTM

85

Transformer

80

AUROC

75

70

65

60

48-IHM

LOS

25-PHE

ComparisonofDifferentTimeSeriesEncoders

80

CNN

LSTM

70

Transformer

09

Score

50

40

30

20

48-IHM

LOS

25-PHE

In Figure 9, we assess the effect of different CXR encoders on the FuseMoE framework. Currently, the FuseMoE framework incorporates DenseNet-121 as the feature extractor for CXR images before their integration into the mTAND module. This setup is compared with the application of the stateof-the-art vision transformer (ViT-B) [17] as an alternative CXR encoder.

In Figure 10, we evaluate the influence of text encoders on the FuseMoE framework. Currently, FuseMoE incorporates Clinical-Longformer [49] as the text encoder before integrating it into the mTAND module. This setup is compared with other state-of-the-art text encoders: GRU-D [10], FT-LSTM [99], and HierTrans [68].

Finally, in Figure 11, we investigate the effect of the mTAND module on each modality, while we removed mTAND for a particular modality, the rest of FuseMoE's components remained constant.

H.3 Ablation Study on MoE Architecture

We then conducted ablation studies to explore the efficiency and effectiveness of MoE architecture on model performance. We mainly use MIMIC-IV as our test bed. Figure 12(a) examines the computational efficiency and resource utilization, positioning FuseMoE approximately in the middle of

(a)

Comparison of Different CxR Encoders

80

DenseNet-121

ViT-B

70

Score

60

1

F

50

40

30

48-IHM

LOS

25-PHE

ComparisonofDifferentCxREncoders

90.0

DenseNet-121

87.5

ViT-B

85.0

AUROC

82.5

80.0

77.5

75.0

72.5

70.0

48-IHM

LOS

25-PHE

(b)

(a)

(b)

ComparisonofDifferentTextEncoders

90 r

GRU-D

FT-LSTM

85

HierTrans

Clinical-Longformer

08

AUROC

75

70

65

60

48-IHM

LOS

25-PHE

ComparisonofDifferentTextEncoders

80r

GRU-D

FT-LSTM

70

HierTrans

Clinical-Longformer

60

Score

50

40

30

20

48-IHM

LOS

25-PHE

(a)

(b)

EffectofIrregularityEncoderonEachModality

90

85

FuseMoE

80

AUROC

70

65

60

48-IHM

LOS

25-PHE

EffectofIrregularityEncoderonEachModality

80

FuseMoE

Score

09

50

40

30

48-IHM

LOS

25-PHE

(a)

(b)

(c)

2000

85

0.5

Vital

Task / Model

1800

■

48-IHM

25-PHE

80

Weight

0.4

Vital

Note

Vital

ation

1600

MISTS

Note

TF

AUROC

0.3

Note

Com

uta

ndu

1200

1400

MAG

FuseMoE

MulT

75

Normalized

0.2

CXR

CXR

70

1000

LOS

Averag

48-IHM

0.1

ECG

CXR

ECG

800

65

25-PHE

ECG

150020002500300035004000

4500

5

10

15

20

25

0.0

48-IHM

LOS

25-PHE

MaxGPU Memory(MiB)

TotalNumberofExperts

Tasks

Audio

Audio

20.1%

32.5%

36.8%

Language

43.1%

39.3%

28.2%

Video

Language

Video

CMU-MOSIdataset

CMU-MOSEIdataset

the comparison. Despite the increase in model parameters due to the incorporation of the MoE layer, its sparse nature does not significantly escalate the computational load. Figure 12(b) illustrates the correlation between the number of experts and task performance across different modalities. Generally, performance improves with the addition of more experts, plateauing once the count exceeds 16. To achieve a compromise between performance and computational expense, we opted to utilize the top 4 experts out of 16 in our experiments. Figure 12(c) and Figure 13 study the influence of each modality on the topk chosen experts. For every expert selected, we calculate the number of samples that include a specific modality, weighted by corresponding weight factors from the gating functions. The outcomes are subsequently normalized across modalities. The analysis of Figure 12(c) reveals that predictions across all tasks heavily depend on vital signs and clinical notes. This reliance is attributed to the abundant samples in these two modalities. Despite the notably smaller quantity of CXR, they play more significant roles in the 25-PHE and 48-IHM tasks, which aligns with our findings in Table 5. The results in Figure 13 demonstrate that the modality weight distribution in the MOSI and MOSEI datasets is more 'spread out', with the audio component carrying a greater weight in the MOSEI dataset.

I Details on Numerical Experiments

We conduct multiple numerical experiments to illustrate the theoretical convergence rates of the MLE ̂ G n to the true mixing measure G ∗ under both exact-specified and over-specified settings. I.1 Experimental Setup

Synthetic Data. Assume that the true mixing measure G ∗ = ∑ k ∗ i =1 exp( β ∗ i ) δ ( W ∗ i ,a ∗ i ,b ∗ i ,ν ∗ i ) is of order k ∗ = 2 . The true parameters for the router, ( W ∗ i , β ∗ i ) ∈ R d × R , are drawn independently from an isotropic Gaussian distribution with zero mean and variance σ 2 r = 0 . 01 /d for 1 ≤ i ≤ 6 , and otherwise are set to zero. Similarly, the true parameters of the experts, ( a ∗ i , b ∗ i ) ∈ R d × R , are drawn independently of an isotropic Gaussian distribution with zero mean and variance σ 2 e = 1 /d

for all experts. For the variances ν ∗ i , we also sample from the Gaussian distribution N (0 , σ 2 e ) , and then take the absolute value of the sample. These parameters remain unchanged for all experiments.

Then, we generate i.i.d samples { ( X i , Y i ) } n i =1 by first sampling X i 's from the uniform distribution Uniform[0 , 1] and then sampling Y i 's from the true conditional density g G ∗ ( Y | X ) of the Laplace gating Gaussian mixture of experts (MoE) given in equation 4.

Maximum Likelihood Estimation (MLE). A popular approach to determining the MLE ̂ G n for each set of samples is to use the EM algorithm [15]. However, since there are not any closedform expressions for updating the gating parameters β 0 i , β 1 i in the maximization steps, we have to leverage an EM-based numerical scheme, which was previously used in [63]. In particular, we utilize a simple coordinate gradient descent algorithm in the maximization steps. Additionally, we select the convergence criterion of ϵ = 10 -6 and run a maximum of 2000 EM iterations.

Initialization. For each k ∈ { k ∗ , k ∗ +1 } , we randomly distribute elements of the set { 1 , 2 , ..., k } into k ∗ different Voronoi cells A 1 , A 2 , . . . , A k ∗ , each contains at least one element. Moreover, we repeat this process for each replication. Subsequently, for each j ∈ [ k ∗ ] , we initialize parameters W i by sampling from a Gaussian distribution centered around its true counterpart W ∗ j with a small variance, where i ∈ A j . Other parameters β i , a i , b i , ν i are also initialized in a similar fashion.

I.2 Exact-specified Setting

Under the exact-specified settings, we conduct 5 sample generations for each configuration, across a spectrum of 10 different sample sizes n ranging from 10 3 to 10 5 . It can be seen from Figure 3 (left) that the MLE ̂ G n empirically converges to the true mixing measure G ∗ under the Voronoi metric D 1 at the rate of order O ( n -0 . 49 ) , which matches the theoretical parametric convergence rate established in Theorem J.2.

I.3 Over-specified Setting

Under the over-specified settings, we continue to generate 5 samples of size n for each setting, given 10 different choices of sample size n ∈ [10 3 , 10 5 ] . From Figure 3 (right), we observe that the MLE ̂ G n empirically converges to G ∗ under the Voronoi metric D 2 at the rate of order O ( n -0 . 47 ) , which aligns with the theoretical parametric convergence rate established in Theorem 3.2.

J Exact-Specified Setting

In this appendix, we study the theoretical behaviors of the MLE under the exact-specified setting, i.e., k = k ∗ , of the Laplace gating Gaussian MoE. We demonstrate that under the exact-specified setting, the rate of estimated conditional density function p ̂ G n to p G ∗ is parametric O n -1 / 2 ) (up to some logarithmic factor).

Theorem J.1. The density estimation p ̂ G n ( Y | X ) converges to the true density p G ∗ ( Y | X ) under the Total Variation distance V at the following rate:

The proof of Theorem J.1 can be done similarly to that of Theorem 3.1 in Appendix K.2. The result of Theorem J.1 indicates that as long as we can establish the lower bound of the total variation distance between p ̂ G n and p G ∗ based on certain loss function between the MLE ̂ G n and the true mixing measure G ∗ , we directly achieve the rate of the MLE under that loss function.

Voronoi Loss We now define that loss function between the MLE and the true mixing measure for the exact-specified setting:

Above, for any ( ρ 1 , ρ 2 , ρ 3 , ρ 4 ) ∈ R 4 , we define Φ ij ( ρ 1 , ρ 2 , ρ 3 , ρ 4 ) = ∥ W i -W ∗ j ∥ ρ 1 + ∥ a i -a ∗ j ∥ ρ 2 + | b i -b ∗ j | ρ 3 + | ν i -ν ∗ j | ρ 4 for any i ∈ A j and j ∈ [ k ∗ ] . We demonstrate in the following theorem that

the rate of MLE to the true mixing measure under the Voronoi loss function D 1 is O ( n -1 / 2 ) (up to some logarithmic factor).

Theorem J.2 (Exact-specified setting) . When k = k ∗ is known, the following Total Variation bound holds guarantetrue for any G ∈ G k (Θ) :

Therefore, we have D 1 ( ̂ G n , G ∗ ) = O ( √ log( n ) /n ) . Proof of Theorem J.2 is in Appendix K.1. The convergence rate of MLE under the Voronoi loss function D 1 implies that the rates of estimating the true parameters W ∗ i , a ∗ i , b ∗ i , ν ∗ i are also O ( n -1 / 2 ) (up to logarithmic factors). These rates are comparable to those under the exact-specified setting of softmax gating Gaussian MoE (cf. Theorem 1 in [64]).

K Proof of Theoretical Results

In this appendix, we provide proofs for all theoretical results in the paper. Throughout this appendix, for any vector v ∈ R d and α := ( α 1 , α 2 , . . . , α d ) ∈ N d , we denote v α = v α 1 1 v α 2 2 . . . v α d d , | v | := v 1 + v 2 + . . . + v d and α ! := α 1 ! α 2 ! . . . α d ! .

K.1 Proof of Theorem J.2

First of all, we need to establish the following bound:

For that sake, it is sufficient to demonstrate two following inequalities:

for some constant ε ′ > 0 .

Proof of inequality A : The inequality A is equivalent to

Assume that the above inequality is not true, then, there exists a sequence of mixing measure G n := ∑ k ∗ i =1 exp( β n i ) δ ( W n i ,a n i ,b n i ,ν n i ) ∈ G k ∗ (Θ) such that both D 1 ( G n , G ∗ ) and E X [ V ( p G n ( ·| X ) , p G ∗ ( ·| X ))] / D 1 ( G n , G ∗ ) go to zero as n →∞ . Now, we define

̸

for any j ∈ [ k ∗ ] as Voronoi cells with respect to the mixing measure G n , where we denote θ n i := ( W n i , a n i , b n i , ν n i ) and θ ∗ j := ( W ∗ j , a ∗ j , b ∗ j , ν ∗ j ) . In this proof, since our arguments are assymptotic, we can assume without loss of generality (WLOG) that these Voronoi cells does not depend on n , that is, A j = A n j . Next, it follows from the hypothesis D 1 n := D 1 ( G n , G ∗ ) → 0 as n → ∞ that each Voronoi cell contains only one element. Therefore, we may assume WLOG that A j = { j } for any j ∈ [ k ∗ ] , which implies that ( W n j , a n j , b n j , ν n j ) → ( W ∗ j , a ∗ j , b ∗ j , ν ∗ j ) and exp( β n j ) → exp( β ∗ j ) as n →∞ . Then, the loss function between G n and G ∗ is given by

Now, we break the rest of our arguments into three steps:

Stage 1 - Density decomposition :

In this step, we aim to decompose the term Q n := [ ∑ k ∗ i =1 exp( -∥ W ∗ i -X ∥ + β ∗ i ) ] · [ p G n ( Y | X ) -p G ∗ ( Y | X )] , which can be represented as follows:

where we denote F ( Y | X ; W,a,b,ν ) := exp( -∥ W -X ∥ ) f ( Y | a ⊤ X + b, ν ) and H ( Y | X ; W ) = exp( -∥ W -X ∥ ) p G n ( Y | X ) . By applying the first-order Taylor expansion, we can rewrite A n as

where R 1 ( X,Y ) is a Taylor remainder that satisfies R 1 ( X,Y ) / D 1 ( X,Y ) → 0 as n → ∞ and g ( X,W ) := exp( ∥ W -X ∥ ) . Recall that f is the univariate Gaussian density, then by denoting h 1 ( X ; a, b ) := a ⊤ X + b , we can verify that

Consequently, we get

where we denote η = α 3 +2 α 4 ∈ N .

Subsequently, we also apply the first-order Taylor expansion to the term B n defined in equation 10 and get that

where R 2 ( X,Y ) is a Taylor remainder such that R 2 ( X,Y ) / D 1 ( G n , G ∗ ) → 0 as n →∞ . From the above results, the term Q n can be rewritten as

in which we respectively define for each i ∈ [ k ∗ ] that

for any ( α 1 , α 2 , η ) = ( 0 d , 0 d , 0) and γ = 0 d . Otherwise, S n i, 0 d , 0 d , 0 = T n i, 0 d := exp( β n i ) -exp( β ∗ i ) .

̸

̸

Stage 2 - Non-vanishing coefficients :

Moving to the second step, we will show that not all the ratios S n i,α 1 ,α 2 ,η / D 1 ( G n , G ∗ ) and T n i,γ / D 1 ( G n , G ∗ ) tend to zero as n → ∞ . Assume by contrary that all of them approach zero when n →∞ , then for ( α 1 , α 2 , η ) = ( 0 d , 0 d , 0) , it follows that

Additionally, for tuples ( α 1 , α 2 , η ) where α 1 ∈ { e 1 , e 2 , . . . , e d } with e j := (0 , . . . , 0 , 1 ︸︷︷︸ j -th , 0 , . . . , 0) , α 2 = 0 d and η = 0 , we get

For ( α 1 , α 2 , η ) where α 1 = 0 d , α 2 ∈ { e 1 , e 2 , . . . , e d } and η = 0 , we have

For ( α 1 , α 2 , η ) where α 1 = α 2 = 0 d and η = 1 , we have

For ( α 1 , α 2 , η ) where α 1 = α 2 = 0 d and η = 2 , we have

As a result, we achieve that

Due to the topological equivalence between norm-1 and norm-2, the above limit implies that

Combine equation 14 with equation 15, we deduce that D 1 ( G n , G ∗ ) / D 1 ( G n , G ∗ ) → 0 , which is a contradiction. Consequently, at least one among the ratios S n i,α 1 ,α 2 ,η / D 1 ( G n , G ∗ ) and T n i,γ / D 1 ( G n , G ∗ ) does not vanish as n tends to infinity.

Stage 3 - Fatou's contradiction :

In this step, we use the Fatou's lemma to point out a contradiction to the results achieved in Step 2. In particular, we denote by m n the maximum of the absolute values of S n i,α 1 ,α 2 ,η / D 1 ( G n , G ∗ ) and T n i,γ / D 1 ( G n , G ∗ ) . Since at least one of the previous ratios does not converge to zero, we deduce that 1 /m n ̸→∞ .

Recall from the hypothesis that E X [ V ( p G n ( ·| X ) , p G ∗ ( ·| X ))] / D 1 ( G n , G ∗ ) → 0 as n → ∞ . According to the Fatou's lemma, we have

This result indicates that | p G n ( Y | X ) -p G ∗ ( Y | X ) | / D 1 ( G n , G ∗ ) tends to zero as n goes to infinity for almost surely ( X,Y ) . As a result, it follows that

Next, let us denote S n i,α 1 ,α 2 ,η / [ m n D 1 ( G n , G ∗ )] → ξ i,α 1 ,α 2 ,η and T n i,γ / [ m n D 1 ( G n , G ∗ )] → κ i,γ with a note that at least one among them is non-zero. From the formulation of Q n in equation 13, we deduce that

for almost surely ( X,Y ) . The above equation is equivalent to

for almost surely ( X,Y ) . It is worth noting that parameters W ∗ 1 , . . . , W ∗ K are pair-wise distinct, thus, the set { ∂ | α 1 | g ∂W α 1 ( X ; W ∗ i ) : i ∈ [ k ∗ ] , 0 ≤ | α 1 | ≤ 1 } is a linearly independent, which implies that

for any i ∈ [ k ∗ ] , 0 ≤ | α 1 | ≤ 1 for almost surely ( X,Y ) . Moreover, since ( a ∗ 1 , b ∗ 1 , ν ∗ 1 ) , . . . , ( a ∗ k ∗ , b ∗ k ∗ , ν ∗ k ∗ ) have pair-wise distinct values, those of (( a ∗ 1 ) ⊤ X + b ∗ 1 , ν ∗ 1 ) , . . . , (( a ∗ k ∗ ) ⊤ X + b ∗ k ∗ , ν ∗ k ∗ ) are also pair-wise different. Therefore, the set

is also linearly independent. Consequently, we obtain that ξ i,α 1 ,α 2 ,η = κ i,γ = 0 for any i ∈ [ k ∗ ] , 0 ≤ | α 1 | + α 2 ≤ 1 , 0 ≤ η ≤ 2(1 -| α 1 | - | α 2 | ) and 0 ≤ | γ | ≤ 1 , which contradicts the fact that at least one among those terms is different from zero.

Hence, we can find some constant ε ′ > 0 such that

Proof of inequality B : Assume by contrary that the inequality B does not hold, then there exists a sequence of mixing measures G ′ n ∈ G k ∗ (Θ) such that D 1 ( G ′ n , G ∗ ) > ε ′ and

This result leads to E X [ V ( p G ′ n ( ·| X ) , p G ∗ ( ·| X ))] → 0 as n → ∞ . Recall that Ω is a compact set, therefore, we can replace the sequence G ′ n by one of its subsequences that converges to a mixing measure G ′ ∈ G k ∗ (Θ) . Since D 1 ( G ′ n , G ∗ ) > ε ′ , this result induces that D 1 ( G ′ , G ∗ ) > ε ′ .

Subsequently, by means of the Fatou's lemma, we achieve that

It follows that p G ′ ( Y | X ) = p G ∗ ( Y | X ) for almost surely ( X,Y ) . According to Lemma L.1, the noisy top-K sparse softmax gating Gaussian mixture of experts is identifiable, thus, we obtain that G ′ ≡ G ∗ . As a consequence, we obtain that D 1 ( G ′ , G ∗ ) = 0 , which contradicts to the fact that D 1 ( G ′ , G ∗ ) > ε ′ > 0 .

Hence, the proof is completed.

K.2 Proof of Theorem 3.1

In this appendix, we employ results for M-estimators in [91] to establish the density estimation rate under the Laplace gating Gaussian mixture of experts (MoE).

Firstly, we introduce some necessary notations and fundamental results. In particular, let P k (Θ) := { p G ( Y | X ) : G ∈ G k (Θ) } be the set of all conditional density functions w.r.t mixing measures in G k (Θ) . Next, we denote by N ( ε, P k (Θ) , ∥ · ∥ ∞ ) the covering number of metric space ( P k (Θ) , ∥ · ∥ ∞ ) . Meanwhile, H B ( ε, P k (Θ) , h ) stands for the bracketing entropy of P k (Θ) under the Hellinger distance h where h ( p, q ) := ( 1 2 ∫ ( √ p - √ q ) 2 dµ ) 1 / 2 for any probability densities p, q dominated by the Lebesgue measure µ . Then, we provide in the following lemma the upper bounds of those terms.

Lemma K.1. If Θ is a bounded set, then the following inequalities hold for any 0 < η < 1 / 2 :

(i) log N ( η, P k (Θ) , ∥ · ∥ ∞ ) ≲ log(1 /η ) ;

(ii) H B ( η, P k (Θ) , h ) ≲ log(1 /η ) .

Proof of Lemma K.1 is in Appendix K.2.2. Subsequently, we denote

In addition, for each δ > 0 , we define a Hellinger ball centered around the conditional density function p G ∗ ( Y | X ) and intersected with the set P 1 / 2 k (Θ) as

To capture the size of the above Hellinger ball, [91] suggest using the following quantity:

where t ∨ δ := max { t, δ } . Given those notations, let us recall a standard result for density estimation in [91].

Lemma K.2 (Theorem 7.4, [91]) . Take Ψ( δ ) ≥ J B ( δ, ˜ P 1 / 2 k (Θ , δ )) such that Ψ( δ ) /δ 2 is a nonincreasing function of δ . Then, for some sequence ( δ n ) and universal constant c which satisfy √ nδ 2 n ≥ c Ψ( δ ) , we obtain that for any δ ≥ δ n

Proof of Lemma K.2 can be found in [91]. Now, we are ready to provide the proof for convergence rate of density estimation in Theorem J.1 in Appendix K.2.1.

K.2.1 Main Proof

It is worth noting that for any t > 0 , we have

Then, the integral in equation 17 is upper bounded as follows:

As a result, by choosing Ψ( δ ) = δ · √ log(1 /δ ) , we can verify that Ψ( δ ) /δ 2 is a non-increasing function of δ . Furthermore, the inequality in equation 18 indicates that Ψ( δ ) ≥ J B ( δ, ˜ P 1 / 2 k (Θ , δ )) . Next, let us consider a sequence ( δ n ) defined as δ n := √ log( n ) /n . This sequence can be validated to satisfy the condition √ nδ 2 n ≥ c Ψ( δ ) for some universal constant c . Therefore, by Lemma K.2, we reach the conclusion of Theorem J.1:

for some universal constant C depending only on Θ .

K.2.2 Proof of Lemma K.1

Part (i). In this part, we will derive the following upper bound for the covering number of metric space ( P k (Θ) , ∥ · ∥ ∞ ) for any 0 < η < 1 / 2 given the bounded set Ω :

To start with, we denote Ω := { ( a, b, ν ) ∈ R d × R × R + : ( β, W, a, b, ν ) ∈ Ω } . As Θ is a bounded set, the set Ω is also bounded. Therefore, we can find an η -cover of Ω , denoted by Ω η . Additionally, we also define ∆ := { ( β, W ) ∈ R × R d : ( β, W, a, b, ν ) ∈ Ω } , and ∆ η be an η -cover of ∆ . Then, it can be validated that

Next, for each mixing measure G = ∑ k i =1 exp( β i ) δ ( W i ,a i ,b i ,ν i ) ∈ G k (Θ) , we take into account two other mixing measures. The first measure is G ′ = ∑ k i =1 exp( β i ) δ ( W i ,a i ,b i ,ν i ) , where

( a i , b i , ν i ) ∈ Ω η is the closest points to ( a i , b i , ν i ) in this set for all i ∈ [ k ] . The second one is G := ∑ k i =1 exp( β i ) δ ( W i ,a i ,b i ,ν i ) in which ( β i , W i ) ∈ ∆ η for any i ∈ [ k ] . Next, let us define then it is obvious that p G ∈ T . Now, we will show that T is an η -cover of metric space ( P k (Θ) , ∥ · ∥ ∞ ) with a note that it is not necessarily the smallest cover. Indeed, according to the triangle inequality, we have

Since the softmax function is no greater than one, the first term in the right hand side can be upper bounded as follows:

Subsequently, we bound the second term ∥ p G ′ -p G ∥ ∞ as follows:

It follows from the results in equation 19, equation 20 and equation 21 that ∥ p G -p G ∥ ∞ ≲ η . This result indicates that T is an η -cover of the metric space ( P k (Θ) , ∥ · ∥ ∞ ) . As a consequence, we obtain that

Part (ii). In this part, we provide an upper bound for the bracketing entropy of P k (Θ) under the Hellinger distance h :

which leads to the conclusion of this part: log N ( η, P k (Θ) , ∥ · ∥ ∞ ) ≲ log(1 /η ) .

Since Θ and X are bounded sets, there exist positive constants γ, ℓ, u such that -γ ≤ a ⊤ X + b ≤ γ and ℓ ≤ ν ≤ u . Let us define

Then, it can be validated that f ( Y | a ⊤ X + b, ν ) ≤ B ( Y | X ) for any ( X,Y ) ∈ X × Y .

Next, let ζ ≤ η which will be chosen later and { p 1 , . . . , p N } be an ζ -cover of metric space ( P k (Θ) , ∥ · ∥ ∞ ) with the covering number N := N ( ζ, P k (Θ) , ∥ · ∥ ∞ ) . Additionally, we also consider brackets of the form [Ψ L i ( Y | X ) , Ψ U i ( Y | X )] where

Then, we can check that P k (Θ) ⊆ ⋃ N i =1 [Ψ L i ( Y | X ) , Ψ U i ( Y | X )] and Ψ U i ( Y | X ) -Ψ L i ( Y | X ) ≤ min { 2 ζ, B ( Y | X ) } .

Let S := max { 2 γ, √ 8 u } log(1 /ζ ) , we have for any i ∈ [ N ] that where S ′ is some positive constant. This inequality indicates that

By setting ζ = η/S ′ , we obtain that H B ( η, P k (Θ) , ∥·∥ 1 ) ≲ log(1 /η ) . Finally, due to the inequality h 2 ≤ ∥ · ∥ 1 , we reach the conclusion of this part:

Hence, the proof is completed.

K.3 Proof of Theorem 3.2

In order to establish the following Total Variation lower bound under the over-specified settings, i.e. when k > k ∗ is unknown:

we need to prove two following inequalities:

for some constant ε ′ > 0 . As the inequality B can be achieved in the same fashion as in Appendix K.1, we concentrate on showing the inequality A in this proof. For that purpose, it suffices to prove that

Assume that the above claim does not hold true, then there exists a sequence of mixing measures G n := ∑ k n i =1 exp( β n i ) δ ( W n i ,a n i ,b n i ,ν n i ) ∈ G k (Θ) such that both the terms D 2 ( G n , G ∗ ) and E X [ V ( p G n ( ·| X ) , p G ∗ ( ·| X ))] / D 2 ( G n , G ∗ ) go to zero as n → ∞ . Let us recall the formulation of the loss D 2 ( G n , G ∗ ) :

Now, we reuse the three-step framework in Appendix K.1.

Stage 1 - Density decomposition :

Firstly, by abuse of notations, let us consider the quantity

Similar to Step 1 in Appendix K.1, we can express this term as

Next, we proceed to decompose A n based on the cardinality of the Voronoi cells as follows:

By applying the Taylor expansions of order 1 and ¯ r j to the first and second terms of A n , respectively, and following the derivation in equation 11, we get that

where R i ( X,Y ) is a Taylor remainder such that R i ( X,Y ) / D 2 ( G n , G ∗ ) → 0 as n → ∞ for i ∈ { 3 , 4 } . Next, we apply the Taylor expansions of order 1 and 2 to the first and second terms of B n , respectively, and following the derivation in equation 12, we get that

where R 5 ( X,Y ) and R 6 ( X,Y ) are Taylor remainders such that their ratios over D 2 ( G n , G ∗ ) approach zero as n →∞ . Subsequently, let us define

̸

for any ( α 1 , α 2 , η ) = ( 0 d , 0 d , 0) and γ = 0 d . Otherwise, S n j, 0 d , 0 d , 0 = T n j, 0 d := ∑ i ∈A j exp( β n i ) -exp( β ∗ j ) . As a consequence, it follows that

̸

Stage 2 - Non-vanishing coefficients :

In this step, we demonstrate that not all the ratios S n j,α 1 ,α 2 ,η / D 2 ( G n , G ∗ ) and T n j,γ / D 2 ( G n , G ∗ ) converge to zero as n →∞ . Assume by contrary that all these terms go to zero. Then, by employing arguments for deriving equation 14 and equation 15, we get that

Taking the summation of ∑ j : |A j | > 1 | S n j,α 1 ,α 2 ,η | D 2 ( G n ,G ∗ ) for all ( α 1 , α 2 , η ) where α 1 ∈ { 2 e 1 , 2 e 2 , . . . , 2 e d } , α 2 = 0 d and η = 0 , we have

Taking the summation of ∑ j : |A j | > 1 | S n j,α 1 ,α 2 ,η | D 2 ( G n ,G ∗ ) for all ( α 1 , α 2 , η ) where α 1 = 0 d , α 2 ∈ { 2 e 1 , 2 e 2 , . . . , 2 e d } and η = 0 , we have

Combine the above limit with the formulation of D 2 ( G n , G ∗ ) in equation 23, we have that

This result implies that we can find some index j ′ ∈ [ k ∗ ] : |A j ′ | > 1 that satisfies

For simplicity, we may assume that j ′ = 1 . Since S n 1 , 0 d , 0 d ,η / D 2 ( G n , G ∗ ) vanishes as n → ∞ for any 1 ≤ η ≤ ¯ r j , we divide this term by the left hand side of the above equation and achieve that

for any 1 ≤ η ≤ ¯ r 1 .

Subsequently, we define M n := max {| ∆ b n i 1 | , | ∆ ν n i 1 | 1 / 2 : i ∈ A 1 } and π n := max { exp( β n i ) : i ∈ A 1 } . As a result, the sequence exp( β n i ) /π n is bounded, which indicates that we can substitute it with its subsequence that admits a positive limit z 2 5 i := lim n →∞ exp( β n i ) /π n . Therefore, at least one among the limits z 2 5 i equals to one. Furthermore, we also denote

From the above definition, it follows that at least one among the limits z 3 i and z 4 i equals to either 1 or -1 . By dividing both the numerator and the denominator of the term in equation 25 by π n M η n , we arrive at the following system of polynomial equations:

for all 1 ≤ η ≤ ¯ r 1 . Nevertheless, from the definition of ¯ r 1 , we know that the above system does not admit any non-trivial solutions, which is a contradiction. Consequently, not all the ratios S n j,α 1 ,α 2 ,η / D 2 ( G n , G ∗ ) and T n j,γ / D 2 ( G n , G ∗ ) tend to zero as n →∞ .

Stage 3 - Fatou's contradiction :

Recall that E X [ V ( p G n ( ·| X ) , p G ∗ ( ·| X ))] / D 2 ( G n , G ∗ ) → 0 as n → ∞ . Then, by applying the Fatou's lemma, we get

which implies that | p G n ( Y | X ) -p G ∗ ( Y | X ) | / D 2 ( G n , G ∗ ) → 0 as n →∞ for almost surely ( X,Y ) .

Next, we define m n as the maximum of the absolute values of S n j,α 1 ,α 2 ,η / D 2 ( G n , G ∗ ) . It follows from Step 2 that 1 /m n ̸→∞ . Moreover, by arguing in the same way as in Step 3 in Appendix K.1, we receive that

as n →∞ . By abuse of notations, let us denote

Here, at least one among ξ j,α 1 ,α 2 ,η , κ j,γ is non-zero. Then, by putting the results in equation 24 and equation 26 together, we get

Arguing in a similar fashion as in Step 3 of Appendix K.1, we obtain that ξ j,α 1 ,α 2 ,η = κ j,γ = 0 for any j ∈ [ k ∗ ] , 0 ≤ | α 1 | + | α 2 | ≤ 2¯ r j , 0 ≤ η ≤ 2(¯ r j -| α 1 | - | α 2 | ) and 0 ≤ | γ | ≤ 1 + 1 {|A j | > 1 } . This contradicts the fact that at least one among them is non-zero. Hence, the proof is completed.

L Identifiability of the Laplace Gating Gaussian MoE

Lemma L.1. For any mixing measures G and G ∗ in G k (Θ) that satisfy p G ( Y | X ) = p G ∗ ( Y | X ) for almost surely ( X,Y ) ∈ X × Y , we have that G ≡ G ∗ .

Proof of Lemma L.1. First, we assume that two mixing measures G and G ∗ take the following forms: G = ∑ k i =1 exp( β i ) δ ( W i ,a i ,b i ,ν i ) and G ∗ = ∑ k ∗ i =1 exp( β ∗ i ) δ ( W ∗ i ,a ∗ i ,b ∗ i ,ν ∗ i ) . Recall that

p G ( Y | X ) = p G ∗ ( Y | X ) for almost surely ( X,Y ) , then we have

Due to the identifiability of the location-scale Gaussian mixtures [85, 86, 87], we get that k = k ∗ and for almost surely X . WLOG, we may assume that

for almost surely X for any i ∈ [ k ] . Since the softmax function is invariant to translations, it follows from equation 28 that W i = W ∗ i and β i = β ∗ i + v 0 for some v 0 ∈ R . Notably, from the assumption of the model, we have β k = β ∗ k = 0 , which implies that v 0 = 0 . As a result, we obtain that β i = β ∗ i for any i ∈ [ k ∗ ] . Then, equation 27 can be rewritten as

̸

for almost surely ( X,Y ) . Next, we denote J 1 , J 2 , . . . , J m as a partition of the index set [ k ∗ ] , where m ≤ k ∗ , such that exp( β i ) = exp( β i ′ ) for any i, i ′ ∈ J j and j ∈ [ m ] . On the other hand, when i and i ′ do not belong to the same set J j , we let exp( β i ) = exp( β i ′ ) . Thus, we can reformulate equation 29 as

for almost surely ( X,Y ) . This results leads to { (( a i ) ⊤ X + b i , ν i ) : i ∈ J j } ≡ { (( a ∗ i ) ⊤ X + b ∗ i , ν ∗ i ) : i ∈ J j } , for almost surely X for any j ∈ [ m ] . Therefore, we have

for any j ∈ [ m ] . As a consequence,

Hence, we reach the conclusion of this lemma.

M Broader Impact

This paper presents research aimed at propelling advancements in the broad domain of machine learning. The implications of our findings are wide-ranging, with potential applications in sectors including healthcare, autonomous driving, and recommendation systems. Based on our current understanding, this research does not warrant an ethics review, and a detailed discussion of the potential societal impacts is not required at the current stage.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: we have added a separate contribution paragraph in the introduction section (section 1), along with the highly summarized contributions in the abstract.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: see section 5 on discussion on general limitations. Section 3 and 4 have also thoroughly discussed the assumptions and experimental settings.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: see section 3 and Appendix J, K and L for assumptions and proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: we have submitted the implementation of the proposed methods and all baselines in the supplementary material. Appendix B, D, F and G also contain all dataset information and implementation details.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: we have submitted the implementation of the proposed methods and all baselines in the supplementary material. Appendix B, D, F and G also contain comprehensive information to reproduce the experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: dataset information and preprocessing procedure can be found in Appendix B and D.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: all results are averaged over 5 random experiments, as specified by the experiment section 4 and additional results in Appendix H.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: computational resources used to run the experiments can be found in Appendix G.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: we have reviewed the NeurIPS Code of Ethics and make sure the paper conforms to this.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: we have attempted to discuss the impacts of this work in Appendix M.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: our paper has no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: all the assets used in the paper have been properly credited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: we have provided documents in the code repository submitted in the supplementary material.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: this paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: this paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper “FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion” introduces a novel framework called FuseMoE, designed to tackle challenges associated with multimodal data in machine learning. This framework addresses key issues such as missing elements, temporal irregularity, and sparsity in data, which are prevalent in fields like healthcare where data is often incomplete and irregularly sampled. The core innovation in FuseMoE is its gating function that integrates various modalities efficiently. The framework enhances predictive performance through: 1. Sparse MoE Layers: Incorporating sparsely gated MoE layers to manage tasks and learn optimal modality partitioning. 2. Laplace Gating Function: A novel gating function theoretically proven to ensure better convergence rates than traditional Softmax functions. 3. FlexiModal Data Handling: Efficiently handling scenarios with missing modalities and irregularly sampled data trajectories. The framework is validated through empirical evaluations on diverse prediction tasks, demonstrating its effectiveness in real-world applications.\n\nSTRENGTHS:\n1. Existing demand of a more flexible gating function in the field of multimodal MoE makes the idea of this paper promising. Also, the possible implementations of this new MoE system is promising and may lead to some practical medical tools.\n2. The theoretical proof is sufficient and rigorous.\n3. The experimental design is comprehensive from the perspectives of scenarios and data, to a certain extent corresponding with the theory. The appendix contains a large amount of supplementary information, which helps verify the completeness and correctness of the article.\n\nWEAKNESSES:\n1. The comparison works are all very outdated (the latest one UTDE[1] is publicized in 2023, with over half of the compared works predating 2020). Why not compare with more recent works?\n2. Fig.1 lacks clear description, as the data processing workflow is not well explained.\n3. Gaussian gating and your proposed form are very similar in format, with formula $h(x)=\\operatorname{Top} \\mathrm{K}\\left(-\\|W-x\\|_{2}^{2}\\right)$. Thus, theoretical comparison with original softmax may be unfair as the form is derived from and similar to gaussian gating. Therefore, although theoretical results may be valid, their importance diminishes due to this aspect.\n\n[1] Improving Medical Predictions by Irregular Multimodal Electronic Health Records Modeling\n\nQUESTIONS:\nPlease refer to the weaknesses.\nThe biggest question at my side is that the similarity with gaussian gating may lead to some theoretical misunderstanding. Any further explanation on this topic could lead to higher score.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations, there is some redundancy in the encoder part that heavily draws inspiration from UTDE's work. From the appendix it can be observed that encoder might have a greater impact on overall performance than what gate function alone contributes to improvement; hence further discussion is needed on whether this gate function must necessarily be tied to methods related to mTAND and imputation for obtaining higher-quality features as inputs for downstream MoE layers from an implementation perspective. Of course, the discussion on this topic is optional.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an MOE-based model to handle multimodal data fusion. It addresses two challenges: missing modalities and irregularly sampled data trajectories. A Laplace gating function is applied to the MoE Backbone. An entropy regularization loss is proposed to ensure balanced and stable expert utilization. The author validates the method in diverse datasets.\n\nSTRENGTHS:\n1. Strong motivation. Two challenges presented in this paper, missing modalities and irregularly sampled data trajectories, are important in the multi-modal data fusion area. It is reasonable to address them through sparsing encoding and gating networks in the MOE layer. \n\n2. Good theoretical analysis. This paper presents a detailed theoretical analysis of the Laplace gating over the standard Softmax gating in MoE. Other mathematical proofs seem to extensively illustrate the characteristics of the proposed MOE model. \n\n3. Good experimental results. Comprehensive evaluations of FuseMoE on image and video datasets seem to validate its effectiveness.\n\nWEAKNESSES:\nAddressing the following weaknesses may improve the paper: \n\n1. The author should clearly distinguish the proposed method and others’ modules. From the paper, the Laplace gating function is proposed as a new one. I am not sure the author made some contributions to the encoder design, router design, and loss design. The author should make more illustrations about the contributions, not just combine other people’s work together. \n\n2. The experimental results are not extensive. The author should clearly demonstrate the data modality of the chosen benchmarks. It seems that other modalities, such as text, are not included. The author should explain this. \n\n3. Please explain more concisely how the gating functions can stabilize the imbalance and sparse multi-modal data. The paper should point this out more concisely and better with experimental results. Too many mathematical proofs and theorems in Sec.3 seem not helpful in illustrating the advantages of the proposed method.\n\nQUESTIONS:\nPlease see the weakness part to answer the questions.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe Paper introduces “FuseMoE”, a novel mixture-of-experts (MoE) framework that can handle multi-modal data even in scenarios with missing elements, sparsity of samples and temporal irregularity.  It proposes an innovative Laplace gating function with theoretical proof to enhance convergence rates and predictive performance across various tasks. The Laplace gating function allows sparser distribution of weights among MoE and encourages more balanced utilization of experts with sharper peaks and heavier tails preventing certain experts to be over dominant. FuseMoE includes modality and irregularity encoder using a discretized multi-time attention (mTAND) module to support generic fleximodal data ingestion with unlimited input modalities instead of other popular pairwise setup that exist in the literature. Paper also outlines various router designs for processing multimodal inputs and discusses the tradeoff between them and demonstrates .\n\nKey:\n\n1. Laplace gating function showed superior performance compared to Softmax gating across multiple tasks\n2. Unlike baseline models, FuseMoE demonstrated improved scalability with increasing number of modalities\n3. Handles missing modalities with tAND and per modality routers with entropy loss function\n4. theoretical analysis of convergence rates for parameter estimation\n5. FuseMoe outperformed baseline methods on several fleximodal datasets.\n\nSTRENGTHS:\n1. Sparse MoE backbone with novel Laplace gating function that ensures better convergence rates compared to traditional Softmax with theoretical guarantees\n2. Modality and Irregularity Encoder to use mTAND module to descretize irregularly sampled observations to mix with continuous features, Effective management of missing data and irregular sampling\n3. MoE fusion layer to integrate embedding from different modalities with different router designs. Flexibility to handle variable number of modalities because of per modal experts routing\n4. Novel methods, strong theoretical contributions and comprehensive emperical results and evaluation\n\nWEAKNESSES:\n1. Potential over-parameterization when input is small\n2. limited discussion of computational needs and scalability to huge datasets\n3. Most of the items discussed were already established and proved. Paper seems to be combining existing literature for these datasets.\n\nQUESTIONS:\n1. The paper claims to work with unlimited input modalities. is it really so? how do you anticipate the fuseMoE to handle extremely sparse data across modalities? Any thoughts/estimation on how far we can go here? How does the complexity of setup scale with number of modalities?\n2. Is Laplace only tested with CIFAR-10 dataset? Why was it not tested with other bigger datasets?\n3. Have you thought about or explored the interpretability side of expert assignement process? \n4. Few other paper discusses and introduces Laplace instead of gaussian for MoE. Can you expand more on how FuseMoE is differnt from these?\n    \n   - Nguyen, Hien D., and Geoffrey J. McLachlan. \"Laplace mixture of linear experts.\" *Computational Statistics & Data Analysis* 93 (2016): 177-191.\n    \n   - Wu, Lc., Zhang, Sy. & Li, Ss. Heteroscedastic Laplace mixture of experts regression models and applications. *Appl. Math. J. Chin. Univ.* **36**, 60–69 (2021). https://doi.org/10.1007/s11766-021-3591-2\n\nLIMITATIONS:\nPotential limitation of over-parameterization of networks especially when input data/size is small is adequately discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

comments: SUMMARY:
This papers proposes a novel MoE arcchitecture for handling and fusing multiple modalities, along with two core contributions:
- a novel router design that can handle missing modalities;
- a laplace gating function that is theoretically proven to ensure better convergence.

STRENGTHS:
- 1 This paper is generally well written and easy to follow; tables and figures are neat and informative; 
- 2 The motivations of core designs are well elaborated; 
- 3 Theoretical proofs are providied to further justify its effectiveness; 
- 4 Extensive experiments, along with diverse modalities. are conducted to validate the effectiveness of the proposed approach. 
- 5 The capabilty of handling missing inputs is interesting (Fig. 4 c).

WEAKNESSES:
- 1 The proposed method achieves promising results on the tested benchmark, which are still comparably small scale datasets, e.g, CMU-MOSI is with 2000+ samples. I am wondering if this work can be applicable to training/fine-tuning with larger scale data. 
- 2 Again, for vision task, only CIFAR-10 is used. This cannot fully justify its effectiveness for more general vision or multi-modal tasks;
- 3 The capability in handling missing modalities is promising. The auhor is encouraged to explain why, in some scenarios, mising modalities + the proposed method surposses the variant with full modalities (Fig 4 c).  I am still wondering its generalizability under more scenarios. 
- 4 The visualization and quanlitative analysis of the learned gaing weight would help the reader better understand the method.

QUESTIONS:
- 1 In table 5, Laplace gaiintg shows limited superiority against other variants, any insight for it? 
- 2 In table 4c, why the method achieves better results with less modaltieis? Any further insight or ablations?

LIMITATIONS:
N
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14