FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion
Authors: Xing Han
Huy Nguyen
Carl William Harris
Nhat Ho
Suchi Saria
Venue: NeurIPS 2024 poster (2024)
Keywords: Multimodal Fusion; Mixture-of-Experts; MoE Theory; Missingness and Irregularity
Primary area: deep_learning_architectures
TL;DR: We propose a Transformer model with MoE fusion layer for multimodal fusion problem with arbitrary missingness or irregularity.

Abstract: As machine learning models in critical fields increasingly grapple with multimodal data, they face the dual challenges of handling a wide array of modalities, often incomplete due to missing elements, and the temporal irregularity and sparsity of collected samples. Successfully leveraging this complex data, while overcoming the scarcity of high-quality training samples, is key to improving these models' predictive performance. We introduce ``FuseMoE'', a mixture-of-experts framework incorporated with an innovative gating function. Designed to integrate a diverse number of modalities, FuseMoE is effective in managing scenarios with missing modalities and irregularly sampled data trajectories. Theoretically, our unique gating function contributes to enhanced convergence rates, leading to better performance in multiple downstream tasks. The practical utility of FuseMoE in the real world is validated by a diverse set of challenging prediction tasks.

FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion
Xing Han
Huy Nguyen
Department of Computer Science
Department of Statisties and Data Sciences
Johns Hopkins University
The University of Texas at Austin
xhan56@jhr huu..uu.. edu
huynm@utexas. edu
Carl Harris"
Nhat Ho
Department Biomedical Engineering
Department of Statisties and Data Sciences
Johns Hopkins University
The University of Texas at Austin
charr1650jhu. edu
minhnhat@utexas.eed edu
Suchi Saria
Department of Computer Science
Johns Hopkins University
ssariaăcs jhu. edu
Abstract
As machine learning models in critical fields increasingly grapple with multi- modal data, they face the dual challenges of handling wide array of modalities, often incomplete due to missing elements, and the temporal irregularity and spar- sity of collected samples. Successfully leveraging this complex data, while over- coming the scarcity of high-quality training samples, is key to improving these models' predictive performance. We introduce "FuscMoE", a mixture-of-ext offeeeersss framework incorporated with an innovative gating function. Designed to integrate diverse number of modalities, FuseMoE effective in managing scenarios with missing modalities and irregularly sampled data trajectories. Theoretically, our unique gating function contributes enhanced convergence rates, leading bet- ter performance in multiple downstream tasks. The practical utility of FuseMoE in the real world is validated by a diverse set challenging prediction tasks.
Introduction
Multimodal fusion is critical and extensively studied problem in many significant domains 9 such as sentiment analysis [26] 54 image and video captioning [42 41,, and med- ical prediction
78 94
90
33 83 Previous research has shown that embracing multimodality can improve predictive performance by capturing complementary information across modalities, outperforming dingle-modale approaches similar tasks [71 32). However, an ongoing challenge lies in the creation of scalable frameworks for fusing multimodal data under variety of conditions, and in creating reliable models that consistently surpass their single-modal counterparts.
"Equal Contribution, EEqual Advising.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

We evaluated the characteristics of FuseMoE against various henchmarks. The pipeline approach X3 relies on a simple feature extraction scheme for each modality, followed by concatenation and classification. doesn't 't incorporate irregularities or missingness in its process, but its use of concatenation and zero-impp -imputaatioon for missing modalities allows it to be adapted to FlexiModal settings. Both [I0I] and (96] tackle multi modaiit fusion, but modalities increase, their method demands exponentially more cross-modal computations and significant model architecture modifications. Finally, (88 presents MoE. for language-image alignment, yet it also requires substantial adjustments for the more intricate and universal Flexi Modal context we explore.
Table 1:
Method
Type
Inregularity
Missingness
Num Mods
Theory
FlexiModal Aduptive?
Soenksen et al.
83
Data Pipeline
X
24
X
7hang et al.
T01
Modality Fusion
/
x
2
X
Zadeh et al.
96
Modality Fusion
3
X
Mustafa et
Multimodal MoE
FuseMoE.
This Paper
24
/
Adapted
Handling a variable number of input modalities remains an open challenge in multimodal fusion, due challenges with scalability and lack of unified approaches for addressing missing modali- ties. Many existing multimodal fusion methods are designed for only two modalities [25] 102] [OI,
rely on costly pairwise comparisons between modalities [90], or employ simple concatenation ap- proaches [3 rendering them unable to scale to settings with large number of input modalities or adequately capture inter-modal interactions. Similarly, existing works are either unable to handle missing modalities entirely [10] 98 or use imputation approaches {89 51 83 of varying sophis- tication. The former methods restrict usage to cases where all modalities are completely observed, significantly diminishing their utility in settings where this is often not the case (such as in clinical applications); the latter can lead to suboptimal performance due to the inherent limitations of im- puted data. In addition, the complex and irregular temporal dynamics present in multimodal data have often been overlooked [IO1 88) with existing methods often ignoring irregularity entirely positional embedding schemes (90] that may not be appropriate when modali- ties display a varying degree of temporal irregularity. Consequently, there is a pressing need for more advanced and scalable multimodal fusion techniques that can efficiently handle broader set of modalities, effectively manage missing and irregular data, and capture the nuanced inter-modal relationships necessary for robust and accurate prediction. We use the term FlexiModal Data to capture several of these key aspects, which haven't been well-addressed by prior works:
[83 or relying
"Flexi" suggests flexibility, indicating the possibility of having any combination of modalities, even with arbitrary missingness or irregularity.
FlexiModal data is most evident in clinical scenarios, where extensive monitoring results in the accumulation of comprehensive electronic health records (EHRs) for each patient. A typical EHR encompasses diverse data types, including tabular (c.g., age, demographics, gender), images (X- rays, magnetic resonance imaging, and photographs), clinical notes, physiological time series (ECG and BEG), and vital signs (blood chemistry, heart rate). In this setting, we observe a variety of modalities, sampled with varying irregularity and high degree of missingness and sparsity.
Contributions In this paper, we introduce novel mixture-of- (MoE) framework, which we call FuseMoE, specifically designed to enhance the multimodal fusion of FlexiModal data. FuseMoE incorporates sparsely gated MoE layers in its fusion component, which are adept at managing dis- tinct tasks and learning optimal modality partitioning. In addition, FuseMoE surpasses previous cross-attention-basee based methods in scalability, accommodating an unlimited array of input modal- ities. Furthermore, FuseMoE routes each modality to designated experts that specialize in those specific data types. This allows FuseMoE to effectively handle scenarios with missing modalities by dynamically adjusting the influence of experts primarily responsible for the absent data, while still utilizing the available modalities. Lastly, FuseMoE integrates a novel Laplace gating function, which is theoretically proven to ensure better convergence rates compared to traditional Softmax functions, thereby enhancing predictive performance. We have conducted comprehensive empirical evaluations of FuseMoE across range of application scenarios to validate its effectiveness.

---

10P
11:8
Viri Signe
P51
FIN
MoL Fusion
Task-
Encoders
Modality
Pas-
BCG
Head
Spune Buann
Add
XL
Specific
Las
Maki
Clinical
Self.
Normalize
Loud
2/15/2025
Time
Neves
ITN
Rahance
Lass
3/10/2202
Layer
Chest-X-
101
1/20/2125
(niaeingg
Rays
MoE, Fusion Laver
Figure 1: An example of addressing the challenge of FlexiModal Data: patients in ICUs often have extensive irregular health status measurements over time: patients with milder conditions only require monitoring across fewer categories. FuseMoE is adept at handling inputs featuring any combination of modalities, including those with missing elements. It starts by encoding inputs using modality-speciff feature extractors. followed by employing multi-time attention mechanism [82] to address temporal irregularities. The core of FuseMoE lies the MoE Fusion Layer, where a routing mechanism trained to categorize multimodal inputs and direct them to the appropriate combinations of MLPs. The outputs from these MLPs are weighted through gating function, resulting in fused embeddings, which are subsequently utilized for further processing.
2 FuseMoE: Enhance Predictive Performance for FlexiModal Data
In this section, we delve into the fundamental components of FuseMoE, illustrated in Figure 1 We focus two critical elements: the modality and irregularity encoder, and the MoE fusion layer.
2.1 Sparse MoE Backbone
The main components of a sparse MoE layer are a network as a sparse gate and an expert network E. (79 proposed Top-K gating function that takes as input token representation z R and then routes it to the Top-K experts out the set {E,}-=1- The gating network parameter W e RDN produces logits (r())) Top K(æ W), which are normalized via Softmax:
is
(x)(h,(z))) G(x) Σ exp(h,(x),)
Each expert network (E; RD. + P) contains a ferwafo layer (FFN) and its parameters independent of other models. The final output of the expert network y is the linearly weighted combination of each expert's output on the token by the gate's output: y = Σ (.)).
Gating Network Design The gating network's advantage lies its capacity to be concurrently trained with FFNs, facilitating the learning of an optimal sparse combination of experts. Essentially, by evaluating the similarity between the input token and the experts, the gating network/router op- timally matches the input partition with the most suitable experts. In many cases, variations in the routing mechanism can greatly influence performance across diverse applications [88 The Soft- max gating the most widely adopted across domains [77]7) [7]. We introduce novel Laplace gating function that offers enhanced convergence guarantees and delivers superior predictive performance, particularly in FlexiModal applications. The function is formulated as follows:
(x) = Top K(-IW ==|2))-
The Laplace gating function, characterized by its Euclidean term exp(-I (l11)), is less prone to converge towards extreme weight distributions due to the bounded nature of this term. In sub- sequent sections, we will illustrate how this gating function facilitates faster parameter estimation rates compared to Softmax gating. Moreover, empirical findings indicate that the Laplace gating exhibits enhanced performance in managing FlexiModal data.
Modality and Irregularity Encoder
To encode the irregularity of sampling in each modality, we utilize discretized multi-time attention (mTTAND) module 82), which leverages time attention mechanism [3 92] to discretize irregularly sampled observations into discrete intervals. Specifically, given set of l continuous time points,

---

FEN
FEN 2
FEN 5
FEN
FEN
FEN 2
FEN
FEN +
FEN
FEN 2
FEN 3
FFN +
0.45
0.35
Router
TS TXT
TXT
IMG
TXT
IMG
(a) Joint experts & router
(b) Per-modality routcrs
(c) Disjoint experts & routers
Figure 2: We present three exemplary designs of the Top-K router for effectivv multimodal fusion, considering input scenario with three modalities: Time-Series (TS), Text (TXT), and images (IMG). (a) The joint router design utilizes concatenated embedding of all modalities, directing this combined input to selected experts. (b) In the modality-speciffe router design, each modality's embedding is independently assigned pool of experts. (c) The third design variant also uses modality-speecif routers but assigns each modality's emhedding to separate pools of experts, cach pool uniquely tailored to process specific modality type.
shared
t E R corresponding to the 4th dimensionality of given modality, we employ embedding functions Φ (T) to embed each examples can be found in Appendix B
E fx in a d, dimensional vector space (detailed definition and
and E The jth dimension of the 1/hh embedding is defined as
Φn (T)
WiTkk
if
sin(wi o.), if 1 is dr.
where ( (wi, dh are learnable parameters. By performing this for each continuous time point in fa. we create a dz dimensional representation of each time point in H different embedding spaces. We then leverage these embeddings discretize the irregularly sampled observations into discretized bins. Specifically, we seek to discretize Ia (with 4 corresponding observation times (4) into regularly sampled intervals y. We do this via an attention mechanism, which, for each embedding function on, (T), takes queries, tu as keys, and as values and produces k E R? embeddings for each sequence. Formally,
(ul())KK) on (Tx) Vlx
k = Softmax
where Qa and K₁, are learnable parameters. This formulation allows us to discretize univariate observations into y regularly-sampled bins. To model irregularity across multivariate set of observations for a given modality with dm dimensions, we repeat this process for each dimension of the input. This allows us to obtain an interpolation matrix X₁. = [ f2....... &d,.. w] E x for each of the h embedding functions. We then concatenate the interpolation matrices across all H embedding functions (i.e.. I = [x,, .... Xa) E (H-.))) and employ linear projection to achieve final, discretized embedding for each modality, Z E RRxd,, where d, denotes the desired dimensionality of each modality's representation. The discretization procedure offers a standard- ized approach to managing irregularly sampled time series across various input types; however, can inevitably result in information loss. On the other hand, relying solely on the mTAND module may yield suboptimal performance due to the potentially varying sampling rates of different vari- ables (31), especially in scenarios where the sample sizes are small. To mitigate this, we combine discretized outputs with continuous representations learned through the mTNN module.
Encoding Multiple Modalities The process described above allows us to discretize an arbitrarily long irregular, multivariate sequence into regularly sampled, discretized embedding with length y and dimensionality d,. We repeat this for each of the M modalities, to create M embeddings, (2 which are then combined to generate predictions.
2.3
MoE Fusion Layer
Router Design Study Upon obtaining embeddings from each the modalities, we propose mul- tiple complementary approaches for processing multimodal inputs. Figure illustrates range of router design options. The most straightforward strategy involves employing common router that handles the concatenated embeddings of all modalities, without imposing any gating constraints.

---

As the complexity increases with additional modalities, we consider more sophisticated alterna- tives: deploying separate routers for each modality's 's embedding and assigning these embeddings to shared pool of experts. This allows for distinct processing while maintaining a unified expert framework. Additionally, we further segregate these common expert pools, allowing each router to direct its respective embedding to dedicated experts skilled in handling such specific inputs. These varied router design choices offer users enhanced flexibility, enabling more fine-grained control of both inter-modal and intra-modal relationships. Details of the respective advantages and challenges of these router design mechanisms can be found Appendix C
We implement an entropy regularization loss to ensure balanced and stable expert utilization, concept supported by various previous studies [58 (5 [22]. It maximizes the mutual information between modalities and experts and serves as an auxiliary loss function in addition to task- specifce loss. Given total of M modalities, and denoting H as the entropy, we define the loss function & as
((x)
1 M 5=1
H(fm, (E)) HI M
1
Σ pm, (E)),
where Pm, (E)) is the distribution over the experts {E,),- for the jth modality. This distribution can be approximated by pm, -E) 1 (E "= where 1 is the number of observations of the jth modality. Intuitively, we actively encourage the input embeddings diminish the uncertainty in selecting experts. By incorporating the loss E, we aim to stabilize the experts' preferences within each modality, while promoting diverse range of expert selections across different modalities.
Missing Modalities In scenarios where certain modalities are missing throughout the data trajec- tories, substitute the original embedding Mmissing with learnable embedding Z, acting generic "missing indicator". This strategy is facilitated by employing per-modality routers, which, in conjunction with entropy regularization, guide Z predominantly toward specific group of less- utilized experts. The new embeddings 2 are dynamically adjusted throughout the model training process to minimize the task-specific loss and the entropy regularization loss. As result, the router will assign lower weights to the experts responsible for processing these embeddings.
3 Theoretical Contribution
In this section, we provide a theoretical guarantee of the benefits of the Laplace gating over the standard Softmax gating in MoF. In particular, we conduct a convergence analysis for maximum likelihood estimation (MLE) under the Lapace gating Gaussian MoE, and demonstrate that the MLE under this model has better convergence behaviors than that under the softmax gating Gaussian MoE.
Problem Setup. Since the convergence analysis of MLE under the Top-K sparse gating MoE has been studied in [63], we will focus on examining the Laplace gating solely in the sequel. Assume that (X2, Yi).... (Xn: Yn) E Rdd x E are i..d. samples drawn from the Laplace gating Gaussian MoE of order k, whose conditional density function PG. (YIX) is
PG. (Y(()
Σ softmax(-11w: - XII + +3))) f(Y((a)) x xx +b,,,
(4)
where we define for any vectors v (n))/= that softmax((:) = Σラ= exp(e;) Above, f(,lu, v)
exp(
denotes a univariate Gaussian density function with mean ja and variance v. For ease of the pre- sentation, we denote G. == Σ exp(3; (6 .at bt.t) as true but unknown mixing measure associated with unknown parameters (5), W a,, b2,)) for i E 21, k.). In the paper, we specifically consider two settings of the true number of experts k: (i) Exact-specified setting: when k. is known; (ii) Over-specified setting: when k. is unknown, and we over-specify the model in equation 4 by a Laplace gating MoE model with k. experts. However, due the space limit, present only the latter setting, and defer the former setting to Appendix
5

---

Maximum Likelihood Estimation. We use the maximum likelihood method to estimate the un- known mixing measure G. 99.. In particular, the MLE is given by
G E arg max (E). 1=1
Σ
log(pa(Y,|X.)),
(5)
a..b.)) a.. : k, (W,, 0z bi p;) E } denotes the set of all mixing measures with most & components. Given the MLE defined in equation [ are ready to present the main results. Before that, let us introduce some necessary notations for our analysis.
where 9n (=) (G= (
Notations. We denote [n|] ;= 21, n for any n E N. For any vector v E IR |||| stands for its 2-norm value. Additionally, the notation |S| indicates the cardinality of given set S, while 8 denotes the Dirac delta measure. Finally, for any two probability densities p, dominated by the Lebesgue measure μ4, denote ((p, 9) J' gldu their Total Variation distance.
Firstly, we demonstrate Theorem 3.1 that the convergence rate of density estimation under the Laplace gating Gaussian MoE is parametric on the sample size
Theorem 3.1 (Density estimation). The density estimation PG., (Y|X) converges to the true density Pc. (X) under the Total Variation distance at the following rate:
Ex Ex(V(r (1X), (1X))) O( ((Vlog(m)/n)
Proof of Theorem 3.1 is in Appendix K.2 The parametric rate O (lllg)m)) of the conditional density function Pa.. indicates that if there exists loss function among parameters D such that
Ex[V(Pa, (1XX), P...p....... (-X)) 2 D(G., G.), then will achieve the parameter and expert estima- tion rates via the bound D( G.) O( (rl)=O((/lo(n///)) /og(n)/n).
Voronoi Loss. Following the above implication, we now define loss function among parameters based on a notion of Voronoi cells as in [55]. Given some mixing measure G, we distribute its com- ponents 8 == (W,,a, a, b,. 14) the following Voronoi cells, which are generated by the components 0j = (W). oj bj. vz)) of the true mixing measure G.:
= A)G)) = (i E [M] |10, 0 S 110 07 VE +}.
(6)
for any S k., Note that, the cardinality the Voronoi cell A, is exactly the number of fitted components approximating 07- For ease of the presentation, let denote (e P2, Ps. P4) == ||W; W Il ||a: - a² 11= + |b, - 6310t + |/s - v31e, for any (p1, P2: P3; P1) IR. Then, the Voronoi loss function D2 (G, G.) used for analysis under the over-specified setting is given by:
k.
D2(G.G.) Σ
Σ EEA,
exp(3.) exp(87)
Σ
exp(8.)0,
j=1
JE[H,HAA,L=EEAA
exp(3.) (2),,((-A;(), F((A))))
Σ
The notation F([A,)) stands for the minimum value of r E N such that the following sys- tem of polynomial equations does not have any non-trivial solutions for the unknown variables ((ge, (2i, ( AA:l.
Σ
Σ
Pag 9
0, for each s = ......r.
(8)
my!m!!
my + 2m
A solution to the above system is regarded as non-trivial if at least among variables is different from zero, whereas all the variables (3: are non-zero. worth noting that the function F(-) was previously studied in 28 to characterize the convergence behavior of parameter estimation under the location-scale Gaussian mixture models. 28 also gave some specific values of that function, namely F(2) = 4 and F(3) 6. Meanwhile, they claimed that it was non-trivial to determine the

---

G.. 0.2 n
DyG G.) 0.2
Loss
Loss
Sample size
Sample size
(a) Exact-specified setting
(b) Over-specified setting
Figure 3: Log-log scaled plots illustrating simulation results under the exact-specified (left) and the over-specified settings (right). The orange curves depict the mean discrepancy between the MLE G, and the true mixing measure G., accompanied by error hars signifying two empirical standard de- viations. Additionally, the gray dash-dotted line represents the least-squares fitted linear regression line for these data points. Finally, the loss functions D and D2 are defined in equations equation 9 and equation respectively. See Appendix for the experimental details.
value of F(m) when m 4, and further techniques should be developed for that purpose. Since Gaussian MoE models are generalization of the Gaussian mixture models, we also involve the func- tion F(-) our convergence analysis. Now, we provide in the following theorem the convergence rate of parameter estimation under the over-specified setting of the Laplace gating Gaussian MoE model (see also Figure for the empirical convergence rates justifying the theoretical rates in The- orem 3.2
Table 2: Parameter estimation rates under the Softmax and Laplace gating Gaussian MoE models. The func-
tion F(-) represents the solvability of system of polynomial equations considered in [64] while F(.) < F(-)
and F(2) 4, F(3) = 6. Additionally, A² == (G... denotes Voronoi cell defined in equation
Gates
exp(3;)
Wi
a3 O(n 1(n-1/2F(LA)D)
Vz O(n ((-11/R(-1D)
Softmax
64
O(n 1/2)
O(n ((--1/2((4"1)
O(n 11-IAA)))
Laplace (Ours)
O(n 1/2)
Onn41/4)
O(n-1/4)
O 1/22//)))))))
Theorem 3.2 (Parameter Estimation). When k > k. becomes unknown, the following Total Varia- tion bound holds true for any mixing measure (x())::
Ex (o(V(p((1X))P (X))) 2 D2(G,G,).
Consequently, we obtain that Dz D2(G,, G.) O( O.) (loggn)))
Proof of Theorem 3.2 is in Appendix K.3 The results of Theorem 3.2 together with the formulation of the loss function D2 in equation reveal that (see also Table
(i) The parameters W," a; b², which are fitted by exactly one component, i.c. |A41 = (A,(G,)) = 1, enjoy the same estimation rate of order O(n (up to some logarithmic factor), which match those in 644.
(ii) The rates for estimating the parameters W,' a,, by v/," which are fitted by more than one com- ponent, i.e. |ATi > 1, are longer homogeneous. On the one hand, the estimation rates for the parameters by and u² are of orders O(n (( and are determined by the function F(-) and vary with the number of fitted components |AA Those are comparable their counterparts [64]. On the other hand, the estimation rates for the gating parameters W and the expert parameters af are all of order On 1/4), which remains constant with respect to the number of fitted components. Meanwhile, those rates in [64] depend on a different system of polynomial equations from that in equation g which are significantly slower.
respectively, both of which Those rates
Advantage of Laplace Gating on FlexiModal Setting In the standard Softmax gating 64 the similarity score is computed as the inner product of token's 's hidden representation and an expert
7

---

Table 3: MoE demonstrates improved performance averaged over 5 random experiments on the CMU-MOSI and MOSEI datasets; the best results highlighted in bold font and the second best results are underlined.
MOSI Dutuset
MOSEI Dataset
Method Data
MAE]
Acc-2f
Cont
F11
MAE]
Acc-2n
Cont
Fift
0.90 1 0.02
80.81
0.34
0.04
80.70
0.18
0.59 0.03
82.50
0.58
0.68 0.02
82.10
MulT
4: 0.01
84.10
0.21
0.02
83.90
0.27
0.02
82.51
0.41
0.71 0.04
0.27
MAG
0.71 + 0.04
86.10
= 0.44
0.80 = 0.03
86.00
2: 0.09
0.57 2 0.07
85.56 0.22
0.79 0.02
=
doftmax-MoE
0.69 1: 0.01
87.09 1: 0.18
0.82 0.02
87.29
1: 0.22
0.55 2 0.03
86.34
1: 0.23
0.76 2 0.05
84.97
0.32
Joint experskrruuter
0.67 + 0.02
87.28 + 0.35
0.82 + 0.03
81.35
+ 0.24
0.54 + 0.01
86.41 + 0.36
0:805
85.43 0.25
Par-mod router
0.65 +: 0.04
88.23 :: 0.57
0.84 =: 0.01
87,39
+: 0.13
0.56 2: 0.02
0.112:+00.1
0.78 0.02
85.07
0.14
Disjoint router
0.73 +: 0.02
86.37 + 0.33
0.81 0.04
86.89
+: 0.21
0.55 :: 0.02
85.67 + 0.31
0.81 = 0.01
85.21 0.22
CIFAR-10 Dataset
Catavet
NIVCWV bocase:
Softmax Gating
Non-miating
Laplace Gating
85
Missing
Accuracy (83
85
Accuneny (%)
8g
37.5
75.0
75
72.5
70.0
VIT-MEE
VIT-Small-MoE
48 HM
L05
25 PHE
(b)
(c)
Figure 4: (a) The Laplace gating mechanism enhances CIFAR-10 classification when integrated into Vision- MoR [77]. We employed Vision Transformer (ViT) [77 and ViT-small as the backbone models and selectively
replaced their FFN layers with MoE layers: (b) FuseMoE improves prediction PAM dataset over baseline
time series models; (c) Per-modality routers and the entropy loss & mitigate the impact of missing modalities.
where a subset of experts dominates the decision-making process, resulting in the redundancy of other experts. This issue likely contributes to the slow rates of estimating expert parameters of in this setting (see Table 2. By contrast, the Laplace gating function partially alleviates this problem by computing the similarity score as the L2-distance between token representations and expert embeddings. This approach does not inherently favor any expert based on magnitude, unlike inner product which can be biased towards experts with larger norms. The Laplace gating ensures that all experts have a more balanced opportunity be selected based on how close they are to the token representation. Therefore, Laplace gating is beneficial when dealing with heterogeneous inputs, such as multimodal data, where its feature distributions can be very different across modalities. This is because it can handle these differences without being overly sensitive to the scale and variance of the input features. In addition, it can gracefully degrade in the presence of missing data, rather than causing abrupt changes in gating probabilities that might occur with inner product-based measures. The improved estimation rates for expert parameters a; under the Laplace gating Gaussian MoE, along with our empirical results on multiple large-scale datasets, substantiate these insights.
emhedding. However, this approach can lead to representation collapse 13 69
4 Experiments
Overview We demonstrate that FuseMoE can provide accurate and efficient predictions when ap- plied to the FlexiModal setting. We tested FuseMoE on diverse set of benchmarks, including MIMIC-III (40) and MIMIC-IV (36), CMU-MOSI and MOSEI 197). the Physical Activity Moni- toring (PAM) dataset (75), and CIFAR-10 10 [4).. Compared to CMU-MOSI and MOSEI, the MIMIC ecosystem exhibits irregular and missing modality patterns and includes distinct modalities unlike PAM and CIFAR-10. Evaluating FuseMoE across these diverse datasets provides various empirical insights into critical aspects of our model's performance. Comprchensive details on the datasets, metrics, parameters, and additional results thoroughly presented in the Appendices.
4.1 Main Results
CMU-MOSI and MOSEI Datasets We first apply our method to the CMU-MOSI and MOSEI datasets 197) which utilize visual, acoustic, and textual data for sentiment analysis and emotion
recognition tasks. Our methodology employs pre-trained T5 [3] for text encoding, librosa (56) for
audio feature extraction, and EfficientNet [8 for video feature encoding. Table 3 details the per-
8

---

Table 4: Comparison of FuseMoE-based methods (gray) and baselines, utilizing vital signs and clinical notes MIMIC-IV B6]. The best results are highlighted bold font, and the second-best results are underlined. All results are averaged across 5 random experiments.
Task Method
MISTS
MalT
MAG
TFN
HAIM
Sothmox
Gaussian
Laplace
48-IHM
AUROC
75.06
1.03
75.95 = 0.84
75.82 1 0.73
78.76 1 0.79
79.65 :: 0.90
79.49 = 0.83
S0.76 + 0.56
81.03 0.25
F1
45.61
0.344
38.81 + 0.22
42.55
0.82
40.61
0.41
39/79 0.90
42.86
46.86
0.24
46,53
LOS
AUROC
80.56
0.33
81.36 =: 1.32
81.13
1 0.66
80.71 1: 0:45
82.58 4: 0.00
82.11 = 0.39
81.92 4: 0.73
82,91
1.02
H1
73.01
0.52
73A45 + 0.59
72.51
0.27
73.84 +: 0.61
73:18 4: 0.00
74.43 = 0.88
74,46 0.52
74.58
25-PHE
AUROC
69.45
0.72
66.58
= 0.41
69.55 0.67
69.18 4: 0.32
63.39 1: 0.00
70.54 = 0.47
70.42
4: 0.26
71.23
0.53
F1
28.59
0.46
28.55 +: 0.31
27.86
2: 0.29
28.52 4: 0.22
42.13 :: 0.00
31.25 = 0.18
30.44 2: 0.27
31.33
4.13:.0.19 0.19
formance of various router design mechanisms within our MoE architecture, utilizing the Laplace gating function, compared against representative baselines. The baselines include (1) the early fu- sion method, Tensor Fusion Network (TFN) [96]; (2) the Multimodal Transformer (MulT), which fuses modalities by modeling their interactions 190]; (3) the Multimodal Adaptation Gate (MAG), which focuses on the consistency and differences across modalities 774); and (4) multimodal fusion using standard MoE with the Softmax gating function. Results indicate that employing an MoE of the gating function chosen whether utilizing per-modality routers enhances performance on the multimodal task.
backbone-regardiie joint experts & router This improvement is attributed to the MoE's ability effectively allocate specific components to handle distinct input modalities, thus better addressing both inter- and intra-modal relationships.
C-FAR-10 Dataset Subsequently, we evaluate our method using the Vision-MoF framework 77 on the CIFAR-I0 classification task [46), with results illustrated in Figure 4 a). In this experiment, we selectively replace the FFN layers with an even number in the Vision Transformer ( (ViT) mod- els with MoË layers. These results, along with Table 3 on the CMU-MOSI and MOSEI datasets comparing Softmax-gating MoE, indicate that the Laplace gating function surpasses the standard Softmax gating function in performance. This outcome consistent with our theoretical claims.
MIMIC-IV and PAM Datasets We then conduct comprehensive evaluations of FuseMoE on MIMIC-IV 36) and the Physical Activity Monitoring (PAM) dataset (75). These datasets feature multiple imput modalities, each characterized by varying degrees of irregular sampling or significant levels of missingness. Our tasks of interest for MIMIC datasets include the 48- 18-hour in-hospital mor- tality prediction (48-IHM), 25-type phenotype classification (25-PHE), and length-of-stay (LOS) prediction. In addition to the previously mentioned baselines, we have incorporated the HAIM method [83], data pipeline specifically designed for integrating multimodal data from the MIMIC- IV dataset. We also include the cross-attention combined with irregular sequences modeling ap- proach (MISTS) [10I. 4 the outcomes of combining irregular vital signs and clinical notes from the MIMIC-IV dataset. In addition to the commonly used Softmax gating function, we also evaluated the Gaussian gating function [93 as a comparative benchmark. The FuseMoE-basee methods surpass baselines in most scenarios, often by non- con-trivial margin. Furthermore, we ob- serve that HAIM shows considerable efficacy in extracting features from time series, resulting in a strong performance in the 4HM and LOS tasks, which are heavily reliant on such data. However, its performance appears more moderate on the 25-PHE task. The PAM dataset captures daily liv- ing activities through 17 sensors, with data from each sensor treated modalities are individually processed through time-series and irregularity encoders before being in- tegrated into the FuseMoE framework. Our baselines include the Transformer (92), GRU-D (10), SeFT (31) a mTAND-only configuration, and IP-Net [8I]. We use the Laplace gating and its joint experts & router structure in these experiments. The results in Figure 4b) have again shown the efficacy of integrating the irregularity encoder with the MoE fusion layer.
separate modality. These
4.2 Ablation Studies
Scalability of FuseMoE with Increasing Modalities Table presents the revised outcomes of the MIMIC-IV dataset after integrating CXR and ECG of corresponding patients, employing the per-modality router and the entropy loss & within FuseMoE. This setup was chosen as it slightly outperformed the joint router with an increase in modalities. Relative to their two-modality ver- sions, FuseMoE has effectively harnessed additional information (notably from CXR), resulting in

---

Table 5: Incorporating CXR and ECG into FuseMoE leads noticeable enhancement compared to their two- moo-mtdlyyty counterparts. results are averaged across random experiments.
Vital Notes &
Vital Notes
ECG
Task Method
HAIM
Softmax
Gaussian
Laplace
HAIM
Softmax
Coussion
Laplace
48-HHM
AUROC
78.87 0.00
=
83.64 0.47
78.87 0.00
82,92
83.03 + 0.85
38.87
0.46
39.78 0.00
44.04 0.26
AUROC
82.46 0.00
0.59
83.64 0.52
83.51 0.51
$2.46 0:00
83.53
83.47 1: 0.37
83.58
LOS
72.75
76.59
75.18 4:
72.75 0.00
75.01
74.43 0.64
AUROC
63.57
73.87 =:
72.68
73.65
0:00
73.74 0.41
23.67 0.71
25-PHE
42,80 0.00
38.09 0:15
43.20 0.00
36.06
36.46 + 0.55
35.81 0.34
a significant enhancement in performance. Conversely, the addition of new modalities did not ben- efit the HAIM method, possibly due to its reliance on vital signs and clinical notes without ade- quately addressing the dynamics between different modalities. Furthermore, HAIM's notably high Fl scores on the 25-PHE task can be attributed to XGBoost's proficiency in managing missing mi- nority classes. Note that, except for HAIM, other baselines were not designed to be agnostic to the quantity and variety of input modalities. Therefore, adapting them to manage extra and missing modalities requires considerable model changes, which might compromise their performance.
Missing Modalities Figure 4c) illustrates the effectiveness of utilizing per-modality routers and the entropy loss & in addressing missing modalities. Initially, we compare the performance of FuseMoE on patients with fully available modalities against those with missing components, em- ploying a joint router mechanism with the importance loss function [79]. to ensure load balancing. The inclusion of datasets with missing modalities, while expanding the sample size, resulted in a de- crease in performance due to the compromised data quality. However, a performance enhancement was observed upon integrating per- modality or disjoint routers with E. Notably, the outcomes for the 48- 48-IHM and LOS tasks with missing modalities surpassed those obtained from datasets with- out any missingness. This is because the per-modality approach can better separate the present and missing modalities, reducing the influence of experts responsible for processing the absent inputs. Therefore, this leads to a more efficient exploitation of a broader array of samples.
5 Discussions and Limitations
In this paper, we introduced FuseMoE, model adept at managing multimodal data characterized by random missingness or irregularity- crucial yet relatively unexplored challenge. FuseMoE integrates MoE fusion layers with modal embeddings and offers multiple router configurations to adeptly handle multimodal inputs across different complexity levels. FuseMoE also employs an innovative Laplace gating function, which provides better theoretical results. Through empirical evaluation, FuseMoE has demonstrated superior performance across diverse scenarios. However, our current approach encoding irregularities may potentially lead to over paraaammtteriinioonnn when the input size is small. In our future work, we aim to identify simpler and more efficient methods to handle the irregularities of input samples while preserving the model's overall performance.
Acknowledgement
Xing Han and Suchi Saria acknowledge support from the National Science Foundation (NSF) and the Gordon and Betty Moore Foundation. Carl Harris acknowledges support from the NSF Graduate Research Fellowship under Grant No. DGE 2139757. Nhat Ho acknowledges support from the NSF IFML 2019844 and the NSF AI Institute for Foundations Machine Learning. Any opinion, findings, and conclusions recommendations expressed in this material are those of the authors(s) and do not necessarily reflect the views of the NSF, Gordon and Betty Moore Foundation.
References
[1] Vibhu Agarwal, Tanya Podchiyska, Juan Banda, Veena Goel, Tiffany I Leung, Evan Minty, Timothy E Sweeney, Elsie Gyang, and Nigam H Shah. Learning statistical models of
10

---

phenotypes using noisy labeled training data. Journal of the American Medical Informatics Association, 23(6): 1673, 2016.
[2] Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. Publicly available clinical bert embeddings. arXiv preprint arXiv:7904. 03323, 2019.
[3] Aryan Arbabi, David Adams, Sanja Fidler, Michael Brudno, et al. Identifying clinical terms in medical text using ontology-guided machine learning, JMIR medical informatics, 7(2)x12596, 2019.
[4] Zachi I Attia, Suraj Kapa, Francisco Lopez-Jimenez, Paul M McKie, Dorothy Ladewig, Gaurav Satam, Patricia A Pellikka, Maurice Enriquez-Sarano, Peter A Noseworthy, Thomas M Munger, et al. Screening for cardiac contractile dysfunction using an artificial intelligence-enabled electrocardiogram. Nature medicine, 25(1):70-744 2019.
[5] Aya Awad, Mohamed Bader-El-Den, James McNicholas, and Jim Briggs. Early hospital mortality prediction of intensive care unit patients using an ensemble learning approach. In- ternational journal of medical informatics, 108:185-195,2011
[6] Dimitris Bertsimas, Jean Pauphilet, Jennifer Stevens, and Manu Tandon. Predicting inpatient flow at a major hospital using interpretable analytics. Manufacturing & &ervvce Operations Management, 24(6):2809-285 2824, 2022.
[7] Serhat S Bucak, Rong Jin, and Anil K Jain. Multiple kernel learning for visual object recognition: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013.
[8] Rhonda R Butler. led-10 general equivalence mappings: Bridging the translation gap from icd-9. Journal of AHIMA, 78(9):84 86, 2007.
[9] Bing Cao, Yiming Sun, Pengfei Zhu, and Qinghua Hu. Multi-modal gated mixture of local- to-global experts for dynamic image fusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23555 2023.
[10] Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Re- current neural networks for multivariate time series with missing values. Scientific reports, 8(1):6085, 2018.
[1]] JunKai Chen, Zenghai Chen, Zheru Chi, and Hong Fu. Emotion recognition in the wild with feature fusion and multiple kernel learning. In Proceedings of the 16th International Conference on Multimodal Interaction, pages 508-513, 2014.
[12] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785-798 794, 2016.
[13] Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma. Barun Patra, Saksham Sing- hal, Payal Bajaj. Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. On the representa- tion collapse of sparse mixture of experts. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[14] Joseph Paul Cohen, Joseph D Viviano, Paul Bertin, Paul Morrison, Parsa Torabian, Mattco Guarrera, Matthew P Lungren, Akshay Chaudhari, Rupert Brooks, Mohammad Hashir, et al. Torchxrayvision: library of chest x-ray datasets and models. International Conference on Medical Imaging with Deep Learning, pages 231-249, PMLR, 2022.
[15] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum Likelihood from Incomplete Data Via the EM Algorithm. Journal of the Royal Statistical Society: Series (Methodological), 3911:11-2 September 1977.
[16] Iman Deznabi, Mohit lyyer, and Madalina Fiterau. Predicting in-hospital mortality by com- bining clinical notes with time-series data. In Findings of the association for computational linguistics: ACL-IJCNLP 2021, pages 4026-4031, 2021.
11

---

[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[18] A Elixhauser. Clinical classifications software (ccs) 2009. http://www. heug-us. ahr4. gov/oollsofftwwrr/csssoo jsp, 2009.
[19] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Re- search, 23(1):5232-522 -2700 2022.
[20] Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Wei Xu. Are you talking to machine? dataset and methods for multilingual image question. Advances in neural information processing systems, 28, 2015.
[21] Ashutosh Garg, Vladimir Pavlovic, and James M Rehg. Boosted learning dynamic bayesian networks for multimodal speaker detection. Proceedings of the IEEE, 91()):155--136, 2003.
[22] Aude Genevay, Entropy-regularized optimal transport for machine learning. PhD thesis, Paris Sciences et Lettres (ComUE), 2019.
[23] Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stan- ley. Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. circulation, 101(23)=211--22) 2000.
[24] Brian Gow, Tom Pollard, Larry A Nathanson, Alistair Johnson, Benjamin Moody, Chrystinne Fernandes, Nathaniel Greenbaum, Seth Berkowitz, Dana Moukheiber, Parastou Eslami, et al. Mimic-iv-ocg-diiee diggnoosiccci electrocardiogram matched subset. 2022.
[25] Wei Han, Hui Chen, Alexander Gelbukh, Amir Zadeh, Louis-philippe Morency, and Soujanya Poria. Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis. In Proceedings of the 2021 International Conference on Multimodal Interaction, pages 6-15, 2021.
[26] Wei Han, Hui Chen, and Soujanya Poria. Improving multimodal fusion with hierarchi- cal mutual information maximization for multimodal sentiment analysis. arXiv preprint (XXv:2109.0041// 2021.
[27] Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific data, 6(1):96, 2019.
[28] N. Ho and X. Nguyen. Convergence rates of parameter estimation for some weakly identifi- able finite mixtures. Annals of Statistics, 44:2726-274 2016.
[29] Nhat Ho, Chiao- Yu Yang, and Michael I. Jordan. Convergence rates for Gaussian mixtures of experts. Journal of Machine Learning Research, 23(323):1-81) 2022.
[30] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 118: 1997.
[31] Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt. Set func- tions for time series. In International Conference on Machine Learning, pages 4353-4363. PMLR, 2020.
[32] Shih-Cheng Huang, Anuj Pareck, Saced Seyyedi, Imon Banerjee, and Matthew P Lungren. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ digital medicine, 3(1):136, 2020.
[33] Shih-Cheng Huang, Anuj Pareek, Roham Zamanian, Imon Banerjee, and Matthew P Lungren. Multimodal fusion with deep neural networks for leveraging et imaging and electronic health record: a case-study in pulmonary embolism detection. Scientific reports, 10(1):22147, 2020.
12

---

[34] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-lleus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 590-597, 2019.
[35] Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural compulation, 3(1):79-87, 1991.
[36] Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimic-iv. PhysioNet. Available online at: https://physionet. org/comtenthiinnicc////llcceseee August 23, 2021), 2020.
[37] Alistair Johnson, Matt Lungren, Yifan Peng, Zhiyong Lu, Roger Mark, Seth Berkowitz, and Steven Horng. Mimic-cxr-jpg-ches radiographs with structured labels. PhysioNet, 2019.
[38] Alistair Johnson, Tom Pollard, Steven Horn,, Leo Anthony Celi, and Roger Mark. Mimic-
iv-note: Deidentified free-text clinical notes, 2023.
[39] Alistair EW Johnson, Tom J Pollard, Seth Berkowitz, Nathaniel Greenbaum, Matthew Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. Mimic-cxi, a de-identified pub- licly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019.
[40] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Moham- mad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
[41] Justin Johnson, Andrej Karpathy, and Li Fei-Fei. Densecap: Fully convolutional localization networks for dense captioning. Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2016.
[42] Andrej Karpathy and Li Fei-Fei. Deep visual-semanti alignments for generating image de- scriptions. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion. pages 3128-3137, 2015.
[43] Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jasprect Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus Brubaker. Time2vec: Learnin& a vector representation of time. arXiv preprint arXiv:1907.05533 2019.
[44] Britt Keuning, Thomas Kaufmann, Renske Wiersema, Anders Granholm, Ville Pettläa Morten Hylander Møller, Christian Fynbo Christiansen, José Castela Forte, Harold Snieder, Frederik Keus, et Mortality prediction models in the adult critically ill: scoping review. Acta Anaesthesiologica Scandinavica, 64(4):424 442, 2020.
[45] Swaraj Khadanga, Karan Aggarwal, Shafiq Joty, and Jaideep Srivastava. Using clinical notes with time series data for icu management. arXiv preprint arXiv:1909.09700 2019.
[46] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[47] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1) 96(11))2278-2324 1998.
[48] Bo Li, Yifei Shen, Jingkang Yang, Yezhen Wang, Jiawei Ren, Tong Che, Jun Zhang, and Ziwei Liu. Sparse mixture-of-experts are domain generalizable learners. arXiv preprint SKKi:22206004044 2022.
[49] Yikuan Li. Ramsey M Wehbe, Faraz S Ahmad. Hanyin Wang, and Yuan Luo. Clinical- longformer and clinical-bigbrrd: Transformers for long clinical sequences. arXiv preprint arXiv:2201.//834 2022.
[50] Ke Lin, Yonghua Hu, and Guilan Kong. Predicting in-hospital mortality of patients with acute kidney injury the icu using random forest model. International journal of medical informatics, 125:55-61, 2019.
13

---

[51] Jinghui Liu, Daniel Capurro, Anthony Nguyen, and Karin Verspoor. Attention Attention-based mul- timodal fusion with contrast for robust clinical prediction in the face of missing modalities. Journal of Biomedical Informatics, 145:104466 2023.
[52] Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshninarasimhan, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Efficient low-rank multimodal fusion with modality-seccific factors. arXiv preprini 1806. 2018.
[53] Karla R Lovaasen and Jennifer Schwerdtfeger. ICD-9-CM Coding: Theory and Practice with 10, 2013/2014 Edition-E-Bood Elsevier Health Sciences, 2012.
[54] Navonil Majumder, Devamanyu Hazarika, Alexander Gelbukh, Erik Cambria, and Sou- janya Poria. Multimodal sentiment analysis using hierarchical fusion with context modeling. Knowledge-ber systems, 161:124 133, 2018.
[55] T. Manole and N. Ho. Refined convergence rates for maximum likelihood estimation un- der finite mixture models. In Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 14979-15006. PMLR, 17-23 Jul 2022.
[56] Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in python. In SciPy, pages 18-24, 2015.
[57] Clara Meister, Elizabeth Salesky, and Ryan Cotterell. Generalized entropy regularization or: There's 's nothing special about label smoothing. arXiv preprint arXiv:2005.00820 2020.
[58] Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Mul- timodal contrastive learning with limoe: the language image mixture of experts. Advances in Neural Information Processing Systems, 35:9564 9576, 2022.
[59] Ara V Nefian, Luhong Liang, Xiaobo Pi, Liu Xiaoxiang, Crusoe Mao, and Kevin Murphy. A coupled hmm for audio-visual speech recognition. In 2002 IEEE International Conference Acoustics, Speech, and Signal P Processing, volume 2, pages 1I-2013. IEEE, 2002.
[60] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Ng. Multimodal deep learning. In Proceedings of the 28th international conference on machine learning (ICML-1I), pages 689-696, 2011.
[61] Huy Nguyen, Pedram Akbarian, and Nhat Ho. Is temperature sample efficient for softmax Gaussian mixture of experts? In Proceedings of the ICML, 2024.
[62] Huy Nguyen, Pedram Akbarian, TrungTTin Nguyen, and Nhat Ho. general theory for soft- max gating multinomial logistic mixture of experts. In Proceedings of the ICML, 2024.
[63] Huy Nguyen, Pedram Akbarian, Fanqi Yan, and Nhat Ho. Statistical perspective of top-k sparse softmax gating mixture of experts. In International Conference on Learning Repre- sentations, 2024.
[64] Huy Nguyen, TrungTin Nguyen, and Nhat Ho. Demystifying softmax gating function in Gaussian mixture of experts. In Advances in Neural Information Processing Systems, 2023,
[65] Huy Nguyen, TrungTin Nguyen, Khai Nguyen, and Nhat Ho. Towards convergence rates for parameter estimation in Gaussian-gan -ated mixture of experts. In Proceedings of The 27th International Conference on Artificial Intelligence and Statisties, 2024.
[66] Xiaonan Nie, Xupeng Miao, Shijie Cao, Lingxiao Ma, Qibin Liu, Jilong Xue, Youshan Miao, Yi Liu, Zhi Yang, and Bin Cui. Evomoe: An evolutional mixture-of-exxi of-epperts training frame- work via dense-to-sparse gate. arXiv preprint arXiv:21 /2. /4397, 2021.
[67] Behnaz Nojavanasghari, Deepak Gopinath, Jayanth Koushik, Tadas Baltrušaitis, and Louis- Philippe Morency. Deep multimodal fusion for persuasiveness prediction. In Proceedings of the 18th ACM International Conference on Multimodal Interaction, pages 284-288, 2016.
14

---

[68] Raghavendra Pappagari, Piotr Zelasko, Jesús Villalba, Yishay Carmiel, and Najim Dehak. Hierarchical transformers for long document classification. In 2019 IEEE automatic speech recognition and understanding workshop (ASRU), pages 838-844, IEFF, 2019.
[69] Quang Pham, Giang Do, Huy Nguyen, TrungTin Nguyen, Chenghao Liu, Mina Sartipi, Binh T. Nguyen, Savitha Ramasamy, Xiaoli Li, Steven Hoi, and Nhat Ho. Competes- moe effective training of sparse mixture of experts via competition. arXiv preprint urXiv:240..02526, 2024.
[70] Soujanya Poria, Erik Cambria, and Alexander Gelbukh. Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment anal- ysis. In Pruceedings of the 2015 conference on empirical methods in natural language pro- cessing. pages 2539-2544 2015.
[71] Gerasimos Potamianos, Chalapathy Neti, Guillaume Gravier, Ashutosh Garg, and Andrew W Senior. Recent advances in the automatic recognition of audiovisual speech. Proceedings of the IEEE, 91(9):1306 10111106-132222 1326, 2003.
[72] Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts. arXiv preprint arXiv:2308.00000 2023.
[73] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21( 2020.
[74] Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, Amir Zadeh, Chengfeng Mao, Louis- Philippe Morency, and Ehsan Hoque. Integrating multimodal information in large pretrained transformers. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2020, page 2359. NIH Public Access, 2020.
[75] Attila Reiss and Didier Stricker. Introducing toring. In 2012 16th international symposium on wearable computers, pages 108-109. IEEE, 2012.
benchmarked dataset for activity moni-
[76] Stephan Reiter, Bjom Schuller, and Gerhard Rigoll. Hidden conditional random fields for meeting segmentation. In 2007 IEEE International Conference on Multimedia and Expo, pages 639-642. IEEE, 2007.
[77] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto. Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583-8595, 2021.
[78] Thanveer Shaik, Xiaohui Tao, Lin Li, Haoran Xie, and Juan D Velásquez. A survey of mul- timodal information fusion for smart healthcare: Mapping the journey from data to wisdom. Information Fusion, page 102040, 2023.
[79] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hin- ton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of- experts layer. arXiv preprint wcXiv:170..0655 2017.
[80] Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scal- ing pision-language models with sparse mixture of experts. arXiv preprint arXiv:2303.07226, 2023.
[81] Satya Narayan Shukla and Benjamin M Marlin. Interpolation networks for irreg- ularly sampled time series. arXiv preprint arXiv:1909.07782. 2019.
[82] Satya Narayan Shukla and Benjamin M Marlin. Multi-time attention networks for irregularly sampled time series. arXiv preprint arXiv:2 10318 2021.
[83] Luis R Soenksen, Yu Ma, Cynthia Zeng, Leonard Boussioux, Kimberly Villalobos Carballo, Liangyuan Na, Holly M Wiberg, Michael I. Li, Ignacio Fuentes, and Dimitris Bertsimas. In- tegrated multimodal artificial intelligence framework for healthcare applications. NPJ digital medicine, 5(1): 2022.
15

---

[84] Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114 14. PMLR, 2019.
[85] H. Teicher. On the mixture of distributions. Annals of Statistics, 31:55-73, 1960.
[86] H. Teicher. Identifiability of mixtures. Annals of Statistics, 32:244-248, 1961.
[87] H. Teicher. Identifiability of finite mixtures. Ann. Math. Statist., 32:1265-1269, 1963.
[88] Sindhu Tipimeni and Chandan K Reddy. Self-supervised transformer for sparse and irregu- larly sampled multivariate clinical time-series. ACM Transactions on Knowledge Discovery from Data (TKDD), 16(6):1-17, 2022.
[89] Luan Tran, Xiaoming Liu, Jiayu Zhou, and Rong Jin. Missing modalities imputation via cascaded residual autoencoder. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1405-1414, 2017.
[90] Yao-Hung Hubert Tsai, Shaojie Bai. Paul Liang, J Zico Kolter, Louis- Phiis-P Moreney, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language se- quences. In Proceedings of the conference. Association for Computational Linguistics. Meet- ing, volume 2019, page 6558. NIH Public Access, 2019.
[91] S. van de Geer. Empirical Processes in M-estimation. Cambridge University Press, 2000.
[92] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
[93] Lei Xu, Michael Jordan, and Geoffrey E Hinton. An alternative model for mixtures of experts. Advances neural information processing systems, 7, 1994.
[94] Bo Yang. Tao Mei, Xian Saennh Hua. Linjun Yang, Shi-Qiang Yang, and Mingjing Li. Online video recommendation based on multimodal fusion and relevance feedback. In Proceedings of the 6th ACM international conference on Image and video retrieval, pages 73 73-80, 2007.
[95] Bo Yang and Lijun Wu. How to leverage multimodal ehr data for better medical predictions? arXiv preprint arXiv:2110.1576: 2021.
[96] Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philipee Morency. Tensor fusion network for multimodal sentiment analysis. arXiv preprint 177.00200 2017.
[97] Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Kouis-Philippe Morency. Multi-attention recurrent network for human communication comprehension. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[98] Xunlin Zhan, Yangxin Wu, Xiao Dong, Yunchao Wei, Minlong Lu, Yichi Zhang, Hang Xu, and Xiaodan Liang. ProductIm: Towards weakly supervised instance-level product retrieval via cross-modal pretraining. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 11782-1 1791, 2021.
[99] Dongyu Zhang. Jidapa Thadajarassiri, Cansu Sen, and Elke Rundensteiner. Time Time-aware aware transformer-based network for clinical notes series prediction. In Machine learning for healthcare conference, pages 566-588. PMLR, 2020.
[100] Xiang Zhang. Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zimik. Graph-guides network for irregularly sampled multivariate time series. arXiv preprint arXiv:2110.05357, 2021.
[101] Xinlu Zhang, Shiyang Li, Zhiyu Chen, Xifeng Yan, and Linda Ruth Petzold. Improving med- ical predictions by irregular multimodal electronic health records modeling. In International Conference on Machine Learning, pages 41300-41313, PMLR, 2023.
[102] Tongxue Zhou, Su Ruan, and Stéphane Canu. A review: Deep learning for medical image segmentation using multi-modality fusion. Array, 3: 300000 2019.
16

---

[103] Yanqi Zhou, Nan Du, Yanping Huang, Daiyi Peng, Chang Lan. Da Huang. Siamak Shakeri, David So, Andrew M Dai, Yifeng Lu, et al. Brainformers: Trading simplicity for efficiency. In International Conference Machine Learning, pages 42531 22531-42542. PMLR, 2023.
[104] Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoe V Le. James Laudon, et al. Mixture-of-expers with expert choice routing, Advances in Neural Information Processing Systems, 35:7103-7114 2022.
17

---

Appendix for
"FuseMoo:: Mixture-of-Expers Transformers for Fleximodal Fusion"
Contents
Introduction
2
FuseMoE; Enhance Predictive Performance for FlexiModal Data
2.1
Sparse MoE Backbone
2.2
Modality and Irregularity Encoder
2.3
MoE Fusion Layer
3
T heoretical Contribution
Experiments
Main Results
Ablation Studies
5
Discussions and Limitations
10
Related Works
20
FlexiModal Data and Tasks of Interest
20
MIMIC-IV and MIMIC-III Datasets
B,2
MOSI and MOSEI Datasets
22
3 PAM Dataset
22
B.4
CIFAR- Dataset
23
Mechanisms of Different Router Designs
23
Joint Experts & Routers
23
Modality-Specific Router
23
Disjoint Experts & Routers
24
D Data Preprocessing
24
D.1 MIMIC-IV
24
D.2 PAM Dataset
25
Modeling Irregularity
26
Unified Temporal Discretization Embeddings
26
E.2
Unifying imputation and m TAND embeddings
26
Baseline Comparison
26
MISTS
26
18

---

F22
MulT
26
E3
MAG
26
THN
27
E55
HAIM
27
F6
Implementation
27
Computational Resources and Hyper- Hyper- Parameters
27
G.1
Computational Resources
27
G.2
Hyper-Parameters
27
Additional Results
27
H.1
FlexiModal Experiments
27
H.2
Ablation Study FuseMoE Building Blocks
28
Ablation Study on MoE Architecture
29
Details on Numerical Experiments
31
Experimental Setup
1.2
Exact-specified Setting
32
1.3
Over-specified Setting
32
J
Exact-Specified Setting
32
Proof of Theoretical Results
33
Proof of Theorem J.2
33
Proof of Theorem
37
K.2.1
Main Proof
38
K.2.2 Proof of LemmalK]
38
Proof of Theorem 3.2
40
identifiability of the Laplace Gating Gaussian MoF
43
M Broader Impact
44
19

---

A Related Works
Multimodal Fusion Initial approaches to multimodal fusion incorporated techniques such as kernel-based methods [T] 77), graphical models [59 21, 76), and neural networks [60] 20 [7]. With the diverse evolution of deep learning models, numerous advanced methods have now been employed in the fusion of multimodal data. In the realm of sentiment analysis, 196] 52 employ
low-rank Tensor Fusion method that leverages both language and video content. Attention- gating mechanisms are used by [74] [95] to generate displacement vectors through cross-modal self-attention, which are then added the input vectors from the primary modality. [90] takes an
alternative approach by integrating multiple layers of cross-modal attention blocks in a word-level vision/language/audioo alignment task.
adopt a late fusion approach to combining vital sign In the context of clinical prediction, [45 16
and text data by concatenating embeddings from pre- rre-tra feature extractors. [83] developed generalizable data preprocessing and modeling pipeline for EHR encompassing four data modali- ties, albeit through direct concatenation of existing feature embeddings for each modality followed by an XGBoost classifier [12). Recently, [101] expanded on the work of [90] by introducing a dis- cretized multi-time attention (mTAND) module [82 to encode temporal irregularities in time series and text data. Their fusion approach involves layering sets of self- and cross-modal attention blocks. However, this approach is limited to just two modalities and is not easily extendable include additional modal components or handle missing modalities. To the best of our knowledge, exist- settings that necessitate the computation of pairwise
ing works are tailored to application-specife cross-modal relationships, which are not scalable to more general settings with arbitrary modalities. Moreover, these studies typically do not account for scenarios where modalities are missing, or rely on imputation approaches based on observed data.
B3 93 has gained significant popularity for managing complex tasks since its introduction three decades ago. Unlike traditional models that reuse the same parameters for all inputs, MoB selects distinct parameters for each specific input. This results in a sparsely activated layer, enabling a substantial scaling of model capacity without a corresponding increase in computational cost. Recent studies have demonstrated the effectiveness of integrating MoE with cutting-edge models across diverse range of tasks 779
Mixture-of-Experts MoE
19 103 These works have also tackled
104 722. Given its ability to assign input partitions to specialized experts, MoE naturally lends itself to multimodal applications. This approach has been explored in fields such as vision- vision-language modeling [58] 80] and dynamic image fusion (9. However, the application of MoE in complex real- world settings, such as those involving FlexiModal Data, remains largely unexplored. This gap presents an opportunity to leverage MoE's potential in handling its intricate and multifaceted nature such as multimodal EHR, where reliable multimodal integration is crucial.
key challenges such as accuracy and training instability 666
MoE Theory While MoE has been widely employed to scale up large models, its theoretical foundations have remained nascent. Recently, [64] provided convergence rates for both density and parameter estimation of Softmax gating Gaussian MoE. They connected these rates to the solvability
of systems of polynomial equations under Voronoi-based loss functions. Later, [63] extended these theories to the top-K sparse softmax gating Gaussian MoE. Their theories further characterize the effect of the sparsity of gating functions on the behaviors of parameter estimation and verify the benefits of using top-1 sparse softmax gating MoE in practice. Other theoretical results for MoE include estimation rates of parameters and experts for multinomial logistic MoE (62), for dense-to- sparse gating MoE [61], for Gaussian gating MoE (65, and for input-indepen independent gating MoE (29.
B FlexiModal Data and Tasks of Interest
Definition of FlexiModal Data We provide a generic definition for the FlexiModal Data as we used throughout the paper. Let D {(=;") , (x;"" 1 ).""),..., (z," (1 ), yi J to be the FlexiModal dataset with N units, where x; represents the input sequence from the 4th unit of the 5th modality, t multimodal EHR as an example, each j"h modality, which may vary from time-series data like heart
my denotes the corresponding time points, and y: is the task- outcome. Take
20

---

rate, blood pressure, and glucose levels to high- sigh-dimensional inputs such as clinical notes and X-rays, contains !T observations. Figure is a more specific illustration of the FlexiModal example.
B.1 MIMIC-IV and MIMIC-II Datasets
Tasks In the ICU, where rapid and informed decisions are crucial, accurate mortality prediction essential to provide clinicians with advanced warnings of patient deterioration, aiding in critical decision-making processes (5]. Similarly, the prediction of patient length-of-stay is indispensable for optimizing treatment plans, resource allocation, and discharge processes [6] Further, phenotyp-
ing of critical care conditions is highly relevant to comorbidity detection and risk adjustment and presents a more challenging task than binary classification. due to the heterogeneous presentation of conditions and the larger number of prediction tasks [[OI]. We concentrate on three critical care tasks as highlighted in [4], performing extensive empirical analysis on each building block of the proposed framework.
48-IHM In this binary classification task, we predict in in-hospital mortality based on the first 48 of the ICU stay for patients who stayed in the ICU for at least 48 hours.
LOS We formulate our length-of-stay task similar to that of 48-IHM: for patients who spent at least 48 hours in the ICU, we predict ICU discharge without expiration within the following 48 hours.
25-PHE In this multilabel classification problem, we attempt to predict one of 25 acute care conditions [18 (53) (e.g., congestive heart failure, pneumonia, shock, etc.) at the end each each patient's 's ICU stay. Because the original task was designed for diagnoses based ICD-9 codes, but MIMIC-IV includes both ICD-9 and ICD-10 codes, we map patients with diagnoses coded using ICD-10 using the conversion database provided by (8)
48-IHM death
LOS discharge
PHE
HR BP
Time series
0
25 ontical care conditions
Glucose
0
0
Note
0
0
0
0
0
0
0 4
0
0
0
CXR
A
4 12
ECG
12
Time (hrs)
48
EOS
96 EOS
EOS
Figure 5: Schematic of tasks of interest. Plotted are example vitals/labs, radiological notes, X-rays, and ECGs sampled over the course of a patient's ICU stay. The first three rows represent example observations from a single modality consisting of three irregularly sampled vital signs (HR, BP), and lab values (Glucose). The following three rows represent irregularly sampled radiological notes, X- rays, and ECGs. Opaque shapes denote observations falling within the observation window (i.e., observations that are used to generate predictions), while translucent shapes are not used to generate predictions. For the 48-IHM task, we use the first 48 hours of observations to predict death at any time during the ICU stay. For the LOS task, use the first 48 hours of observations to predict whether the patient will be discharged (alive) during the following 48 hours. And in the phenotyping task (PHE), we use all observations to predict one of 25 critical care conditions.
We implement an in-hospital mortality prediction (48-IHM) task to evaluate our method's ability predict short-term patient deterioration. Similarly, an accurate determination of patient discharge times is crucial for optimizing patient outcomes and hospital resource allocation (6. which motivates our length-of-stay (LOS) task. We frame 48-IHM and LOS as binary classification problems and use 48-hour observation window (for patients who spent at least 48 hours in the ICU) to predict in-hospital mortality (48-IHM) and discharge (without expiration) within the 48 hours following the observation window (LOS). Lastly, identifying the presence of specific acute care conditions in patient records is essential for various clinical objectives, including the construction of cohorts for clinical studies and the detection of comorbidities MI Traditional methods, often reliant on manual chart reviews or simple billing code-based definitions, are increasingly being supplemented by machine learning techniques 27:: automating this process requires high-fidelity classifications, motivating our 25-type phenotype classification (25-PHE) task. In this multilahel classification problem, we attempt to predict one of 25 acute care conditions using data from the entire ICU stay.
21

---

Evaluation In our initial analysis, we focused on patients with no missing modalities, resulting in dataset comprised of 8.770 ICU stays for the 48-IHM and LOS tasks, and 14.541 stays for the 25-PHR task. For our analyses with missing observations, we include a total of 35, stays for 48-IHM and LOS,, and 71.173 for 25-PHE. To evaluate the single-label tasks, 48-IHM and LOS,, we employ the F1-score and AUROC as our primary metrics. In line with previous studies [01 50 3), we use macro-averaged Fl-score and AUROC to assess the 25-PHE task.
Dataset Information We leveraged data from MIMIC-IV 36], comprehensive database with records from nearly 300k patients admitted to a medical center from 2008 to 2019, focusing on the subset of 73,181 ICU stays. We were able link core ICU records (containing lab results and vital signs) to corresponding chest X-rays [39], radiological notes [38], and electrocardiogram (ECG) data (24) taking place during given ICU stay. We allocated 70 percent of the data for model training, with the remaining 30 percent evenly split between validation and testing.
The total number of samples for each our three tasks (i.c., those in which at least one vital sign was recorded in the specified observation window), along with the total number of observations per-modality, are shown in Table
Missingness Rates
Table 6: We present the total number of ICU stays in each task, taking into account observations with missing modalities. The total number stays with at least one observation of the corresponding modality shown in the three right-most columns.
(ss
Total
Text
CXR
ECG
IH-I & LOS
35,129
32,038
8,781
18,271
25-PHE
73,173
56,824
14,568
35,925
B.2 MOSI and MOSEI Datasets
Task We focus on the multimodal sentiment analysis (MSA) task which aims predict sentiment polarity E {positive, negative, and neutral} and sentiment intensity, which is a real number ranging from -3 to +3 under multimodal setting.
Evaluation Following previous work such as (26). we adopt mean absolute error (MAB), Pearson correlation (Corr), binary classification accuracy, Fl score computed for nenaeiigviveeeeeatieee class as evaluation metrics.
Dataset Information The CMU-MOSI dataset contains 1 1284/229/686 train/validationntest sam- ples, and the CMU-MOSEI dataset contains 16326/187 1/4659 train/validationnesst samples. They are the largest dataset multimodal sentiment analysis and emotion recognition to date. The datasets contain utterance videos from numerous online YouTube speakers, which are transcribed and properly punctuated, leading to multimodal input consisting of video frames, text, and audio signals.
B.3 PAM Dataset
Task Physical Activity Monitoring (PAM) dataset measures the daily living activities of subjects with 3 inertial measurement units. PAM is labeled into 8 classes where each class represents an activity of daily living.
Evaluation
We choose common classification accuracy as the evaluation metric for this task.
Dataset Information The processed PAM dataset contains 5.333 segments (samples) of sensory signals. Each sample is measured by 17 sensors and contains 600 continuous observations with the sampling frequency 100 Hz. PAM does not include static attributes and the samples are approxi- mately balanced across all 8 categories.
22

---

Table 7: Dataset Summary
Dataset
Research Area
Modalities
Sample Size
Tasks
MIMIC-II
Healtheare
Time-Series, Text
36,212
Mortality, longth-of-stay, phenotyping
MIMIC-IV
Healthcare
Time-Series, Text, Images, ECG
73,173
Mortality, length-of-stay, phenotyping
CMU-MOSI
Affective Computing
Text, Video, Andio
2,199
Sentiment
CMU-MOSEI
Affective Computing
Text, Video, Audio
22,777
Sentiment, emotions
PAM
Healthcare
Time Series
5,333
Activity recognition
CIFAR-10
Multimedia
Images
60,000
Image classification
B.4 CIFAR-10 Dataset
CIFAR-10 [46] is an established computer-vision dataset used for object recognition. It consists of 60,000 32x32 color images containing one of 10 object classes ("plane", "car", "bird", "cat", "deer", "dog", "frog", "horse", "ship", "truck"), with 6000 images per class.
C Mechanisms of Different Router Designs
C.1 Joint Experts & Routers
In this approach, a concatenated embedding of all modalities is created, and this combined input is directed to selected experts by the router. This method allows the model to capture interactions be- tween modalities at the input level, as the concatenated embedding provides a unified representation that includes all modalities. The router and experts work with this comprehensive view, enabling the model to learn correlations and interactions directly from the fused data. However, this ap- proach might fully capture modality-specifff nuances since the characteristics of each modality are blended into single representation.
Advantages
1. Captures inter-modal relationships by considering all modalities together.
2. Simplifies the routing mechanism by treating the concatenated embedding as a single input.
Challenges
1. May overlook modality-specifff features due to the blending of all modalities into one representation.
2. Could be less efficient if some modalities are irrelevant for certain tasks experts.
C.2 Modality-Speciffe Router
Each modality's 's embedding is independently assigned to a shared pool of experts by modality- specific routers. This setup allows the model to maintain the distinctiveness of each modality while still leveraging a common pool of expertise. By doing so, can better capture modality-specifi nuances and how they contribute independently to the overall task. However, this approach might be less effective in capturing complex inter-modal interactions since the initial routing is done inde- pendently for each modality.
Advantages
1. Preserves modality-specific information by routing each modality independently.
2. Flexible in directing modalities to the most relevant experts, potentially improving effi- ciency.
3. Captures interactions between modalities to some extent, but may not be as effective as joint routing approaches.
Challenge sights.
needs additional coordination between independent routes leverage cross-modal
23

---

C.3 Disjoint Experts & Routers
In this configuration, modality-specifie routers assign each modality's embedding to separate pools of experts, with each pool uniquely tailored to process a specific modality type. This method maxi- mizes the ability of the model to capture and exploit modality-specifie features and relationships, as each pool of experts is optimized for a particular type of data. However, this setup might limit the model's ability to learn from the interactions between modalities, as each is processed in isolation.
Ad Adaatage
1. Allows for highly specialized processing of each modality, potentially improving perfor- mance on modality-specific tasks.
2. Modality-specifee experts can develop deeper insights into the characteristics and patters within their designated data type.
Challenges
1. Inter-modal relationships might be underutilized due to the segregated processing of each modality.
2. Requires additional coordination or subsequent integration stages to combine insights from different modality-specifff experts.
Each router type offers unique benefits and faces specific challenges in capturing the subtle rela- tionships between modalities. The choice among them depends on the specific requirements of the application, including the importance of preserving modality-speciff information versus capturing inter-modal interactions, and the computational efficiency of managing multiple experts and routers. For example, we found that modality-specinee routers are more effective in ameliorating the effect of missing modality in our experiments.
D Data Preprocessing
D.1 MIMIC-IV
In the preprocessing stage, we focused on 30 pertinent lab and chart events from each patient's s ICU record for vital sign measurements. For chest X-rays, we utilized a pre-trained DenseNet- model [14], which was fine-tuned on the CheXpert dataset (34), to extract 1024-dimensional image embeddings. For radiological notes, we obtained f68-dimensional embeddings using the BioClini- calBERT model (2). ECG signals were processed using a convolutional autoencoder, adapted from to generate 3i--dimnniional embedding for each ECG.
Nine of these were vital signs: heart rate, mean/systoliccdiastolii blood pressure. respiratory rate, oxygen saturation, and Glascow Coma Scale (GCS) verbal, eye, and motor response. We also included 21 lab values: potassium, sodium, chloride, creatinine, urea nitrogram, bicarbonate, anion gap, hemoglobin, hema- tocrit, magnesium, platlet count, phosphate, white blood cell count, total calcium, MCH, red blood cell count, MCHC, MCV, RDW, platlet count, neutrophil count, and vancomycin. We standard scale each time series value to have mean 0 and standard deviation 1, based on the values in the training set.
Time series We selected 30 time series events for inclusion. following 83
Chest X-rays To incorporate medical imaging modality into our analyses, we use the MIMIC- CXR-JPG [37 module available from Physionet (23). which includes 377,110 JPG format images derived from the DICOM based MIMIC-CXR database [39]. Following [83], for each image, we
resize each JPG image to 224 x 224 pixels and then extract embeddings from the last layer of
Densenet121 model. We identify X-rays taken while the patient was in the ICU by first matching subject IDs MIMIC-CXR-IPG with the core MIMIC-IV database, then limiting these matched X-rays to those with chart time occuring between ICU admission and discharge.
24

---

Radiological notes To incorporate text data, we use the MIMIC-IV-Note module [3), which contains 2,321,355 deidentified radiology reports for 237,427 patients that can be matched with patients in the main MIMIC-IV via similar approach to chest X-rays. We note that we were unable to obtain intermediate clinical notes (i.e., notes made by clinicians throughout a patient stay), as those have not yet been publicly released. We extract note embeddings using Bio-Clinical BERT 21
To include ECGs as an additional modality in our models, we uti- lize the MIMIC-IV-ECG 24 module, which includes approximately 800,000 ECGs (10 seconds, sampled at 500 Hz) collected from nearly 160,000 unique patients. To transform the ECGs so that they are suitable for input our model, we adopt convolutional autoencoder approach, adapted from [4], that compresses each ECG into 256-dimensional vector. Specifically, each diagnostic ECG contains 5000 x 12 dimensional vector (5000 time points 12 ECG leads). To prepare the ECG for input to the autoencoder, we only include the first 4096 time points. We then train the autoencoder compress the ECG into a 256-dimensional latent vector, and then reconstruct the original ECG using upsampling layers, using mean squared error as our loss function. The ar- chitecture is shown in Figure 6 We train the autoencoder with 90% of the ECGs available in the MIMIC-IV-ECE projection and use the rest for validation. We selected a batch size of 2048, and reduced the learning rate by factor of 0.5 if the validation loss had plateaued for epochs. Training stopped if the validation loss had not decreased for 6 epochs. For our encoder, we use filter numbers of [16, 16, 32, 32, 64, 64], kernel widths of [5, 5, 5, 3, 3, 3] and dropout rate of 0.1. For the decoder, we use the same filter numbers and kernel widths in reverse, and maintain a dropout rate of 0.1
Elletrocardiograms (ECGs)
Input ECG
Reconstructed ECG
Temporal conv
Dropout
Batch norm
Batch norm
X
x
Dropout
Convolution
Max pooling
Upsampling
Flatten
Reshape
Dense layer
Dense layer
Dropout (0.5)
Dropout (0.5)
Latent vector (256-dim)
Figure 6: CNN Autoencoder Architecture The encoder consists of 6 convolutional blocks (tempo- ral convolution, batch normalization, dropout, and max pooling layers), followed by a dense layer that reduces the dimensionality of the representation of 256. The decoder reconstructs the input ECG (dimensionality 4096 12) from this latent vector via dense layer, followed by 6 upsampling convolutional blocks (upsampling, convolutional, batch normalization, and dropout layers).
D.2 PAM Dataset
We follow the preprocessing procedure from 100 as published from their official GitHub reposi-
tory
25

---

E Modeling Irregularity
E.1
Unified Temporal Discretization Embeddings
Unlike the embeddings in chest X-rays, clinical notes, and ECGs, vitals/la/lime-srricc values present temporal irregularity across dimensions. That is, for the former three modalities, each di- mension of the corresponding is observed at each irregular time point т. By contrast, the sampling for vitals/labs irregular in both within and across dimensions. For example, we might observe heart rate values sampled at times THR {0, 0.2, 0.8, 1.2, 2.8}, and glucose values sampled at time 3.43.
Tulu = (0.1, 3.4. Given this unique challenge present in vitals/labs, we adapt the Unified Tem- poral Discretization Embedding (UTDE) approach described in [00I, which combines the m'TNND approach described in Section 2.2 with a simpler imputation-bant based discretization scheme. Specif- ically, given set of / observations x E R' observed at irregular times E RC, we simple imputation scheme to discretize x into target bins 7 (c.g., 7 (0.........77 Specifically, given bin value N 7, apply the following rules:
If there exists a previously observed value of x (i.c., 3r E T st. T S 7:). we set the imputed value of x at time 7, By the closest previously observed value.
If no previously observed value exists, we set the value of irg to the global mean of x.
We do this for each possible vitals/lab, to generate matrix of imputation embeddings I E R were dvvials is the number of vitals/labs. We then input this embedding into a ID causal convolutional layer with stride 1 to obtain our final imputation embeddings with hidden dimension da. Ermp Ryxds.
E.2 Unifying imputation and m'ANDD embeddings
We combined simple imputation and mTAND embeddings via gating function g. Following 101 we let ETTAND E xxx denote the mTANND embeddings for vitals/labs derived from the process described in Section 2.2 and let Emmp E xd,, denote the simple imputations from the process described above. We use each of these discretization embeddings to derive final set of embeddings for vitals/labs Evitals via a one-layer MLP gating function ʃ. Specifically, we g = [(Empp 4 EmTAND), where denotes the concatenation operator. We then calculate Evitals
Evitals g + Eimp (1 g) EMTAND IRxd,,
where @ denotes posnt-wise multiplication.
F
Baseline Comparison
F1 MISTS
This approach, from 101]. casts time series and clinical notes multivariate, irregularly-sample time series (MITTS) and uses layers of self- and cross-attention to fuse modalities. The method Veme 45 encoding scheme represent the irregularity of observation times. We use the same hyperparameters as in the original paper (e.g., 3 self- and cross-attention blocks, 128- dimensional time embedding, etc.)
F.2 MulT
This model from [90] relics on multiple stacks of pairwise and bidirectional cross-modal attention blocks (without self-attention mechanism) to attend to low-level features. The results of cross- modal attention are then sent to modality-speciiff transformers, concatenated, and used to make predictions.
E3 MAG
This method introduces the Multimodal Adaptation Gate (MAG) as an extension BERT and XI- Net, allowing these pre-trained models to incorporate visual and acoustic data during fine-tuning.
26

---

By generating a conditio- shift in their internal representations, MAG enables enhanced sentiment analysis performance on multimodal datasets, achieving human-level accuracy in the field
74
F44
TFN
The proposed Tensor Fusion Network approach (TFN) integrates three core components: Modality Embedding Subnetworks for generating rich embeddings from unimodal inputs, a Tensor Fusion Layer for capturing all levels of modality interactions through a 3-fold Cartesian product, and Sentiment Inference Subnetwork tailored to perform sentiment analysis based on the fusion layer's output [96]
E5 HAIM
The multimodal fusion approach detailed by 83 extracts a single set of features for each ICU stay, and uses this to predict the outcome interest (in- sossopiia mortality, etc.). For vitals/lab values, the authors extract set of 11 generic time series features: signal length, maximum, minimum, mean, median, SD, variance, number of peaks, and average time-series slope and piece- wise change over time of these metrics. This is done independently for each of the 30 events, leading to 30 x 11 330 vital/lab features per ICU stay. To provide a fair comparison with our method, we only include the most recent five notes and 128 vitals measurements in calculating embeddings. We only include entries for which all modalities are observed. For note/X- -ote/X-rayEEGG embeddings, we compute the mean embedding across all observations occurring during the specified time frame (i.e., the first 48 hours of 48-IHM and LOS, the entire stay for PHE). As with our method, we standardize seale values based on the training set. (83] uses an XGBoost [12] classifier to predict the outcomes of interest. We follow the hyperparameter optimization approach described the paper. Specifically, conduct grid search across the following sets of hyperparameterss: max depth (5, 6, 8), number estimators = {200, 300), learning rate = (0.3, 0.1 0.05). Hyperparameters are selected based the maximum AU-ROC from five-fold cross-validation.
F.6
Implementation
We integrate F.I through F.4 into our workflow using the implementation provided by [[01]. For adapt the time series (e.g., series variance, mean, etc.) feature extraction and model fitting code from the repository released by the corresponding paper. The original paper doesn' use ECG waveforms, adopt similar approach to ECG embeddings with image and note embeddings, and take the mean value of the latent vector across all included observations.
F.5
G Computational Resources and Hyper-Parameters
G.1 Computational Resources
We train models using Lambda Workstation with four A550 GPUs with 24 GB of memory. We are able to train models using a single GPU. An analysis of computation time and memory requirements is shown in Figure 12
G.2 Hyper-Parametes
The set of parameters we used for experiments be found in Table
H Additional Results
H.1 FlexiModal Experiments
We present additional results comparing FuseMoE to baselines using the MIMIC-III dataset, which includes only vital signs and clinical notes (Table 9). and the MIMIC-IV dataset, featuring vital signs and CXR (Table 10). All experiments utilize the "joint experts and router" configuration. In these settings, FuseMoE demonstrates noticeable advantages.
27

---

Table 8: Hyperparameters used for MoE framework and general architecture.
Hyper-Parameter Type
Parameter Name
Value
Number of experts
16
FFN hidden size
512
Top k
4
MoE
Disjoint top
Hidden activation function
GeLU
Number of MoE layers
Random seed
[32, 42, 52, 62, 72]
Training epochs
8
Training batch size
Eval batch size
8
CNN kernel size
Gradient accumulation steps
16
BERT update epochs
Other Parameters
BERT learning rate
2.00E-05
Time series encoder learning rate
4.00E-04
Number of notes to include for patient
5
Get notes from beginning or last Attention embedding dimension
Last
128
Number of attention heads
Maximum time for irregular time series
48
Time embedding dimension
64
Table 9: Comparison of FuseMoE- methods (gray) and baselines, utilizing vital signs and clinical notes of MIMIC-III. The best results are highlighted in bold font, and the second best results underlined. All results are averaged across 5 random runs. Since HAIM is not designed for the MIMIC-III dataset, we use the concatenation method from e.g.
455 as a replacement.
Laplace
48-IIM
AUROC
89.14
0.57
$7.26 = 0.35
86.53 121
87.22 1 0.89
86.72 1: 0.76
90.25 = 0.74
90.77 : 0.18
91.19 0.52
56,4 1,30
54.13
1.20
53.20
51.44 +
52.77 0.70
=
56.21 0.17
AUROC
86.06
0.06
85.96
= 0.07
85.94 0.07
84.74 4: 0.16
85.94 1.221
86.41 0.75
85,96 0.64
86.72 0.27
25-PHE
F1
0.38
54.20
0.33
53.73 = 0.37
49.84 + 0.83
53.30 +: 0.35
55.02 0.23
55.29
55.38
H.2
Ablation Study on FuseMoE Building Blocks
In Figure 7 we evaluate the impact of various irregularity encoders on the performance of the FuseMoE framework. Our baseline approaches include the following methods:
1. employing only the imputation (discretization) module from the time-series irregularity encoder, as detailed in Appendix E
2. utilizing solely the mTAND module
82 within the time-series irregularity encoder
3. implementing the SeFT method
31
as an irregularity encoder
4. adopting the RAINDROP method 100 as an irregularity encoder
Table 10: Comparison of FuseMoE-basee methods (gray) and baselines, utilizing vital signs and
CXR of MIMIC-IV. The best results highlighted in bold font, and the second-best results underlined. All results are averaged across random experiments.
Task Method
MISTS
MalT
MAG
HAIM
Soflmxx
Gaussian
Laplace
AUROC
81.36
0.24
77.20 + 0.44
81.19
1.25
76.92 + 0.65
80.87 + 0.00
82.08 - 0.26
81.26
0.18
82.97
0.49
48-1HM
F1
43.35
0.39
28.40 1 0.75
39,59
0.43
46.59
0.33
40.88 1 0.00
38.14 0.31
44.59 a: 0.24
47.48
AUROC
82.07 0.82
81.94
0.26
81.66 + 0.76
81.47 + 0.89
81.69 0.00
82.96 = 0.47
82.74 + 088
83.22
0.68
LOS
F1
74.07 2.188
74.46
1 0.17
73,89
0.93
73.39 4: 0.14
72.93
à: 0.00
75.67
0.59
75.16
0.42
75.43
0.19
AUROC
71.50
0.22
71.20
= 0.76
70.89 0.47
70.55
- 0.29
63-43 1: 0.00
71.38 = 0.31
70.87
:: 0.67
71,44 0.24
25-PHE
F1
33,52
0.39
32.90 + 0.18
38.14
0.61
33.56 + 0.74
42.48 0.00
33.49
0.15
31.94
0.09
34.13 0.56
28

---

Comparison Different Irregularty Encoders
Comparison of Different Irregularity Encoders
Imputationonnly
Imputationnnny
mTAA only
MTAAN only
SeFT RAINDROP FusalMeE
SeFT RAINDROP
FuculMpE
Score
AUROC
8
50
48-IHM
LOS
25-PHE
48-IHM
LOS
25-PHE
(b)
Figure The irregularity encoder employed by FuseMoE achieves the best average results compared with 4 baseline irregularity encoders. The performance of these approaches is averaged over random runs. We utilized the vital signs and clinical notes components of the MIMIC-IV dataset.
In Figure 8 evaluate the impact of various time-series encoders the performance of the Fuse- MoE. framework. The original FuseMoE framework feeds time-series embeddings obtained from the irregularity encoder into the Transformer [92] and extracts the last hidden states of the Trans- former output to pass through fully connected layers to make predictions. Our baseline approaches include CNN 47 and LSTM 30 encode time-series embeddings from the irregularity encoder.
Comparison Different Time Series Encoders
Comparison Different Series Encoders
80
CNN
CNN
LSTM
LSTM
Transfomer
Tham aenformer
00
Score
AUROC
70
48-IHM
LOS
25-PHE
48-IHM
LOS
25-PHE
(b) Figure 8: Transformer is more effective in acting as the time-series encoder than CNN and LSTM. The performance outcomes of these approaches are derived from averages over 5 random runs. We utilized the vital signs and clinical notes components of the MIMIC-IV dataset.
In Figure 3 we assess the effect of different CXR encoders on the FuseMoE framework. Currently, the FuseMoE framework incorporates DenseNet-11 121 as the feature extractor for CXR images before their integration into the mTAND module. This setup compared with the application of the state- of-the-ar art vision transformer (ViT-B) 17 as an alternative CXR encoder.
In Figure 10
evaluate the influence of text encoders the FuseMoB framework. Currently, FuseMoE incorporates Clinica--Longforner [49] as the text encoder before integrating into the mTTAND module. This setup is compared with other state-of-the-art text encoders: GRU-D (T0), FT-LSTM [99]. and HierTrans [68].
Finally, in Figure II investigate the effect of the mTAND module each modality, while removed mTAND for particular modality, the rest of FuseMoE's components remained constant.
H.3 Ablation Study on MoF. Architecture
We then conducted ablation studies to explore the efficiency and effectiveness of MoE architecture on model performance. We mainly use MIMIC-IV as our test bed. Figure 12 a) examines the com- putational efficiency and resource utilization, positioning FuseMoE approximately in the middle of
29

---

Comparison of Different CXR Encoders
Comparison of Different CXR Encoders
DenseNiet 121
Vi-B
AUROC
02.
Score
01.3
725
48-IHM
LOS
25-PHE
48-IHM
LOS
25-PHE
(b)
Figure 9: VIT could further improve predictive results in tasks by providing better CXR em- beddings. The performance of these approaches is averaged over 5 random runs. We utilized all modalities (vital signs, clinical notes, CXR, and ECG) of MIMIC-IV. Note that while we vary the CXR encoders, the rest of our framework's s components remain constant.
Comparison Different Encoders
Comparison Different Encoders
GRLAD FTLSTN
GRL-B
FTLSTN
Herchrn ChrcalLorgfomerr
HeeeTrne
CovcalLonglomer
AUROC
F1
48-IHM
LOS
25-PHE
48-IHM
LOS
25-PHE
(b)
Figure 10: Clinical- Lonnfal--ongformmr as the text encoder achieves the best performance compared with baselines. The performance of these approaches is derived from averages over 5 random runs. We utilized all modalities (vital signs, clinical notes, CXR, and ECG) of the MIMIC-IV dataset, while varied the text encoders, the rest of our framework's components remained constant.
Effect of inregularity Encoder on tach Modality
Effect irregularity Encoder on tach Modality
Villed mDop An
200 xtlvro
20c stwel
AUROC
Score
48-IHM
LOS
PHE
4B-IHM
LOS
25-PHE
(b)
Figure 11: Encoding irregularity using the mTAND module improves the overall performance. The positive effect of the irregularity encoder is most evident in vital signs and clinical notes. The performance outcomes of these approaches are averaged over random runs. We utilized all modalities (vital signs, clinical notes, CXR, and ECG) components of the MIMIC-IV dataset.
30

---

2000
85
0.5
Vital
Task Model
1800
48 IHM
Vital
25-PHE
80
Hormalizeed Weight
0.4
Vital
1600
TF MISTS
1.400
AUROC
0.3
Note
CXR
1200
TuseMot
Averoge
1.000
LOS
CXR
IHM
ECG
65
25: PHE
ECG
1500
2000
2500 3000 3500 4000 4500
10 Total Number of Experts
20
25
0.0
48-IHM
LOS Tasks
25-PHE
Max GPU Memory (MIB)
(a)
(b)
(c)
Figure 12: Results of ablation studies on MoE architecture: (a) The computational efficiency and resource utilization each method when applied to vital signs and clinical notes from the MIMIC- IV dataset: (b) The relationship between the number of experts and task performance across different modalities, including vital signs, clinical notes, and CXR; (c) The impact of each modality on the top-A experts within a disjoint router structure.
Audio
Audio
20.1%
5%.5%
36.8%
Language
43.1%
28.2%
Video
Language
Video
CMU-MOSI
CMU-MOSEI
Figure 13: Modality weight composition of CMU-MOSI and CMU-MOSEI datasets.
the comparison. Despite the increase in model parameters due the incorporation of the MoE layer, its sparse nature does not significantly escalate the computational load. Figure b2 illustrates the correlation between the number of experts and task performance across different modalities. Gen- erally, performance improves with the addition of more experts, plateauing once the count exceeds 16. To achieve a compromise between performance and computational expense, we opted to utilize the top 4 experts out of 16 in our experiments. Figure 12 c) and Figure [3 study the influence of each modality on the top-k chosen experts. For every expert selected, we calculate the number of samples that include specific modality, weighted by corresponding weight factors from the gating functions. The outcomes are subsequently normalized across modalities. The analysis Figure [2[c) reveals that predictions across all tasks heavily depend vital signs and clinical notes. This reliance attributed to the abundant samples in these two modalities. Despite the notably smaller significant roles in the 25-PHE and 48-IHM tasks, which aligns with our findings in Table The results in Figure [3 that the modality weight distribu- tion in the MOSI and MOSEI datasets is more "spread out",, with the audio component carrying greater weight in the MOSEI dataset.
quantity of CXR, they play
I Details on Numerical Experiments
We conduct multiple numerical experiments to illustrate the theoretical convergence rates of the MLE G2 to the true mixing measure G, under both exact-specified and over-specified settings.
L.1 Experimental Setup
Synthetic Data. Assume that the true mixing measure G. = Σ exp( o2.0).)) of
order k, = 2. The true parameters for the router, (W", (2) Rd x R. are drawn independently from an isotropic Gaussian distribution with zero mean and variance o² = 0.01/d for S < 6, and otherwise are set to zero. Similarly, the true parameters of experts, (a b)) TRd R, are drawn independently of an isotropic Gaussian distribution with zero mean and variance of = l/d
31

---

for all experts. For the variances 14, we also sample from the Gaussian distribution. (0, o..) and then take the absolute value of the sample. These parameters remain unchanged for all experiments.
Then, we generate i.i.d samples ((X,, Y=) by first sampling X,'s from the uniform distribution Uniform (0, 1] and then sampling Yj's from the true conditional density gG. 9G_ (Y]X) of the Laplace gating Gaussian mixture of experts (MoE) given in equation 4
Maximum Likelihood Estimation (MLE). A popular approach to determining the MLE Gn for each set of samples is use the EM algorithm [[]] However, since there are not any closed- form expressions for updating the gating parameters 8ox, 31s in the maximization steps, we have to leverage EM-based numerical scheme, which was previously used in 633 In particular, utilize a simple coordinate gradient descent algorithm in the maximization steps. Additionally, we select the convergence criterion of e 10-6 and run maximum of 2000 EM iterations.
Initialization. For each E {k.,k, k. + 1]. we randomly distribute elements of the set (1, k} into k. different Voronoi cells A2...... each contains at least one element. Moreover, repeat this process for each replication. Subsequently, for each j E [k.], we initialize parameters W; by sampling from a Gaussian distribution centered around its true counterpart W with small variance, where i E Aj. Other parameters 3i, ai, bi, Us are also initialized in a similar fashion.
1.2 Exact-specified Setting
Under the exact-specified settings, we conduct sample generations for each configuration, across a spectrum of 10 different sample sizes n ranging from 10 to 10 can be seen from Figure (left) that the MI.E Gn empirically converges to the true mixing measure G. under the Voronoi metric Dj at the rate of order O(n -0.99,, which matches the theoretical parametric convergence rate established in T'eeore
1.3 Over-specified Setting
Under the over-specified settings, we continue to generate 5 samples of size for each setting, given
10 different choices of sample size n E [103, 109). From Figure (right), we observe that the MLEE G, empirically converges to G. under the Voronoi metric DD at the rate of order O(n Onn-0.47), which aligns with the theoretical parametric convergence established in Theorem 3.2
J Exact-Specifted Setting
In this appendix, we study the theoretical behaviors of the MLE under the exact-specified setting, i.., k k.. of the Laplace gating Gaussian MoE. We demonstrate that under the exact-specified setting, the rate of estimated conditional density function PG, PG. parametric On On-1/22 (up to some logarithmic factor).
Theorem J.1. The density estimation Pe.. (Y converges to the true density pG. (YIX) under the Total Variation distance V at the following rate:
Ex(V(pa, (1x)), ((xx)) 0 ((oooo())/)))
The proof of Theorem J.1 can be done similarly to that of Theorem 3.1 Appendix K.2 The result of Theorem J.I indicates that as long as we can establish the lower bound the total variation distance between Pê., and PG. based on certain loss function between the MLE G, and the true mixing measure G., we directly achieve the rate of the MLE under that loss function.
Voronoi Loss We now define that loss function between the MLE and the true mixing measure for the exact-specified setting:
Di = Σ
Σ VeA,,
exp((,) exp(37) +
Σ Σ exp (xp(3,)0. (1. 1,1,,
(9)
j=1
JEIRJHAA,E=LEE,
Above, for any (p1, P2, P3, P4) E R we define D2y (e1, P2, P3, P4) ||; +1+aa 11a=-1
bb, - 67|ps |t - 1/ for any i E A, and E [k,]. We demonstrate in the following theorem that
32

---

the rate of MLE to the true mixing measure under the Voronoi loss function D1 is OO (up to some logarithmic factor).
Theorem J.2 (Exact-specified setting). When k = ki is known, the following Total Variation bound holds guarantetrue for any G E (+))::
Ex (x (1 (1x)) Di(G,G.).
Therefore, we have Di G.) = ((oooo(())/))
The convergence rate of MLE under the Voronoi loss function Di implies that the rates of estimating the true parameters WT. aq b are also (/221 (up to logarithmic factors). These rates are comparable to those under the exact-specified setting of softmax gating Gaussian MoE (cf. Theorem 1 in (64])
Proof of Theorem 122 is in Appendix K.1
K Proof of Theoretical Results
In this appendix, we provide proofs for all theoretical results in the paper. Throughout this appendix, for any vector E IRd and (0), 02,..., Odd) Nd, we denote v 1 2/2 U. |0] "n + 02 + + and a! == a1!ao!.... a.l.
K.1 Proof of Theorem J.2
First of all, we need to establish the following bound:
(+xx)) (ptVVpp-l)))) (G G.).
For that sake, it is sufficient to demonstrate two following inequalities:
(Vx((a())))).m (1x)) > 0;
Inequality A. infces... (G.G.))
DI
Ex (x(V<pa(-)XX (+X))] 0,
Inequality B. (o)
D1 G.)
for some constant e' >>> 0.
Proof of inequality The inequality equivalent to
Ex Ex(V(pa(-IX)) PG. (-1x) 0. G.,
lim 8 GESu. (GG.GEE
inf
Assume that the above inequality is not true, then, there exists sequence of mixing (O) such that both Di Di G.) and
measure Gn = Eh. (8 as oer) Ex[V (1X). PG. ((X)))/D G.) go to zero as n + co. Now, we define
A (,.) = E [k.) |
011 S 110 07 サイ).
for any j E [ki,] as Voronoi cells with respect to the mixing measure G, where we denote 0 == (WT,, af 6% v!") and 03 == (W; aj. v2, In this proof, since our arguments are assymptotic, we can assume without loss of generality (WLOG) that these Voronoi cells does not depend on n, that is, A, = AG. Next, it follows from the hypothesis Dm, := Di (G G.) + as n + oo that each Voronoi cell contains only one element. Therefore. we may assume WLOG that A, = {i} for any [k=]. which implies that (W" a b ./5") + (Wj aj, 65, v/5) and exp( + exp(37) as
WWjaa,.
+ cc. Then, the loss function between G and G. given by
D, (G,, G.)
Σ
[|AWT]| + | | + ||v²)]) + exp(33) -
-xxp(3"))cxp(33))) exp(3:)
1=1
where we denote A = 3
3 Aap = af a;, A = b% - by and Au = up V.
Now, we break the rest of our arguments into three steps:
33

---

Stage Density decomposition:
In this step, we aim to decompose the term Qn
Σ -
- x11++33)
1pc, (
PG. (YIX),, which can be represented as follows:
Qn
exp((35)
H(Y(x:WT) - - -
+
Σ -xx(()")
(F(((//)))(((XXXXXXXXX a7,7 -
==1
An Bn En:
(10)
where we denote F(Y|X: x;; W, a, b, v)
(I) + b) and HY W) exp(-I|W - ()).. (Y By applying the first-order Taylor expansion, we can rewrite A,, as
exp(3,) a!
An
(4)))))) (A
i= [=|=1
glail+|al +astas F am Da 0hes Hu
+ RX(
h,
exp(39))
(AW?) ( (A (Av,))
ad
d|alg x Oaa (7;
alazltag f 0 Syas
+ 4,,, (x x + (X,,),,
where YX, is Taylor remainder that satisfies R1 (1 + as n - x and g(X, W) = exp([WW XII.. Recall that f the univariate Gaussian density, then by denoting b); == a X + b, we can verify that
go.
1 geas f (a TX + +,,,))
(a((a) x b7, 2a
aaa
Th
Consequently, we get
exp(8")
Ap
(AWT) (Aa (Au)))
20
slanlg
glazltag+ ff
x X
(X;W2)
(Y((a))
X + +b,, + R)(X,Y)
aWai
1h 0o +0s+204
1-|an|
2(1 1
Jai (aal)
exp(
Σ
Σ
Σ
(Aa))) (Au)))
20
101
02)
05aa tao 1la 1002
glalg lg
glaaltyf
x 00
OWai (X:W,).
(Yl(a)) TX +x,,,44 + RX,,Y,
2h 1al0
(11)
where we denote 77 03 2ag N.
34

---

Subsequently, we also apply the first-order Taylor expansion to the term B,, defined in equation I0 and get that
Bn exp(
H
(AW?))
+
exp (3
0 9 (AW?))
(X; W; Pc. po. (o., (2(X,
(12)
where R2(X, Y) is a Taylor remainder such that (x D1 (G, G.) - 0 n - 000 From the above results, the term Qn can be rewritten as
k. 1-/a. 1-aa))
_ blailg g
Slaaltns
Qn
S Siamono
(x; W.)
oh ahlallnn (YI(a))T x + +0, v4)
OWa1
i= |a
|=0
|=
0hlg
WT)P,,, (XIX)++++))))))))) + (X( R2(X,Y),
(13)
Ow7
i=1
7/=0
in which we respectively define for each i E [k.] that
cxp(
S ",100.07
Σ
2
03 20.
$50s+0455
102l
exp(
T
7 (AWT)"
for any (a), 02 11) # (O,, O2, 0) and * O2. Otherwise, 00 TT,o. == exp((")) ex))
Stage Non-vanishing coefficients:
Moving to the second step, we will show that not all the ratios Sy / (G,, G.) and Th, / (Gn; G.) tend to zero as + co. Assume by contrary that all of them approach zero when n + oo, then for (a1, 022, 1) = (O,, Og: 0), it follows that
S [
exp( - exp(8:) = Σ
+
(14)
D (G G.)
Di G.)
Additionally, for tuples (a1, 022 where a1
fer, (2.,ea...,,aal ca)
with ej
(0, (O...., O, O, O,...,.0 0), 02
O2 and 1j O, we get
j th
S Sayaa
Σ exp( exp(3))|AWT11
Σ i=1
- 0.
DG G.)
DI Gx)
For (ai, 022 11) where ai Og, 02 {EI, ,2. eaf and 17
have
S a22. D (G,, G.)
0.
(G G.) 1=1
For (a), (1) where a = 02 = 0₂ and 17 = 1, we have
1
Σ
k.
Sm | 0. Dr G.)
Di G.) 1=1
For (a1, 022 1) where a1
022 Oa and 77 2, we have
4.
5n
Dz G.)
exp( | Av
=
D Gn G.)
i, .n01 + 0.
35

---

As result, we achieve that
(p)) |AWT]I |I || |l + | + |
- 0.
D1 GG G.)
Due to the topological equivalence between norm-l I and norm-2, the above limit implies that
exp(
(IAWT |AWT
|| |I + | | 0.
(15)
Di G.)
Combine equation 14 with equation 15 we deduce that Di G.)/DI G.) 0, which is a contradiction. Consequently, at least one among the ratios S 201,0200
/ / G.) and
T, /Di (Gn, G.) does not vanish as tends to infinity.
Stage 3- Fatou's contradiction:
In this step, we use the Fatou's lemma to point out a contradiction to the results achieved in Step 2. In particular, we denote by my the maximum of the absolute values of Di..,,a2.7/ 'Di (Gn, G.) and
T (Gn, G.). Since at least one of the previous ratios does not converge to zero, deduce that 1/m,, A 00.
Recall from the hypothesis that Ex (pG., cording to the Fatou's 's lemma, we have
G.) - as n - 00. Ac-
Ex Ex(V(ee (pe.. (.IX), PG. (1x))
PGG. (V PG. (V
lim
inf
d(x, Y) 2 0.
71-000
(G,,n, G.)
2
n
Di G.)
This result indicates that Ipa., (YIX) - PG_ (V(X)I/Di(G, G.)
tends to zero as goes to infinity
for almost surely (X, As result, follows that
On
|pc.. (YI PG. (YIX) O.
lim 2-000 m. (I G.)
lim n-000
m, D2 GGn, G.)
Next, lct us denote ST Stan. [m, (Gn, G.)l + Stan 022n and T [m, Dj (Gny Gz)] + Kij with note that at least one among them non-zero. From the formulation of On in equation 13 we deduce that
la
2212 (=1z))
glarl g oW
Σ
Sia
X
(X;W,)
(Y((a)) TX + b2,)
Dh 12l
7=0
@hlg
Kiny
(X;W;) PP,, Pc., (Y[X) = 0,
ow
h7==0
(16)
for almost surely (X The above equation is equivalent to
h.
lul 2(1
- (oz)
go2-nf
E
X
ahartn (a X + v7) + Kt, (X
|02==0
glarl l
x aW
(X: W;") = 0.
for almost surely (X,Y) Y). It is worth noting that parameters W W are pair-wise distinct, { aanl g
thus, the set that aawa (X; WF) : [k.]. S lail S $1}
a linearly independent, which implies
Jas| 2(1 Jail (azl)
Eio X 8a2tn f
0h (Y (a;) X + bi, v,) 8i PG. (Yl 0,
00/==0
36

---

for any i E [kx]. 0 S 01 < 1 for almost surely (X. since (ai, bi, uÿ), ,..., (at, 1
Moreover,
v2.) have pair-wise distinct values, those of ((at) + bi v)) .... ((17.) ]x X Ui,, 1/h)) are also pair-wise different. Therefore, the set
x gattns f
omlaty (a) X + v;), Pc. (YIX):
s 02 S |ai) 0 S 2(1 - - |ai| -a22))
is also linearly independent. Consequently, obtain that .i....2.77 K44, 0 for any [k:,], 0 S |ay| + 02 & 1, Σ S 2(1 |
|02)) and 0 I hy I 1, which contradicts the fact that at
least one among those terms is different from zero.
Hence, we can find some constant
e'
such that
Ex (V
(p) > 0.
inf GE (G...)))
D1(G,G.)
Proof of inequality B: Assume by contrary that the inequality B does not hold, then there exists sequence mixing measures G (O) such that Dj (G G.)
s' and
Ex [V (po; (-1X), PG. (-12))
lim
0.
n
(G G.)
This result leads to Ex (PG, (+I),, PG_ (- (+X)) - 0 as + co. Recall that is compact set, therefore, we can replace the sequence G", by one of its subsequences that converges to mixing measure G' Gk. (6). Since D2 DI(G", G.) > e', this result induces that Di G.) e'.
Subsequently, by means of the Fatou's lemma, we achieve that
lim (p(2V(pa; ( (-IX), (-1x)) lim inf pay, (Y - PG. (YPX) d(X,Y). n=00
n-+00
It follows that par (Y PG. (Y for almost surely (X, Y). According to Lemma L.1 the noisy top-K sparse softmax gating Gaussian mixture of experts identifialle, thus, we obtain that G" ='= G.. As consequence, we obtain that (G',, G.) = 0, which contradicts to the fact that DI(G", > 0.
Hence, the proof is completed.
K.2 Proof of Theorem 3.1
In this appendix, employ results for M-estimators in 991 establish the density estimation rate under the Laplace gating Gaussian mixture of experts (MoE).
Firstly, we introduce some necessary notations and fundamental results. In particular, let Px(O): := {po (Y|X) G E (x())) be the set of all conditional density functions w.r. mixing measures in Gn (O). Next, we denote by [(e,, (+) || Ilxo) the covering number of metric space (Px (9), || llxo). Meanwhile, (e, (+), h)) stands for the bracketing entropy of (+) under the Hellinger
distance h where h(p,g) q) [ J(vp vq)2du) 1/2
for any probability densities p. q dominated
by the Lebesgue measure p. Then, we provide in the following lemma the upper bounds of those terms.
Lemma K.1. If is a bounded set, then the following inequalities hold for any 0 < < 1/2:
(i) log ((, Pa(O), || (1oo) S log(1/n);
(ii) (n (x) h) & (og(1/n).
Proof of Lemma
is in Appendix K.2.2 Subsequently, we denote
(0) = (PIGHG. G (u)):
(1) 1p/ (YlX) Ge (e)).
37

---

In addition, for each 8 > 0, we define a Hellinger ball centered around the conditional density function Pc. (X) and intersected with the set Pa/2(() as
kk (0, = {p:/² (() PG.) 5 8)-
To capture the size of the above Hellinger ball, 91 suggest using the following quantity:
Jn 11 (0.5) = 18 H (t. 11/2 (0, -Il)dd v 6,
(17)
where tVă: = maxft, 8}. Given those notations, let us recall standard result for density estimation
Lemma K.2 (Theorem 7.4. 191), Take v0() 2 JB(ò, Pk increasing function of 8. Then, for some sequence (8n) and universal constant which satisfy vnda 2 (8), we obtain that
/1/2 (0, 6)) such that Ye (6) is non-
P (Bxx MPe. (1x). > 6) S cexp(-no0
for any 5 2 6n
Now, we ready to provide the proof for convergence rate density estimation in Theorem Appendix K.2.1
Proof of Lemma K.2 can be found in 91
K.2.1 Main Proof
It is worth noting that for any t >> 0, we have
(t (O, t), -12))l))) S HB(t, (X,,
Then, the integral in equation 17 is upper bounded as follows:
8
H (t, h)dt / 62
8
Jo JB (0,
log(1/t)dt v 6, (18)
62/21a
where the second incquality follows from part (īi) of Lemma K.1
As a result, by choosing (0) = 5 /ogg(1/8), we can verify that (6))52² is non-increasing function Furthermore, the inequality in equation 18 indicates that () 2 J8(6, / (9, 6)).
Next, 1et us consider sequence (6,) defined as 6, == Vlog(n//n. This sequence can be validated
to satisfy the condition vndō 2 (d) for some universal constant c. Therefore, by Lemma K.2
reach the conclusion of Theorem J.I
P (EEx|h(PE, (. (+IX)) Cy (olog())/)) In
for some universal constant C depending only on e.
K.2.2 Proof of Lemma
Part (i). In this part, we will derive the following upper bound for the covering number of metric space (P) Iloo) for any 0 < 1 < 1/2 given the bounded set Q:
log N (7, (e). l| )|x) Z log(1/n).
To start with, we denote = {(a, (.)) E Rd x x R (3, W a, b, D) E S.. As θ is a bounded set, the set Ω is also bounded. Therefore, we can find an covove of Ω denoted by stp. Additionally, we also define A == {(8, W) E IR x Red (3, W b,,bb,v) E 22), and A, be an cocover of A. Then, it can be validated that
Ptp < O(n (d+2)
A s On (d+1)k)
Next, for each mixing measure G Σ count two other mixing measures. The first measure is G' Σi= ( where
E Ge(O), we take into ac-
38

---

(T,, b,, vy.) E Sn is the closest points to (a,, bg, 14) in this set for all i E [k]. The second one is
G G: exp( (3:) (w, ........) in which (B,, W.) E Dg for any E [k]. Next, let define
T: = (Po E Pa(O): W.) E An (T,, by, p.) E m,, Vi e [k]]
then it is obvious that Pc: E T. Now, we will show that T is an ry-oover of metric space (Px( || (lo)) with note that it is not necessarily the smallest cover. Indeed, according to the triangle inequality, we have
IPG Palls & lleG - Pa loc + lIpG - Pa llac.
(19)
Since the softmax function is no greater than one, the first term the right hand side can be upper bounded as follows:
IPGG - Po oo
sup softmax - xII+ ( 6,, 5(X x + 6, vi)
[(Y|a,
Xex
sup [(Y|a X u. M(aaa X bi, vz)
Xex
||a: - aiحl + |bb; - bull + ||u - -vill)
sup XE.
||a: a.ll + |16: - -211 + ||: - 1zll
1=1
S n
(20)
Subsequently, we bound the second term IpG. - Pez loo as follows:
||p Polloo
sup softmax(-1111 -II,,- B.) softmax(- W, X11 + B,)
XEX
x |V X bree br)
softmax(- W|, -XII + 8.) coftmax(-1111) - X1I + B.3]
sup
XEX
||W, -x|1 + B, + I xII 3a
sup
XEX
sup [W - Will + 1B: 3,1l
I 7
(21)
follows from the results in equation result indicates that T is an np-cover of the metric space (Px(O), loo). As consequence, we obtain that
19 equation 20 and equation 21 that ||p Pc S n. This
Nnn. Px))), ll (l)) A, x 1 S (1
which leads to the conclusion of this part: log N(7, Pa( 11 < log(1/n).
Part (ii). In this part, we provide an upper bound for the bracketing entropy of Px(9) under the Hellinger distance h:
HB (n, (x) h) log(1/n)-
Since θ and X are bounded sets, there exist positive constants 7, e, u such that -7 S T x
and u. Let us define
B(Y|X)=
exp
y²2
8u
for |Y| 2 27
for YY < 27
V2nt
39

---

Then, it can be validated that f(Yla X +b, I B(Y for any (X,Y) E X x
Next, let " which will be chosen later and (pr...., (P.....-PN) be (-cover of metric space (Px(O), || |(x)) with the covering number N = ((c,Px(()) |l)) Additionally, we also
consider brackets of the form (VII())) V (V[X) where
(Y]X) =- pa - 5.0)
/// (Y]x) max(p + ç, B(Y])))
Then, we can check that Px( S I. min{2ç, ())]
(X]x) and u (Y]x) v((((x) s
Let == max (2,, v (u) log(l//)))/0// (1/S), we have for any E [M] that
(V|x) (X((IX) d(x,)) + /
Vi 7 11
- - ((X,))
18|<27
1X1227
S²
SC +
exp
< S'C,
2u
where S' is some positive constant. This inequality indicates that
(, II)) < log ((C, |1 s log(1/S)-
By setting {= n/S', we obtain that (n Ps(O), ||- (11) & log(1/n). Finally, due to the inequality h < 11, we reach the conclusion of this part:
Hn(n, Px(O), h)) & log(1/m)
Hence, the proof is completed.
K.3 Proof of Theorem 3.2
In order to establish the following Total Variation lower bound under the over-specified settings, i.e. when k > k. is unknown:
(1 (+x)) (x(V(pc(1XX) (GGG..))
we need to prove two following inequalities:
Ex (V (x),((1))Pa. 0;
Inequality infce....
D2
Inequality B. inf Ex (PG((1-X) PG. (-1x))] > O.
G.)
for some constant e' > 0. As the inequality B can be achieved in the same fashion as in Ap- pendix K. we concentrate on showing the inequality A in this proof. For that purpose, it suffices to prove that
Ex (pa( (-), (-1X)) 0.
lim - C.G.
inf
(22)
D2,
Assume that the above claim does not hold true, then there exists a sequence of mixing mea- sures Gn = Σie exp(8r Jewr ot wP)) E (x( such that both the terms D2 G.) and Ex(V(pG. (.))),())))/D G.) go to zero as + co. Let us recall the formulation of the loss (2, G.):
D2 G.) = Σ
Σ exp( |AW + || + I + | +
[A;121
+ h exp(3") [IAWA] + +|A4,ا]1 + + + +
Σ Σ
-
jE[R.), AEAr [A:==1
1=1
iEA,
(23)
40

---

Since D2(Gn, G.) G.) + O, we deduce that EiEd, exp(35) + exp(6;) and (W?" a. .4)) 7 (Wj. az D3. uz) for all E A, and j [k.].
Now, we reuse the three-step framework Appendix K.I
Stage 1- Density decomposition:
Firstly, by abuse of notations, let us consider the quantity
k+
Qn
XII+ [3]] [pc.. (YIx) - PG. (Y|x)]
Similar to Step in Appendix K.1
we can express this term as
h-
Qn exp(
j= TEA,
exp(
jEA
Σ
exp( -xp((
-
j=1 iEA;
= An
Bn
+
En:
Next, we proceed to decompose An based on the cardinality of the Voronoi cells as follows:
An = Σ
Σ
exp(
F(Y|X: WT" af, UG, uF)) - F(Y W3, 43. vj)
;:Ay iEA,
Σ A,,,,=iE,,
Σ
exp()")
F(YIX: of, 8y v!)) F(Y| W7,
By applying the Taylor expansions of order 1 and r, the first and second terms of A,, respectively, and following the derivation in equation II we get that
2(1-/oa1-1o2)
exp(3")
An Σ
Σ -1 iEA,
Σ
Σ as+204=
(AW3) (Dat))
20s
ji -
100|=0 10
m=0
D5agta451 (on 102l
Jlan g
glazl tf
x (A a0 (AV X02
(Yl(a)) x + 05, + Rx(X,Y)
lai
2(F
lan ||aal)
exp(3")
Σ
Σ
(AW (Dat))
204
1: 1|A, |>1 iEAd
|00|
0=0 |02|=0
as as+2a
0 0Sagto45rJ - (m 1m1
glanl g
glaz|+
x (Ab 03 (Ap
aWai (X: Wj)
a |a2lto (YI(a)) x R4
where Rx(XY) is a Taylor remainder such that / G.) + 0 as n + xa for e (3,4) 4). Next, we apply the Taylor expansions of order and 2 to the first and second terms of B,, respectively, and following the derivation in equation 12
we get that
BB = ΣΣΣ
exp(67)
(AW
abl g
(X; WW),,,, + +X(,Y)
!!
aw
HA, =1
h==
ΣΣΣ
(xp
(AW?)*
al g dw (X: )o. (X|X) Ro(X,
7
H h|
41

---

where Rs(X,Y) and Rs(X. Y) are Taylor remainders such that their ratios over D2 G.) ap- proach zero as n 7 cc. Subsequently, let us define
exp(
S Janaam
(AV))
2040!
SeA,
+ 224 0500 +04 Sty on |02l
cxp(
(A
for any (a1, 02 n) # (0,, Og, 0) and 7 * Og. Otherwise, S $0000000 T5,0, == Eien, exp(s?") exp(33). As consequence, follows that
R. 5 0n 2(r ||2l)
aoanl g ŌWai
glealtos S
On = Σ
S "an,a.01
02
(X:W))
@hlo2l41 (x)) x + bz.
j=1
|a
10
|
7
h. 14
d OW7
g
T
R R (X(,
==1
In
(24)
Stage 2 Non-vanishing coefficients:
In this step, demonstrate that not all the ratios S S /D2 (D2 G.) and Th (D2(Gnn G.) converge to zero as n + co. Assume by contrary that all these terms go to zero. Then, by employing arguments for deriving equation [4tiiioonnnan and equation [5] get that
(xp( exp(83)
D2 Ga)
EEA,
Σ
exp( (IAWT)) ||aaj]
| + |Av])]] 0.
5:|4, =1iEA;
Taking the summation Ej
18 D2(G..G.)
for (a1,02;1) where an E {2e1, 2e2, 2ea).
0- = 0g and = 0, we have
Σ
Σ
exp(3")) O.
D2(Gn, G.)
18 og
Taking the summation of Σ} D.(G,.G.)) for all (01,02, 7) where æn Od, 02
{2e1, .e....., ] and 17 = 0, we have
Σ
Σ
0.
(,,
Combine the above limit with the formulation of D2(Gnn G.) in equation 23 we have that
Σ exp(
Ab
+ |
2 A O.
D2 G.)
This result implies that we can find some index jr E [k]] |Ap.
that satisfies
Σ exp(8")
+
2 A
A O.
Dz G2) iEA,
For simplicity, we may assume that jr = Since ST.o, any S Tj, we divide this term by the left hand side of the above equation and achieve that
O..y/ Si.o,..... (D G.) vanishes as n - oo for
exp(
EieA, E as-22m=
(AbR) jos (AvR)
Isagtash
2a4
0,
(25)
EieA, exp(3?))
(16 P + Auv 4
42

---

for any 1 12 S. F1.
Subsequently, we define M, := max Ari 1/ : A1). As result, the sequence exp(3))) is bounded, which indicates that we can substitute with its subsequence that admits positive limit o == lim exp( / Therefore, at least among the limits zis equals to one. Furthermore, also denote
A1} and Tn := max(exp((")) :
(Ab)) /M, + 234. (2MM) - 218-
From the above definition, it follows that at least one among the limits 23: and 24 equals to either or -1. By dividing both the numerator and the denominator of the term in equation 25 by T,
we arrive at the following system polynomial equations:
Σ BEA1 02+2aa=1
Σ
24
O,
0:3! a4!
15agtai$h
for all 1 F1. Nevertheless, from the definition of F1. know that the above system does not admit any non-trivial solutions, which is a contradiction. Consequently, not all the ratios Sj.a an /nni.ann/////ae/e (D2(Gn, G,) and /D,//DD(Gm G.) tend to zero Stage 3- Fatou's contradiction:
Recall that Ex(V (-IX), (. ((X))) G.) + a + oc. Then, by applying the Fatou's lemma, we get
0 0= lim Ex(V (-IX), PC. (-1x))
1
inf inf IGG.. (Y[X) (YIx) d(,,Y). n-030
lim
D2(G,n G.)
2
D2(Gn...)
which implies that Paa. (Y (X (XIX))//PPGG...... + Oasn + oo for almost surely (X, Y).
/ Next, define m, the maximum of the absolute values of /j /D2 G.). It follows from Step 2 that 1/m,, A co. Moreover, by arguing in the same way in Step in Appendix K.I we receive that
Qn//m,D2(G.., G.)] +
(26)
oc. By abuse of notations, let us denote
Sjanaa boaan//mm [mn D2(G,n G+)] + f1,00,0mm
/i/lmm D2 G.)] - Kjn.
Here, least one among Ej,0 orr oz Kjn non-zero. Then, by putting the results in equation 24 and equation 26 together, we get
F 1--lan120 2(F,
ala g OWai
flazlt
(X;
Wx)
(Y + +5,,
ah 1azl+n
1=1
1on
0
la2
14
glng g aw
Kjn
X: W PG, (Y|X) = 0.
h|=0
Arguing in a similar fashion as in Step 3 of Appendix
K.I
obtain that {j.o Kj,y 0 for
any j E [k=,], 0 S |an| + |a2| < 2F,, 0 < 17 S 2( This contradicts the fact that at least one among them is non-zero. Hence, the proof is completed.
an - la2l) and N hn S 1 ++
L Identifiability of the Laplace Gating Gaussian MoE
Lemma L.1. For any mixing measures G and G. in Ge(O) that satisfy pa (V|X) = (Y|x) for almost surely (X,Y) e X y. we have that G = G.-
Proof of Lemma L. First, we assume that two mixing measures G and G, take the following (xp(3.))(,,,,a))) and G, Σh. exp(3) )ow; (. Recall that
forms: G
Σ
43

---

X) = PG. (YIX) for almost surely (X, Y), then we have
softmax(- - I[W, xII + B.) A(Ylay X b2,
softmax(-||1a) - - +
+33)) + +22,
(27)
Due to the identifiability of the location-scale Gaussian mixtures and
85 86
we get that k.
coftmax(-11lW)) X|| + 3.) : [ softmax x|| : [+]]
for almost surely X. WLOG, we may assume that
softmax(-1114
X|| + 3:) softmax(-||w"" X + 81).
(28)
for almost surely X for any E [A]]. Since the softmax function is invariant to translations, it follows from equation (28 that W; W and 3, = 3 + up for some up E IR. Notably, from the assumption of the model, we have 53 = 3% 0. which implies that uo = 0. As result, we obtain that 3; 3
for any i [k..] Then, equation 27 be rewritten as
k-
- X b1
exp(3i) exp((()epp(-1 exp(-IIWW XID)((((0)X +b,
(29)
for almost surely (X,Y). Next, we denote J1, J2, ......... Jm partition of the index set [k.]. where m I k,, such that exp(3x) = exp(3r) for any i, 1 E Jj and j E [m]. On the other hand, when i and i' do not belong to the same set Jj, let exp(8:) * exp(3r). Thus, we can reformulate equation 29 as
Σ
Σ /=1ie,,
(xp(3))ex((-ll
- XV))S(Y( a X + bi,
exp(33))xx((- - xI))(Y(((a)) x x +b,11,)
iEJ,
for almost surely (X, Y). This results leads to {((a)) xx Jj} (((aa)xx(;, x+b, i Jz). for almost surely X for any j [m]. Therefore, have
{(a,, v;) Jj} 6,,,,,,1))))) J/}.
for any j E [m]. As consequence,
m
G Σ Σ
exp (( = G..
j=1iEJ;
1 xEJ,
Hence, reach the conclusion of this lemma.
M Broader Impact
This paper presents research aimed at propelling advancements in the broad domain of machine learning. The implications of our findings wide-ranging, with potential applications in sectors including healthcare, autonomous driving, and recommendation systems. Based on our current understanding, this research does not warrant an ethics review, and detailed discussion of the potential socictal impacts is not required at the current stage.
44

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: we have added separate contribution paragraph in the introduction section (section D. along with the highly summarized contributions in the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: see section 5 on discussion on general limitations. Section 3 and have also thoroughly discussed the assumptions and experimental settings.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to system might not be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
45

---

Answer: [Yes]
Justification: see section 3 and Appendix
J
K and L for assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Ys]]
Justification: we have submitted the implementation of the proposed methods and all base- lines in the supplementary material. Appendix B D information and implementation details.
and G also contain all dataset
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurlPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
46

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: have submitted the implementation of the proposed methods and all base- lines in the supplementary material. Appendix B and Glso contain comprehensive information reproduce the experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips) cc/
public//uuides/Coeettbii.ss sosiooooooiisssssooooooocc for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
http:psscccpuubcccccccccee wwwiiiigggde////eeee Code ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Ye]]
Justification: dataset information and preprocessing procedure be found in Appendix B and D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: all results are averaged over 5 random experiments, as specified by the exper- iment section 4 and additional results in Appendix H
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
47

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative еπτог rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: computational resources used to run the experiments can be found in Ap- pendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpgs//paurrppp.../pbbbb/////tt
Answer: [Yes]
Justification: have reviewed the NeurlPS Code of Ethics and make sure the paper con- forms to this.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: we have attempted discuss the impacts of this work in Appendix M Guidelines:
The answer NA means that there is no societal impact of the work performed.
48

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: our paper has no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions to access the model or implementing safery filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: all the assets used in the paper have been properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
49

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, wwtpppppppppoooooo
com/datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Ys]]
Justification: have provided documents in the code repository submitted in the supple- mentary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can cither create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: this paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, cura- tion, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects. and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: this paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
50

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
51


Reviews:
[{"review_id": "sPmTsKlUdX", "reviewer": "Reviewer_P6x6", "summary": "This papers proposes a novel MoE arcchitecture for handling and fusing multiple modalities, along with two core contributions:\n- a novel router design that can handle missing modalities;\n- a laplace gating function that is theoretically proven to ensure better convergence.", "questions": "- 1 In table 5, Laplace gaiintg shows limited superiority against other variants, any insight for it? \n- 2 In table 4c, why the method achieves better results with less modaltieis? Any further insight or ablations?", "limitations": "N", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- 1 This paper is generally well written and easy to follow; tables and figures are neat and informative; \n- 2 The motivations of core designs are well elaborated; \n- 3 Theoretical proofs are providied to further justify its effectiveness; \n- 4 Extensive experiments, along with diverse modalities. are conducted to validate the effectiveness of the proposed approach. \n- 5 The capabilty of handling missing inputs is interesting (Fig. 4 c).", "weaknesses": "- 1 The proposed method achieves promising results on the tested benchmark, which are still comparably small scale datasets, e.g, CMU-MOSI is with 2000+ samples. I am wondering if this work can be applicable to training/fine-tuning with larger scale data. \n- 2 Again, for vision task, only CIFAR-10 is used. This cannot fully justify its effectiveness for more general vision or multi-modal tasks;\n- 3 The capability in handling missing modalities is promising. The auhor is encouraged to explain why, in some scenarios, mising modalities + the proposed method surposses the variant with full modalities (Fig 4 c).  I am still wondering its generalizability under more scenarios. \n- 4 The visualization and quanlitative analysis of the learned gaing weight would help the reader better understand the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer `P6x6`,\n\nWe deeply appreciate your insightful comments and positive feedback. We are heartened by your recognition that our paper is well-written, with well-elaborated motivations. We are also pleased that you acknowledge the theory and extensive experiments we provided to justify the effectiveness of our method. Below, we address your questions and concerns in detail.\n\n**Q: The proposed method achieves promising results on the tested benchmark, which are still comparably small scale datasets, e.g, CMU-MOSI is with 2000+ samples. I am wondering if this work can be applicable to training/fine-tuning with larger scale data.**\n\n**A:** Yes, we have tested FuseMoE on large-scale multimodal datasets including MIMIC-III, MIMIC-IV, and CMU-MOSEI. As shown in Table 1 of the MultiBench paper [1], these datasets are all featured as large-scale multimodal datasets, with MIMIC-III containing 36,212 samples, MIMIC-IV containing 73,173 samples, and CMU-MOSEI containing 22,777 samples. Even though CMU-MOSI is relatively small-scale with 2,199 samples, our experiments overall have tested a sufficient number of large-scale multimodal datasets. These datasets span various data types including text, images, time series, ECG, video frames, and audio signals, demonstrating strong generalization capacity across different domains.\n\nPlease find tables summarizing the information of these datasets in the attached PDF file.\n\n**Q: Again, for vision task, only CIFAR-10 is used. This cannot fully justify its effectiveness for more general vision or multi-modal tasks**\n\n**A:** As explained above, several of the multimodal datasets we tested include vision components, such as chest X-rays in MIMIC and video frames in CMU-MOSI and MOSEI. Additionally, we have conducted further evaluation of the ImageNet dataset. Detailed information can be found in the attached PDF document.\n\n**Q: The capability to handle missing modalities is promising. The author is encouraged to explain why, in some scenarios, missing modalities + the proposed method surpasses the variant with full modalities (Fig 4 c). I am still wondering about its generalizability under more scenarios.**\n\n**A:** Full modalities mean that patient records with incomplete modalities will be discarded. In contrast, incorporating data with missing modalities allows us to access a broader array of samples, resulting in the utilization of all available patient data. As mentioned in Table 6 of Appendix B.1, for instance, when examining the 48-IHM or LOS tasks in the MIMIC dataset, the total number of patient samples is 35,129. However, the number of patients with clinical notes is only 32,038, and with chest X-ray (CXR) only 8,731. If we strictly enforce the inclusion of complete modalities involving CXR, the total number of samples available is constrained by this bottleneck.\n\nTherefore, being able to utilize incomplete modalities is a critical step in making use of the vast amount of patient data without full modalities. This approach will greatly improve generalization due to the increased number of available samples.\n\n**Q: The visualization and qualitative analysis of the learned gating weight would help the reader better understand the method.**\n\n**A:** We have included the visualization and qualitative analysis of the learned gating weights in Figure 12 of Appendix I. To better address your request, we have also added additional results on the CMU-MOSI and MOSEI datasets in the attached PDF file.\n\n**Q: In table 5, Laplace gating shows limited superiority against other variants, any insight for it?**\n\n**A:** In the results of Table 5, Laplace gating is still close to the best-performing method. We wish to emphasize that the advantage shown in Table 4 comes from the MoE architecture improvement. Most fusion methods listed in Table 4 are only suited to two-modality problems and cannot be extended as we scale to more modalities in Table 5. The comparison between Softmax, Gaussian, and Laplace gating functions in Table 5 reflects improvements within the same MoE architecture.\n\nThese two types of improvements are distinct: sometimes one factor (e.g., the MoE architecture) is more important than the other (e.g., the gating function). Additionally, the performance advantage of each method also depends on the modality combination. Adding or removing modalities can influence the effectiveness of the Laplace gating mechanism.\n\n**Q: In table 4c, why the method achieves better results with less modalities? Any further insight or ablations?**\n\n**A:** Based on the context, we believe you mean Figure 4c instead of Table 4c. We would like to refer you to our answer to your first question for more details on this matter.\n\nPlease let us know if there are specific aspects or sections you would like us to elaborate on further, or if you have any additional questions. Thank you!\n\nSincerely,\n\nAuthors\n\n**Reference**\n\n[1] Liang et al., (2021) MULTIBENCH: Multiscale Benchmarks for Multimodal Representation Learning, NeurIPS 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "64TbQwGq8N", "reviewer": "Reviewer_6Ay7", "summary": "The Paper introduces “FuseMoE”, a novel mixture-of-experts (MoE) framework that can handle multi-modal data even in scenarios with missing elements, sparsity of samples and temporal irregularity.  It proposes an innovative Laplace gating function with theoretical proof to enhance convergence rates and predictive performance across various tasks. The Laplace gating function allows sparser distribution of weights among MoE and encourages more balanced utilization of experts with sharper peaks and heavier tails preventing certain experts to be over dominant. FuseMoE includes modality and irregularity encoder using a discretized multi-time attention (mTAND) module to support generic fleximodal data ingestion with unlimited input modalities instead of other popular pairwise setup that exist in the literature. Paper also outlines various router designs for processing multimodal inputs and discusses the tradeoff between them and demonstrates .\n\nKey:\n\n1. Laplace gating function showed superior performance compared to Softmax gating across multiple tasks\n2. Unlike baseline models, FuseMoE demonstrated improved scalability with increasing number of modalities\n3. Handles missing modalities with tAND and per modality routers with entropy loss function\n4. theoretical analysis of convergence rates for parameter estimation\n5. FuseMoe outperformed baseline methods on several fleximodal datasets.", "questions": "1. The paper claims to work with unlimited input modalities. is it really so? how do you anticipate the fuseMoE to handle extremely sparse data across modalities? Any thoughts/estimation on how far we can go here? How does the complexity of setup scale with number of modalities?\n2. Is Laplace only tested with CIFAR-10 dataset? Why was it not tested with other bigger datasets?\n3. Have you thought about or explored the interpretability side of expert assignement process? \n4. Few other paper discusses and introduces Laplace instead of gaussian for MoE. Can you expand more on how FuseMoE is differnt from these?\n    \n   - Nguyen, Hien D., and Geoffrey J. McLachlan. \"Laplace mixture of linear experts.\" *Computational Statistics & Data Analysis* 93 (2016): 177-191.\n    \n   - Wu, Lc., Zhang, Sy. & Li, Ss. Heteroscedastic Laplace mixture of experts regression models and applications. *Appl. Math. J. Chin. Univ.* **36**, 60–69 (2021). https://doi.org/10.1007/s11766-021-3591-2", "limitations": "Potential limitation of over-parameterization of networks especially when input data/size is small is adequately discussed in the paper.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Sparse MoE backbone with novel Laplace gating function that ensures better convergence rates compared to traditional Softmax with theoretical guarantees\n2. Modality and Irregularity Encoder to use mTAND module to descretize irregularly sampled observations to mix with continuous features, Effective management of missing data and irregular sampling\n3. MoE fusion layer to integrate embedding from different modalities with different router designs. Flexibility to handle variable number of modalities because of per modal experts routing\n4. Novel methods, strong theoretical contributions and comprehensive emperical results and evaluation", "weaknesses": "1. Potential over-parameterization when input is small\n2. limited discussion of computational needs and scalability to huge datasets\n3. Most of the items discussed were already established and proved. Paper seems to be combining existing literature for these datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: Yes, we mentioned this as one of our limitations in line 316 of Section 5. This issue was observed during our empirical evaluation, where the Time2Vec method we employed transforms a univariate time series into a high-dimensional vector. This transformation helps capture trend/seasonality components and long-term dependency. However, when the number of samples is small and lacks sufficient information, transforming to a high-dimensional vector can be redundant and lead to over-parameterization.\n\nThis problem only occurred in a few scenarios among the numerous experiments we conducted. As mentioned in Section 5, we plan to address this issue in future work.\n\n**W2**: We have included a discussion comparing the computational efficiency of various methods in Figure 12 of Section H.3. Additionally, we have described the computational resources used in Section G.1.\n\nWe have also tested FuseMoE on large-scale multimodal datasets, including MIMIC-III, MIMIC-IV, and CMU-MOSEI. As shown in Table 1 of the MultiBench paper [1], these datasets are all featured as large-scale multimodal datasets, with MIMIC-III containing 36,212 samples, MIMIC-IV containing 73,173 samples, and CMU-MOSEI containing 22,777 samples. Even though CMU-MOSI is relatively small-scale with 2,199 samples, our experiments have overall tested a sufficient number of large-scale multimodal datasets compared to most prior works (e.g. [2]). These datasets span various data types, including text, images, time series, ECG, video frames, and audio signals, demonstrating strong generalization capacity across domains.\n\nPlease find tables summarizing the information of these datasets in the attached PDF file.\n\n**W3**: We respectfully disagree that we are merely combining existing literature. In our work, the MoE fusion layer and missing modalities sections are newly proposed and form important parts of FuseMoE. The theoretical insights of Laplace gating are another integral aspect of this paper, leading to extensive empirical evaluations under multiple circumstances. Most importantly, the new setting and motivation can potentially be connected to many real-world applications.\n\nWhile the encoder models we employed are based on existing work, this is primarily for demonstration purposes and does not constitute a significant portion of the context. We want to emphasize that this level of prior-work reuse has been common in many prior works, including our baselines from [2]. We truly appreciate your advice in pointing out this issue and will ensure to clarify this in the revised version.\n\n**Q1**: We wish to first emphasize that “unlimited” does not mean infinite. According to [3], most real-world multimodal problems do not exceed four modalities. FuseMoE can scale to recently introduced high-modality problems, which combine modalities from different tasks. The scalability advantage is particularly prominent when compared with the pair-wise cross-attention-based approach.\n\nFuseMoE is designed to address real-life multimodal datasets with missing modality or irregularity issues, which can be caused by malfunctioning equipment, different patient conditions, human-level decisions, etc. In principle, the current method can be applied to arbitrary sparsity or modality combinations, as we have only tested on available real-world multimodal datasets. However, we did not manually create sparse data to meet the potential “extremely sparse” criteria. If such situations occur, we can employ additional strategies to enhance the robustness of the current framework, such as leveraging hierarchical MoE where a specific subgroup of experts is assigned to handle missing modalities, and applying regularization or dropout during training to make the model robust to missing modalities.\n\nGiven that the model architecture remains the same, as the number of modalities N increases, the FuseMoE-based method scales linearly in $\\mathcal{O}(N)$, whereas the cross-attention method scales in $\\mathcal{O}(N^2)$.\n\n**Q2**: We have added additional results on ImageNet using the Vision-MoE framework for the Laplace gating, this can be found in the attached PDF file. In our paper, we have also tested the Laplace gating on MIMIC-III, MIMIC-IV, and PAM. Additionally, the results presented on the CMU-MOSI and CMU-MOSEI datasets utilize the Laplace gating function.\n\n**Q3**: Yes, please see Figure 12 (c) in Appendix H, which visualizes the modality contribution of the MIMIC dataset. Additionally, we have included the modality contributions of the CMU-MOSI and CMU-MOSEI datasets in the attached PDF file.\n\n**Q4**: There are three main differences between our FuseMoE and the Laplace MoE considered in the referenced papers:\n\n1. Conditional Distributions:\nUnder the FuseMoE model, the conditional distribution of $Y | X$, with the density given in Equation (4), where $X$ is an input and $Y$ is an output, is a mixture of Gaussian distributions. In contrast, the Laplace MoE model utilizes a mixture of Laplace distributions.\n\n2. Sparse MoE versus Dense MoE:\nIn FuseMoE, only a subset of Gaussian distributions is activated for each input $X$. Specifically, the conditional distribution of $Y | X$ is a mixture of K Gaussian distributions, where K is often set to one or two in practice. This mechanism is enabled by the Top K operator described below line 154. Meanwhile, in the Laplace MoE model, all Laplace distributions are activated for each input.\n\n3. Gating Kernels:\nIn FuseMoE, the mixture weights in Equation (4) use the Laplace kernel with the scale parameter set to one. In contrast, the gating kernel in the Laplace MoE model is an exponential value of a linear kernel.\n\n**References**\n\n[1] MULTIBENCH: Multiscale Benchmarks for Multimodal Representation Learning\n\n[2] Improving Medical Predictions by Irregular Multimodal Electronic Health Records Modeling\n\n[3] High-Modality Multimodal Transformer: Quantifying Modality \\& Interaction Heterogeneity for High-Modality Representation Learning", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PCPl3lhfln", "reviewer": "Reviewer_6SUH", "summary": "This paper proposes an MOE-based model to handle multimodal data fusion. It addresses two challenges: missing modalities and irregularly sampled data trajectories. A Laplace gating function is applied to the MoE Backbone. An entropy regularization loss is proposed to ensure balanced and stable expert utilization. The author validates the method in diverse datasets.", "questions": "Please see the weakness part to answer the questions.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Strong motivation. Two challenges presented in this paper, missing modalities and irregularly sampled data trajectories, are important in the multi-modal data fusion area. It is reasonable to address them through sparsing encoding and gating networks in the MOE layer. \n\n2. Good theoretical analysis. This paper presents a detailed theoretical analysis of the Laplace gating over the standard Softmax gating in MoE. Other mathematical proofs seem to extensively illustrate the characteristics of the proposed MOE model. \n\n3. Good experimental results. Comprehensive evaluations of FuseMoE on image and video datasets seem to validate its effectiveness.", "weaknesses": "Addressing the following weaknesses may improve the paper: \n\n1. The author should clearly distinguish the proposed method and others’ modules. From the paper, the Laplace gating function is proposed as a new one. I am not sure the author made some contributions to the encoder design, router design, and loss design. The author should make more illustrations about the contributions, not just combine other people’s work together. \n\n2. The experimental results are not extensive. The author should clearly demonstrate the data modality of the chosen benchmarks. It seems that other modalities, such as text, are not included. The author should explain this. \n\n3. Please explain more concisely how the gating functions can stabilize the imbalance and sparse multi-modal data. The paper should point this out more concisely and better with experimental results. Too many mathematical proofs and theorems in Sec.3 seem not helpful in illustrating the advantages of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer `6SUH`,\n\nWe are grateful for your positive feedback and insightful comments. It is particularly encouraging to hear that you found our manuscript to have strong motivation and address important challenges, with good theoretical analysis and comprehensive experimental results. In the following, we address your major concerns in detail.\n\n**Q: The author should clearly distinguish the proposed method and others’ modules. From the paper, the Laplace gating function is proposed as a new one. I am not sure the author made some contributions to the encoder design, router design, and loss design. The author should make more illustrations about the contributions, not just combine other people’s work together.**\n\n**A:** We respectfully disagree with your claim that we are merely combining existing literature. In our work, the MoE fusion layer and missing modalities sections are newly proposed and form important parts of the FuseMoE method. The theoretical insights of the Laplace gating function are another integral and unique aspect of this paper, leading to extensive empirical evaluations under multiple circumstances and applications. Most importantly, the new setting and motivation can potentially be connected to many real-world applications.\n\nWhile the encoder models we employed are based on existing work, this is primarily for demonstration purposes and does not constitute a significant portion of the context. We would like to emphasize that this level of prior-work reuse has been common in many prior works, including our baselines from [1].\n\nNevertheless, our intention was not to combine other people’s work but to provide a clearer demonstration. We truly appreciate your advice in pointing out this issue and will ensure to clarify this in the revised version.\n\n**Q: The experimental results are not extensive. The author should clearly demonstrate the data modality of the chosen benchmarks. It seems that other modalities, such as text, are not included. The author should explain this.**\n\n**A:** We have created a table summarizing the modality components and sample size of each dataset in the attached PDF file. These details can also be found in the dataset details section in Appendix B. The datasets we tested include multiple modalities such as time series, text, image, ECG, and video frames.\n\n**Q: Please explain more concisely how the gating functions can stabilize the imbalance and sparse multi-modal data. The paper should point this out more concisely and better with experimental results.**\n\n**A:** We start with the computation of gating functions to demonstrate the advantage of Laplace gating. For each token $\\mathbf{h}$, and each expert  $i$ with embedding $\\mathbf{e}_i$, the similarity score  $s_i$  is computed as:\n\n$s_i = \\mathbf{h} \\cdot \\mathbf{e}_i,$\n\nwhich represents the affinity between the token and the expert. These scores are then passed through a Softmax function to obtain the gating probabilities $g_i$:\n\n$g_i = \\frac{\\exp(s_i)}{\\sum_{j} \\exp(s_j)},$\n\nwhere the gating probabilities determine the weight or importance of each expert in contributing to the final decision. **Representation collapse** occurs when the Softmax gating mechanism consistently assigns high probabilities to a small subset of experts, causing these experts to dominate the decision-making process while the others become redundant. This issue arises due to:\n\n- If a few experts have significantly higher similarity scores  $s_i$ than others for most tokens, the Softmax function will amplify these differences, leading to very high gating probabilities for these experts.\n- The exponential nature of the Softmax function makes it sensitive to differences in similarity scores, causing the highest scores to dominate.\n\nAs for the Laplace gating function, instead of using the inner product, it computes the similarity score  $s_i$  as the negative L2-distance between a token’s hidden representation  $\\mathbf{h}$  and an expert embedding  $\\mathbf{e}_i$:\n\n$s_i = -\\| \\mathbf{h} - \\mathbf{e}_i \\|_2,$\n\nwhich is a distance-based similarity measure. The $L_2$-distance measures how far apart the token representation and expert embedding are in the feature space. This approach does not inherently favor any expert based on magnitude, unlike inner product which can be biased towards experts with larger norms. By considering the distance, the Laplace gating function ensures that all experts have a more balanced opportunity to be selected based on how close they are to the token representation, rather than being dominated by a few experts with higher dot products.\n\nWhen dealing with heterogeneous inputs, such as multimodal data (e.g., text, images, time series), the feature distributions can be very different across modalities. The $L_2$-distance is a more robust measure that can handle these differences without being overly sensitive to the scale and variance of the input features. In addition, it can gracefully degrade in the presence of missing data, rather than causing abrupt changes in gating probabilities that might occur with inner product-based measures.\n\nFor experimental results on the Laplace gating function, we have added results on ImageNet using the Vision-MoE framework, which can be found in the attached PDF file. In our paper, we have already tested the Laplace gating function on MIMIC-III, MIMIC-IV, CIFAR-10, and PAM. Additionally, the results presented on the CMU-MOSI and CMU-MOSEI datasets also utilize the Laplace gating function.\n\nSincerely,\n\nAuthors\n\n**Reference**\n\n[1] Zhang et al., (2023) Improving Medical Predictions by Irregular Multimodal Electronic Health Records Modeling, ICML 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gJIu8NSC9T", "reviewer": "Reviewer_sZSN", "summary": "The paper “FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion” introduces a novel framework called FuseMoE, designed to tackle challenges associated with multimodal data in machine learning. This framework addresses key issues such as missing elements, temporal irregularity, and sparsity in data, which are prevalent in fields like healthcare where data is often incomplete and irregularly sampled. The core innovation in FuseMoE is its gating function that integrates various modalities efficiently. The framework enhances predictive performance through: 1. Sparse MoE Layers: Incorporating sparsely gated MoE layers to manage tasks and learn optimal modality partitioning. 2. Laplace Gating Function: A novel gating function theoretically proven to ensure better convergence rates than traditional Softmax functions. 3. FlexiModal Data Handling: Efficiently handling scenarios with missing modalities and irregularly sampled data trajectories. The framework is validated through empirical evaluations on diverse prediction tasks, demonstrating its effectiveness in real-world applications.", "questions": "Please refer to the weaknesses.\nThe biggest question at my side is that the similarity with gaussian gating may lead to some theoretical misunderstanding. Any further explanation on this topic could lead to higher score.", "limitations": "The authors adequately addressed the limitations, there is some redundancy in the encoder part that heavily draws inspiration from UTDE's work. From the appendix it can be observed that encoder might have a greater impact on overall performance than what gate function alone contributes to improvement; hence further discussion is needed on whether this gate function must necessarily be tied to methods related to mTAND and imputation for obtaining higher-quality features as inputs for downstream MoE layers from an implementation perspective. Of course, the discussion on this topic is optional.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Existing demand of a more flexible gating function in the field of multimodal MoE makes the idea of this paper promising. Also, the possible implementations of this new MoE system is promising and may lead to some practical medical tools.\n2. The theoretical proof is sufficient and rigorous.\n3. The experimental design is comprehensive from the perspectives of scenarios and data, to a certain extent corresponding with the theory. The appendix contains a large amount of supplementary information, which helps verify the completeness and correctness of the article.", "weaknesses": "1. The comparison works are all very outdated (the latest one UTDE[1] is publicized in 2023, with over half of the compared works predating 2020). Why not compare with more recent works?\n2. Fig.1 lacks clear description, as the data processing workflow is not well explained.\n3. Gaussian gating and your proposed form are very similar in format, with formula $h(x)=\\operatorname{Top} \\mathrm{K}\\left(-\\|W-x\\|_{2}^{2}\\right)$. Thus, theoretical comparison with original softmax may be unfair as the form is derived from and similar to gaussian gating. Therefore, although theoretical results may be valid, their importance diminishes due to this aspect.\n\n[1] Improving Medical Predictions by Irregular Multimodal Electronic Health Records Modeling", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer `sZSN`,\n\nThank you very much for your constructive suggestions. We are encouraged by your acknowledgment that our idea is promising, the theoretical proof is rigorous, and the experimental design is comprehensive, containing a large amount of supplementary information. Below, we address your concerns in detail.\n\n**W1:** We wish to emphasize that all the baselines we compared are well-known and representative methodologies within specific categories: concatenation, Tensor Fusion, Multimodal Adaptation Gate, and cross-attention-based methods. The MoE-based fusion method is a completely new line of work, and the comparison is essentially within the concept of fusion approaches.\n\nTo the best of our knowledge, based on the latest baselines we compared with, there is neither new work that opens up a new line of fusion methods nor any work that is far superior to the method and baselines we presented. However, in response to your comments and those of other reviewers, we have added additional experiments and ablation studies on new datasets and settings to further strengthen our results. These can be found in the attached PDF file.\n\n**W2:** Figure 1 illustrates the example of multimodal electronic health records that possess irregularity and missing modality problems. This illustration reflects the situation of the MIMIC-IV dataset. Detailed task descriptions and data preprocessing workflows are discussed in Appendices A and D. We will add detailed explanations to Fig. 1 in the revised version.\n\n**W3:** Softmax gating is more commonly used than Gaussian gating in real-world problems involving vision, text, and potential multimodal applications, and it also achieves relatively better performance. This is why we mainly compare our method with Softmax gating. Gaussian gating has been compared with Laplace gating in a different work, which illustrates that our proposed Laplace gating is more sample-efficient than Gaussian gating as shown in Nguyen et al. [1].\n\nFirst, recall that the general form of the Gaussian gating is given by \n$$h(x)=\\mathrm{TopK(-(W-x)^{\\top}\\Gamma^{-1}(W-x)/2)},$$ \nwhere $W$ is a mean vector, and $\\Gamma$ is a covariance matrix.\nThen, Nguyen et. al. [1] consider two settings of the mean parameters $W^*_j$. Under the first setting (resp. second setting), Nguyen et. al. [1] argue that due to an interaction between the mean parameters $W^*_j$ and the covariance matrix $\\Gamma^*_j$ (resp. the expert parameters $a^*_j$) via the partial differential equation (PDE) in Eq.(8) (resp. Eq.(11)) in [1], the rates for estimating mean parameters $W^*_j$ (resp. $a^*_j$) decrease when the number of their fitted atoms increases, and are no faster than $O(n^{-1/8})$. On the other hand, it can be verified that such parameter interactions do not occur under the FuseMoE. Thus, the estimation rates for the mean parameters $W^*_j$ and the expert parameters $a^*_j$ when using the Laplace gating remain unchanged of order $O(n^{-1/4})$. To strengthen our arguments, we also include a table where we summarize the parameter estimation rates under the Gaussian MoE when using the Laplace gating and the Gaussian gating.\n\nFinally, we explain why the aforementioned PDEs affect the parameter estimation rates. In particular, a key step in our proof is to decompose the density difference $p_{G_n}(Y|X)-p_{G_*}(Y|X)$ into a combination of linearly independent terms using Taylor expansions. However, if those PDEs hold true, then components in that decomposition become linearly dependent, which is undesirable and leads to slow parameter estimation rates.\n\n*Summary of parameter estimation rates under the Gaussian MoE with the Gaussian gating and the Laplace gating. The function $\\widetilde{r}(\\cdot)$ is defined in Eq.(12) in [1], while the function $\\bar{r}(\\cdot)$ is defined in Eq.(9) in our paper. Note that $\\widetilde{r}(\\cdot)\\leq\\bar{r}(\\cdot)$ and $\\widetilde{r}(2)=4$, $\\widetilde{r}(3)=6$. Additionally, $\\mathcal{A}^n_j:=\\mathcal{A}_j(\\widehat{G}_n)$ is a Voronoi cell defined in Eq.(7) in our paper:*\n|`Gating`|$W_{j}^{*}$|$a_{j}^{*}$|$b^*_{j}$|$\\nu_{j}^{*}$|\n|:---:|:----:|:----:|:----:|:----:|\n|Gaussian [1]: setting I|$\\mathcal{O}(n^{-1/2\\bar{r}(\\|\\mathcal{A}^n_{j}\\|)})$|$\\mathcal{O}(n^{-1/4})$| $\\mathcal{O}(n^{-1/2\\bar{r}(\\|\\mathcal{A}^n_{j}\\|)})$|$\\mathcal{O}(n^{-1/\\bar{r}(\\|\\mathcal{A}^n_{j}\\|)})$|\n|Gaussian [1]: setting II|$\\mathcal{O}(n^{-1/2\\widetilde{r}(\\|\\mathcal{A}^n_{j}\\|)})$|$\\mathcal{O}(n^{-1/\\widetilde{r}(\\|\\mathcal{A}^n_{j}\\|)})$|$\\mathcal{O}(n^{-1/2\\widetilde{r}(\\|\\mathcal{A}^n_{j}\\|)})$| $\\mathcal{O}(n^{-1/\\widetilde{r}(\\|\\mathcal{A}^n_{j}\\|)})$|\n|Laplace (Ours)|$\\mathcal{O}(n^{-1/4})$|$\\mathcal{O}(n^{-1/4})$|$\\mathcal{O}(n^{-1/2\\bar{r}(\\|\\mathcal{A}^n_{j}\\|)})$|$\\mathcal{O}(n^{-1/\\bar{r}(\\|\\mathcal{A}^n_{j}\\|)})$|\n\n**Limitations:** We will revise our paper to avoid potential information overlaps with prior UTDE work. However, we wish to emphasize that the encoder part was not proposed by UTDE itself; the authors of that paper also utilized prior works (i.e., mTAND, Time2Vec) for demonstration purposes, similar to our approach.\n\nRegarding the gating function and encoders, we appreciate your insights. Indeed, all the ablation studies on encoders presented in Appendix H.2 are based on the Laplace gating function of our FuseMoE framework. Choosing appropriate encoders is beyond the scope of this paper, which is why we defer this discussion to the Appendix. However, as the reviewer points out, it is beneficial to determine whether gating functions or encoders contribute more significantly to overall performance improvement. Therefore, we have added additional experimental results on mutating different gating functions in conjunction with different encoders and compared the differences in results. The figures can be found in the attached PDF file.\n\n**Reference**\n\n[1] Nguyen et al., (2024). Towards convergence rates for parameter estimation in Gaussian-gated mixture of experts.\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Area Chairs and Reviewers,

We want to thank you for your valuable feedback and insightful reviews, which have greatly contributed to improving our paper. The following endorsements are truly motivating:

- Writing: Our paper is well-written and easy to follow, with informative tables and figures (Reviewer `P6x6`).
- Motivation: The motivation of our paper is promising and well-elaborated (Reviewers `P6x6`, `6SUH`, `sZSN`).
- Method: FuseMoE is novel and performs better than baselines (Reviewer `6Ay7`).
- Theory: Our paper is theoretically sound (Reviewers `6Ay7`, `6SUH`, `sZSN`).
- Experiments: The experiments in our paper are extensive and comprehensive (Reviewers `P6x6`, `6Ay7`, `6SUH`, `sZSN`).

To better address the reviewers’ concerns, we have added additional results in the attached PDF file. We summarize the results below:
- **Table 1**: Summary of information on all the datasets we tested in the experiment section, including important categories such as modalities and sample sizes. These results complement our answers to Reviewers `P6x6`, `6Ay7`, and `6SUH`.
- **Figure 1**: ImageNet classification accuracy results using Vision-MoE-Base and Vision-MoE-Large models. For Vision-MoE-Base, we used 8 experts, 512 hidden dimensions, and 12 layers; for Vision-MoE-Large, we used 16 experts, 768 hidden dimensions, and 16 layers. The models are trained for 300 epochs. These results complement our answers to Reviewers `P6x6`, `6Ay7`, `6SUH`, and `sZSN`.
- **Figure 2**: Visualization of modality weight on the top-$k$ chosen experts using CMU-MOSI and CMU-MOSEI datasets. For every expert selected, we calculate the number of samples that include a specific modality, weighted by corresponding weight factors from the gating functions. The outcomes are subsequently normalized across modalities. The results for the MIMIC dataset can be found in Figure 12(c) of Appendix H.3. These results complement our answers to Reviewers `P6x6`, `6Ay7`, and `sZSN`.
- **Tables 2 - 5**: Experimental results on mutating different gating functions in conjunction with different encoders of FuseMoE. All results are averaged over 5 random runs. These results complement our answers to Reviewer `sZSN`.

We hope that our responses below and additional results will satisfactorily address your questions and concerns. We sincerely appreciate the time and effort you have dedicated to reviewing our submission, along with your invaluable suggestions.

Sincerely,

Authors

Meta-review: This paper presents work on multi-modal data, particularly including time series data.  A mixture of experts framework is developed, with a Laplace gating function.  A variety of fusion/gating strategies is explored as examples of the general MoE / gating approach.

The reviewers appreciated the problem formulation and contribution of the Laplace gating function with its accompanying theoretical analysis.  Concerns were raised over the computation needs, scalability, and effectiveness of the routing/gating mechnanism versus the modality-specific encoders.

After considering the authors' response the reviewers evaluated their positions.  The additional notes on experiments, comparisons, and clarificiations resolved most of the reviewers' concerns.  

Overall, the paper presents interesting work on learning from multi-modal data with a solid contribution in the form of a novel gating function and promising results.  As such, it is recommended for acceptance in NeurIPS.
Decision: Accept (poster)