Building Massively Multimodal Foundation Models with Interaction-aware Mixture-of-Experts
Authors: Xing Han
Hsing-Huan Chung
Joydeep Ghosh
Paul Pu Liang
Suchi Saria
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Interaction
Mixture-of-Experts
Transformer
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We introduced a novel MoE architecture that integrates temporal multimodal interactions into model training.

Abstract: Modern applications increasingly involve many heterogeneous input streams, such as clinical sensors, wearable device data, imaging, and text, each with distinct measurement models, sampling rates, and noise characteristics. We define this as massively multimodal setting, where each sensor constitutes a separate modality. As modality counts grow, capturing their complex, time-varying interactions such as delayed physiological cascades between sensors, has becomes essential yet challenging. Mixture-of-Experts (MoE) architectures are naturally suited for this setting since their sparse routing mechanism enables efficient scaling across many modalities. However, existing MoE architectures route tokens based on similarity alone, overlooking the rich temporal dependencies across modalities: this prevents the model from capturing delayed cross-modal effects, leading to suboptimal expert specialization and reduced accuracy.
We propose a framework that explicitly quantifies temporal dependencies between modality pairs across multiple discrete time intervals, defined as delays between an event in one input stream and its manifested effect in another, and uses these to guide MoE routing. A interaction-aware router dispatches tokens to specialized experts based on interaction type. This principled routing enables experts to learn generalizable interaction-processing skills. Experiments across healthcare, activity recognition, and affective computing benchmarks demonstrate substantial performance gains and interpretable routing patterns aligned with domain knowledge.

Published as a conference paper at ICLR 2026
BUILDING MASSIVELY MULTIMODAL FOUNDATION MODELS WITH INTERACTION-AWARE MIXTURE-OF-EXPERTSS
Xing Han
Hsing-Huan Chung & JJydeep Ghosh
Johns Hopkins University Baltimore, MD 21218, USA xhan5 jhu.edu
University of Texas at Austin
Austin, TX 78712, USA
(hhchung, ighosh www.osheetteesssccoooooooo @utexas.edu
Paul Pu Liang"
Suchi Saria"
Massachusetts Institute of Technology Cambridge, MA 02139, USA pp illiann@miit.ee
Johns Hopkins University Baltimore, MD 21218, USA ssarial juujjueeee edu
ABSTRACT
Modern applications increasingly involve many heterogeneous input streams, such as clinical sensors, wearable device data, imaging, and text, each with distinct measurement models, sampling rates, and noise characteristics. We define this as massively multimodal setting, where each sensor constitutes a separate modal- ity. As modality counts grow, capturing their complex, time-varying interactions such as delayed physiological cascades between sensors, has becomes essential yet challenging. Mixture-of-Experr
(MoE) architectures naturally suited for this setting since their sparse routing mechanism enables efficient scaling across many modalities. However, existing MoE architectures route tokens based on similarity alone, overlooking the rich temporal dependencies across modalities: this prevents the model from capturing delayed cross-modal effects, leading suboptimal expert specialization and reduced accuracy. We propose a framework that explicitly quantifies temporal dependencies between modality pairs across multiple discrete time intervals, defined as delays between an event in one input stream and its manifested effect in another, and uses these to guide MoE rout- ing. A interaction-aware router dispatches tokens to specialized experts based on interaction type. This principled routing enables experts to learn generalizable interaction-processee skills. Experiments across healthcare, activity recognition, and affective computing benchmarks demonstrate substantial performance gains and interpretable routing patterss aligned with domain knowledge.
INTRODUCTION
Multimodal learning has traditionally focused on integrating two or three canonical modalities such text, image, and audio (Baltrušaitis et al., 2018: Liang al. 2024b). Yet real-world applica- tions increasingly involve massively multimodal data: dozens to hundreds of heterogeneous input streams, each with distinct measurement models, sampling rates. noise characteristics, and tem- poral dynamics (Liang et al., 2021; 2022] SSenksen et al. 2022). In healthcare alone, a single patient generates continuous signals from heart rate monitors, pulse oximeters, blood pressure cuffs, ECGs, respiratory sensors, and laboratory analyzers, alongside imaging and clinical notes (Johnson) ctt al. 2023). Each stream constitutes a distinct modality requiring different processing strategies (Soenksen et al. 2022). As modality counts grow, the space of cross-modal interactions explodes: some pairs provide redundant information, others capture unique signals, and still others exhibit synergistic patterns that emerge only through joint analysis (Williams & Beer 2010 Liang al. 2023). These dependencies often unfold over time with characteristic delays (Weissman et al, 2012 Varley 2023), a predictable time intervals between cause in modality and its observable effect in another, further complicating the modeling challenge. In response, MoE models (Shazeer et al.
"Equal Advising.

---

Published as a conference paper at ICLR 2026
Tasic RUS Auoliary Loss
Synergy Experts
Uniqueness Experts
Redundancy Experts
FFN
McE
Expert
Epprrt 2
Dipert (FFN)
Expert (FRN]
Expert i+] (FFN)
RUS
Temponi
Mof
MHSA
Encoder stack
layest
Sertrm
Expert N
RUS Volues
Transfo
MHSA
Routing Pathi ce
choice
rUS-Aware Router
Temporal RUS for (m,, ma), Gma, (maa
Modality
Encoders
Temporal Embeddings
Time Scries
Tobic
Unique
Redundanc
Synorgy
Figure 1: Overview of MERGE. The left panel illustrates the overall architecture, where multi- modal inputs are processed through N stacked encoder layers composed of alternating Transformer and MoE blocks. The core innovation of MERGE lies in the MoE layers, detailed the right. The temporal RUS-aware router is the essential part, which leverages temporal multimodal interactions to guide the routing of token embeddings across different time lags. The router determines, based on interaction dynamics, which modality pairs should (or should not) be routed to the same expert, thereby enabling more principled and interpretable expert specialization. As an example, 1/ (yel- low) and my (green) exhibit high redundancy according to their temporal RUS values; therefore, the router is more likely to assign them to the same expert (yellow and green arrows).
2017 Guo et al. 2025 Jiang et al. 2024) offer a natural foundation for this setting. It provides a principled way to allocate computation to modality experts: LIMoE (Mustafa et al. 2022) designed multimodal MoE that processes both images and text through shared sparse MoEs with contrastive learning, demonstrating modality specialization through entropy regularization. Fuse- MoE (Han et al. 2024) addressed the "FlexiModal" setting with irregularly sampled and missing modalities, introducing Laplace gating function with theoretical convergence guarantees supe- rior to softmax routing in multimodal applications. Building on this, Flex-MoE (Yun et al., (2024) proposed a missing modality bank and dual-router design handle arbitrary modality availability. Hierarchical MoE (Nguyen et al. 2024) demonstrated that Laplace gating at two hierarchical levels eliminates undesirable parameter interactions, accelerating expert convergence in multimodal tasks.
Despite the successful application of MoE in multimodal problems, most existing MoE architectures rely on routers that consider only the similarity between input tokens and experts; such routers only operate on static modalities (Riquelme et al. 2021 Fedus et al.) (002)), or static features from temporal modalities (Xie et al.] [025]] Zhu et al.] 2025p. However, in real-world applications, using multimodal interaction at single time point often fails to capture the full landscape of cross-modal relationships, as delayed effects between variables or modalities are common (Zhang et al. [025] Zhou et al. 2025). For example, when understanding human communication, a brief eyebrow raise followed 400-400 ms later by rising intonation and a half-smile turns the same words into sarcasm; in medical diagnosis, a slow overnight drift in SpO2 and respiratory rate, followed hours later by fever and elevated lactate, flags early sepsis. This raises an essential question: Can we incorporate temporal multimodal interaction to guide MoE training and inference? To capture such temporal interactions, a successful MoE model should be able to: (1) employ a well-defined quantitative measure of multimodal interaction that accounts for time-delayed interactions between modalities, and (2) incorporate an enhanced MoE architecture capable of leveraging this information during training. With these components, the model can become more proactive and context-aware, leading to improved performance by better understanding complex, time-evolving multimodal processes.
Motivated by this challenge, we propose Massively-multimodee Expert Routing for Generalized Exchange (MERGE), a novel framework that leverages temporal multimodal interactions to guide the routing process of MoE. The overall framework is illustrated in Figure [ We begin by intro- ducing method to explicitly compute temporal multimodal interactions (Redundancy, Uniqueness, Synergy, or RUS) between input modalities over time with respect to the target outcome. To handle nigh-dimensional and temporal data, we design scalable approach based on multi-scale BATCH estimator to compute multi-step temporal interactions efficiently. The resulting estimates are then used to guide the MoE training process. Specifically, design interaction-awar router that
2

---

Published as a conference paper at ICLR 2026
incorporates the context of temporal RUS sequences and dynamically routes tokens to experts based on these temporal interactions. This routing mechanism is further reinforced using auxiliary loss functions during training. The core of MERGE consists of two components: (I) computing tem- poral RUS (section[3.) and (2) using the resulting RUS sequences to guide MoE training (section 3.2 We demonstrate the effectiveness of MERGE in ways: first, by analyzing the insights pro- vided by temporal RUS and showing that they capture meaningful and application-relevnn patterns; and second, by illustrating that the learned routing patterns align with our expectations and lead to significant performance improvements across diverse set of multimodal tasks.
2 RELATED WORKS
Multimodal Interactions define the degrees of commonality between modalities and the ways they combine to provide new information for task (Liang et al.) 2024b). A core problem lies in un- derstanding the nature of how modalities interact and modeling these interactions using data-driven methods. The study of multimodal interactions has involved semantic definitions based on research in multimedia (Marsh & Domas White 2003), verbal and nonverbal communication (Partan & Mar- ler| 2005; Fllm & Bahrick 2007 et al., 2006), social interactions (Mai et al. [019 Jung al., 2018), and instruction tuning (Shan et al. (025). These have also inspired statistical methods to quantify multimodal interactions from unimodal predictions (Mazzetto al. 202T), , trained model
weights and activations (Sorokina et al., 2008 Tsang et al., 2018 2020; Hessel & Lee 2020), fea-
Auffarth al. 2010), and information theory (Williams & Beer 2010 Bertschinger et al. 2014). For information theory-based methods,
ture selection (Ittner al. [021] Yu Liu 2003 2004:
recent studies have investigated extensions to continuous (Pakman et al. 202T EErlich et al. 2024)
or Gaussian (Venkatesh et al., 2023) distributions, many modality decomposition (Varley) 2024),
sample-level quantifications (Yang et al., 2025), and enable accurate estimation from large-scale multimodal datasets (Liang et al., 2023 2024a). Recently, Varley (2023) introduced a framework for modeling multimodal interactions over time. While insightful, the approach has limitations:
it does not naturally extend to continuous variables, and its lattice-based based structure does not scale beyond small systems, limiting its practicality for large-scale multimodal applications.
PID and Multimodal Interaction MoEs. Partial Information Decomposition (PID) (Williams & Beer 2010 Bertschinger et al. 2014) provides a principled framework for quantifying multimodal interactions by decomposing the total information into redundant (R),, unique (U), and synergis- tic (S) components. Redundancy captures the shared information across modalities, uniqueness measures modality-specific contributions, and synergy reflects information that emerges only when modalities are combined. To the best of our knowledge. PID has not yet been exploited to improve multimodal MoEs. In contrast, existing multimodal MoE approaches typically rely on heuristic ex- pert assignments (Shazeer et al.) 2017: He et al. 2021 Han et al. 2022 Akbarian et al. (024), which often result in opaque and seffheult-to-interree routing decisions (Liu et al. [024]] (Shen et al. 2024). Multimodal interaction provides a natural and principled way to enhance expert specialization. For example, Yu et al. (2023) introduced the Mixture of Multimodal Interaction Experts, where specific experts are assigned to process predetermined types of modality interactions. Recently, Xin et al. (2025) extended this idea by incorporating interaction-type categorization into the MoË training process and expanding the framework to support more than two modalities. However, these designs make the number of experts tightly coupled to the number of modalities, which limits scalability. A more natural and flexible solution is the model to dynamically determine which modalities should (or should not) be processed together. Moreover, both works rely on a binary label agreement method based on unimodal classifiers to approximate the PID-type multimodal interactions, making the approach heavily dependent on the performance of the individual classifiers. Additionally, this method captures only static interactions, overlooking the temporal and continuous nature of real-world multimodal data.
multimodal interactions to guide expert routing, enabling
3 RUS-GUIDED MoE FOR MASSIVELY MULTIMODAL LEARNING
We introduce MERGE, framework for guiding MoE with temporal multimodal interactions in massively multimodal settings. The approach consists of two parts: (1) capturing temporal multi- modal interactions, and (2) leveraging these quantified interactions inform the training of MoE.
3

---

Published as a conference paper at ICLR 2026
3.1 CAPTURING TEMPORAL MULTIMODAL INTERACTIONS
We first present our methodology for capturing temporal multimodal interactions. We detail the procedure for computing temporal RUS across different settings, describe how to efficiently estimate these interactions, and provide potential insights that can be derived from the temporal RUS patterns.
Formulation of temporal RUS. Capturing information interactions over time remains an important yet underexplored problem. Our goal is to characterize how past and present values interact across time for task. For example, in the medical domain, one may wish to understand how the interac- tions of past treatments influence patient's s health outcomes; in activity recognition, it is crucial to capture how past motions of individual body parts (e.g., arms and legs) contribute to predicting fu- ture overall motion. However, the existing PID framework (Eq. (15-77 in Appendix (A) is computed using standard mutual information, which only captures static interactions and cannot be directly extended to the temporal setting. To this end, build directed information (Weissman et al. 2012), which enables scalable modeling of temporal interactions while maintaining alignment with PID. Directed information respects temporal relationships by considering information flow from past to present, which enables PID analysis across multiple time lags, providing insights into both short-term and long-term influences. We begin by defining multi-source directed information, which quantifies the flow of information from multiple input modalities to the target variable over time.
Definition (Multi-Soure Directed Information). Given multiple source processes X (X1,1, x..,,..., -1,,-1), X = (X2,1, 1221,Xi
X2,(-1) and the target process yi-1 = (Y,, V2 Yi-1), the directed information from X2) to Y over n steps with time lag is:
DI(T) ( [(X]
,X-- + r "X2
Σ
As shown in Figure 2 at each time lag т, DI(T) can be decomposed into four components based
on the conditional joint distribution (7 22, y) = P(X = I1, X -T Zz2 Y = y y/Y"-!), where DI(r)= = R(() + (T) + U2(T) S(T): each of its component quantifies interaction at R(T) min TQ,
+ Tq, x²= -lq, (Y"; x)
Q+EA,
(r) Tq; (Y"; x1--) - R(T). U2(T) lq: : x2 - R(T).
(3)
S(T) Ip, (Y X XX - lo: (Y": X -,,,,
(4)
where A represents the probability simplex and A, is the set of marginal-matching distributions defined as
1 1 (x!-1)
S(r)
A Q- v) = (ri y), Vy Y, E X,, E {1, } it characterizes the set of joint distributions Q, (z), x2, y) that preserve the time- specific bivariate marginals (x), and P, Pr(x2, v) while allowing the coupling between X1 and X₂ to EA, TQ- the optimal distribution that has eliminated the syner- gistic information between X, and X2. After further simplifying the formulation, we obtain
A, = (Q+
==1 r= 2=/
(+(+)
U2(+)
R(r)
vary; O = arg min
is
1(y": X
Figure 2: Decomposed directed informa-
tion components across time lag T.
I(T)) = max Q-EA,
Io, (XT" :
(5)
Ui( = min
To, X U2(T) = min lo, Q. EA,
(6)
C,Ea,
S(T) = Ip.
"X"T;"")) min lo (X) X Y Q-EA.
Note that, X1 and X2 do not necessarily have the same time lag T, and exhaustively enumerating all pairwise combinations of across modalities would further increase memory costs during model training. To ensure efficiency, we design X, and X₂ to be aligned at the same time lag, while noting that the framework naturally extends to cross-lagged interactions across different time steps.
Efficient computation of temporal RUS in high dimensions. Computation of PID was limited to discrete and small support (Bertschinger al. 2014 Griffith & Koch 2014), or continuous but
4

---

Published as a conference paper at ICLR 2026
R(x)
(x(3) """""
U2(r)
S(r)
wng mnn.o..e. 1q. x
SGD
Qx (r)
(x,
x
X
X
Alignment Tensor
Optimize QN Or 4
D1.:
,(
x2-3
X2-2
X
Lag =
Lag Logrz?
Lag
Lag Embedding e(r)
(()
Di2.: X(
Shared Encoders
Fusion
910 (2)
Figure 3: Schematic overview of the multi-scale BATCH estimator. The procedure consists of four stages: (1) encoding empirical datasets D, = { X y²)} with shared encoders g. with each lag T further embedded by an encoder e to produce e(T), (2) training lag-conditioned discriminators D1.0. D2,8, D12,9 to estimate P, together with MLPs to generate embeddings for each т, (3) updating the alignment tensor to enforce marginal distribution matching between Q and P. yielding the optimized distribution Qr, and (4) decomposing the resulting estimates of Q and P into redundancy, uniqueness, and synergy sequence across all time lags.
low-dimensional variables (Pakman al. 2021). The BATCH estimator (Liang et al. 2023) offers an effective solution for handling high-dimg dinensninnsii distributions: it parameterizes the distribution components of interest using neural networks and approximates the true distribution based on sub- sampled batches. The direct way to obtain temporal RUS values is to compute Eq. t [n-7,1) nl., where all the distributional components P, and Q, are estimated using the BATCH estimator. However, this introduce significant computational overhead, the optimization quired to obtain Q; may need be repeated at every step. To leverage the multitask nature of neural network backbones,
7 7 for each
enhance the BATCH estimator by formalizing multi-scale approach that trains a single model to predict the temporal RUS at multiple time lags, as shown in Figure 3
For each lag т, construct the empirical dataset D, {X"" (1x""T,0" XX, ,Y")], with chosen such that n > 1. We first estimate distributions P. (Y PY(Y)))) (Y X2) by training lag-conditioned discriminators Di, D2, and D12 for all т. For example, D12 is defined as
P(Y|Xi; X2, r) D12 (
(8)
where 9-2,0()) is the encoder, is the fusion operator, and e(() is learnable lag embeddings that encode temporal relationships. We then construct the optimal distribution Q; that satisfies O = arg ming,ea, To. (X "x"") This achieved by constructing the alignment ten- sor align_ E NNXNK that measures the compatibility between samples i and j from modalities X, and X2, within class k, at lag T. Here, N represents the batch size and C' the number of classes.
(1,k.7) ((ik.T))
(i. exp
4x
Gx2
(G.k.r) NN. (olg (x) e(T))) where 4x.
Vd
The alignment tensors are subsequently normalized via the Sinkhorn -noopp algorithm (Knight) 2008) to enforce marginal-matching constraints, yielding the optimal distribution Q'. We perform this step for all simultaneously by leveraging the efficient parallelism of tensor operations. Using Eq. we can then compute temporal RUS values directly on high-dimensional embeddings.
3.2 MERGE: BUILDING RUS-AWARE MOE ROUTERS
We now discuss how to link routing decisions with information-theoree principles to make the MoE interaction- We then present the RUS-award router, which incorporates the context of RUS sequences into MoE. Finally, we describe the auxiliary loss that enforces these routing strategies during training.
We begin by outlining the routing strategies for different interaction types.
5

---

Published as a conference paper at ICLR 2026
RUS Values
Routing Strategies
Expert Types
Equivalent Fusion Types
High Rmi.ma
Route tokens together
Regular Expert (shared)
Early Fusion
High Um
Diversify routing
Regular Expert
Late Fusion
High Smy,m
Route tokens together
Cross-Modal Expert
Hybrid Fusion
Table I: Summary of our proposed routing strategies that leverage temporal RUS insights. Such mechanism can also be viewed as a mixture of different fusion techniques (Baltrušaitis al. 2018
Routing strategies. Our model is based on the principle that different types of interactions call for different computational strategies: (1) When multiple modalities contain similar information (R), they can be routed to the same expert. This is supported by both theoretical insights from Kol- mogorov complexity (Rissanen, 1978) and empirical findings in multimodal learning (Baltrušaitis et al. 2018 Ngiam et al. 20111. (2) When modalities contain distinct information (U), they should be routed to different experts to fully capture their uniqueness. Such routing diversification has also been highlighted in recent studies (Mustafa et al. 2022 Han al. 2024) means of ensuring balanced expert utilization. (3) Since high synergy (S) benefits from explicit interaction modeling (Liang et al. 2023 Liu et al., 2018), design dedicated synergy experts to capture cross-modal interactions. Each synergy expert consists of a cross- cooss-attention module followed by a cord-forward layer. In Table we summarize the routing strategy and corresponding fusion techniques.
RUS-aware router is the central component that incorporates temporal RUS to guide MoE training. As illustrated in Figure 4 the atten- tion mechanism focuses pairwise redundancy and synergy between modalities, which weights the importance of different pairwise interactions based on the current token's is characteristics. A GRU-based module captures the temporal dy- namics of uniqueness, it further combines with attention-weeihhted interaction contexts. The fi- nal routing decision fuses token-specific repre- sentations with RUS-derived features. Formally, for each modality mi {1,2, ... M) and all other modalities m2 mi, let L denote concatena- tion. The RUS-aware router is formalized as
Temporal RUS
Unn,
Rie,m
Key
Routing Distribution 0000 lna0
Sm,
Valuue
Cross- Attention
FFN
mi Token Embeddings
Query
Figure 4: Router structure: routing decisions weighs RUS-derived features and token embeddings.
RUSContextm; Attention (Querym, {[Rmi, .mg. Sm mm m2 Bm / @ GRU(U )
(10)
Logits MLP (TokenFeaturesm. + RUSContextm..)
(11)
This design ensures that routing decisions for modality mi are informed by its temporal interactions with all other modalities m2 (captured by temporal RUS), not just its individual content.
Auxiliary losses for multimodal interaction experts. The training objective is formulated to en- sure that the router adheres the predefined principles. It combines the task-specific loss with RUS-aware auxiliary loss, where the latter decomposes into components aligned with each interac- tion type. For redundancy, at each time step quantify the discrepancy between the routing distributions Prouter of mi and m2. As shown in Eq. 12 minimizing Credundancy encourages m and m2 be routed to the same expert whenever their redundancy exceeds the threshold TR-
employ ShhnnnnSSannon Divergence (JSD) to
JSD(P(t.mm))
pp(tmg))
Leddnndancy AR Nredundont
router
(12)
(ms mms::m..):Bmm, 2
Similarly, for uniqueness, as shown in Eq. 13 whenever their uniqueness exceeds the threshold TV (uniqueness is modality-specifie, not pairwise).
we encourage diversified routing of mi and m2
Luniqueness
-AU
JSD JSD(p(t.m)) p(t.mz))
(t.m.)
(13)
N Vunique (m² ma, m): .t S Swg.S
We promote routing to synergy experts whenever synergistic information is high: p(t.ms))
+ Psjn p(t.ma)
Lugnergy = As
Σ
1
Pyn
(14)
N Naynergy ()). ....m))m........ 1278
2

---

Published as a conference paper at ICLR 2026
Table 2: MERGE demonstrates superior results across different benchmarks and datasets. The best results are highlighted in bold font, and the second-best results are underlined.
Datasets
PAMAP2
MIMIC-IV IHM
MIMIC-IV LOS
MOSI
WESAD
Opportunity
Metries
Accuracy
AUROC
F1
AUROC
F1
Accuracy
AUROC
Accuracy
AUROC
Accuracy
Transformer
82.48
82.57
80.18 2 78.96
75,46 :44 72.31 u.5 68.39
68.47
53.84 121 74.39 1.5
59
mTAND
74.62 2uss 74.38
0/5 80.89 = 79.35 040
73.45
2.44 70.07 844 29.94
48.22 n.s 71.66
70.26 = 200
MulT
82.23 81.87
0:43 81.63 a 647 81.55 033
76.68 109 72.52 am 68.80 69.05
47.63
71.43 2:33
72.61
MISTS
85.79
0.31 82.21 = 6.5 80.56 0.22
77.49 063 73.86 0.3 69:42: 0.2
69.32 0:51
=
73.29 mm
79.36 141
FuseMoE
87.74
840 86.51
82.33
645
81.74
75.18 9.41 75:65 78.43
$3.92
76.31
2:0
83.15
12MolF
84.55
84.24
om 82.59
1:08
74.36
n.8 71.91
74.88
1:33
52.64
1:06
75.52
1:54
82.166
MERGE
91.37 90.44
85.40 84.97
0.35
81.88 n.s 0. 72.04
77.25
55.74 189 77.34
84.32 1.30
The overall objective combines the task-specific loss and the interaction-aware losses defined in Eq. Importantly, the train-test split used for temporal RUS computation is aligned with that of MERGE's 's training, ensuring that the estimated interactions directly facilitate the learning of the cor- responding tokens. Note that, the RUS estimation performed separately from the MERGE train- ing phase. While it is possible to build an end-to-end framework that jointly learns temporal RUS values and MoE model, we intentionally decouple these two components for the following reasons: (1) the multi-scale batch estimator is designed to reflect intrinsic information-theorett structure of the dataset. It should not be optimized to minimize task specifie losses, doing so would entangle the RUS values with downstream objectives, undermining their role as task- agnostic signals of tempo- ral multimodal interactions. (2) Since temporal RUS is a property of the dataset, it only needs to be computed once. The resulting values can then be cached and reused across any number of down- stream tasks for this dataset. We expect MERGE to leverage these implicit interaction dynamics, thereby enhancing both the performance and the interpretability of downstream tasks.
12
14
EXPERIMENTS
Overview. We conduct comprehensive empirical evaluations of MERGE, aiming to answer the following questions: (1) What insights can temporal RUS provide in different scenarios? (2) Can leveraging temporal RUS improve MoE training accuracy? (3) How do temporal RUS influence task outcomes? (4) How can we interpret the learned routing assignments? Our evaluation spans diverse domains, including healthcare, activity recognition, and affective computing. In these evaluations, we incorporate a wide range of heterogeneous input streams, including sensors, wearables, medical imaging, text, and FCG, as distinct modalities to demonstrate MERGE's ability to capture complex temporal interactions among them and its effectiveness in massively multimodal settings.
4.1 EXPERIMENT SETUP
Dataset information. РАMАР2 (Reiss & Stricker 2012) is a physical activity monitoring bench- mark that includes recordings of 18 activities, carried out by 9 participants equipped with inertial measurement units and a heart rate monitor. The MIMIC-IV (Johnson et al., 2023) ecosystem con- tains deidentified clinical data from ICU patients, including lab measurements, vitals, notes, and chest X-rays. We consider two tasks: in-hospital mortality (IHM) and length-of-stay (LOS) predic- tion. CMU-MOSI (Zadeh al., 2016) is a multimodal dataset of 2,199 annotated YouTube vlog opinion segments with audio, video, and text, used here for binary sentiment classification. WESAD (Schmidt et al. 2018) wearable sensor dataset from 15 subjects in a lab study, with physiolog- ical and motion data from chest- and wrist-worn devices. It supports affect recognition across six states, including stress, amusement, and meditation. Opportunity (Chavarriaga et al. (013) pro- vides muuti-modal sensor recordings from wearable, object, and ambient devices, capturing natural and scripted daily activities of 4 subjects in a realistic home-like environment.
Baselines. To validate the effectiveness of MERGE in enhancing multimodal learning, we bench- mark it against a diverse set of state-of-the-aa of-the-art models. The comparison covers: (1) standard mono- lithic backbones, including multi- self-ateentoon based Transformers (Vaswani et al. (017)) and multi-time attention for irregular time series (mTAND) (Shukla & Marlin (022]: (2) sssc-fpec
fusion models, such as Mul'I (sai et al. 2019) and MISTS (Zhang al. 2023); and (3) altema-
7

---

Published as a conference paper at ICLR 2026
Insulin Furaemiice
Furoseride Slood Uroa hitragen
Chosi, Hand) Duily Activity
[ECS, Respiration Chest Temparature
wt_.pp.......... 18.92.2500 42.12
price monton (ste))
Mommatoon (bbal
Micholi
Lsmaga A4l
U. Olouue Rt
byyee upt
00-22 Time Lag (hours)
Time Lag thours
Time Lag (seconas)
Time Eng Iseconds]
(a) MIMIC-IV
(b) MIMIC-IV
(c) PAMAP2
(d) WESAD
Figure 5: Insights from temporal RUS values across different applications include: (a) insulin and furosemide exhibit strong synergistic effect at the time of administration, while insulin's 's unique effect becomes more pronounced later; (b) as time progresses after furosemide administration, its physiological impact increases; (c) in activity recognition, chest and hand motion display coupled movements during locomotion, reflecting strong redundancy; and (d) in physiological monitoring, ECG and respiration signals from one second prior provide better predictions of current chest tem- perature, capturing the natural response delay to stimuli.
tive multimodal MoE approaches, including FuseMoE (Han et al. 2024) that does not incorporate multimodal interaction, and 12MoE (Xin et al. 2025) that uses interaction -peccfffi--sesiccccc experts.
Implementation. We split PAMAP2. WESAD, and Opportunity into training, validation, and test sets by subject. For MIMIC-IV, we use the first 48 hours of each ICU stay as input and randomly assign the stays to training, validation, or test splits. For MOSI, we adopt the official train/val/test splits provided in Multibench (Liang et al. 2021). Unlike the other datasets, MIMIC-IV and MOSI contain many more sequences, and the target for each sequence is single statie label rather than label sequence. When computing RUS values for these datasets, we focus on single time step n, taking Y" as the label for each input sequence while still considering multiple lags T from the end of the sequence. Because the RUS computation aggregates across multiple sequences, the resulting variety of labels ensures that the estimation of P(Y") does not produce a degenerate distribution.
4.2 PERFORMANCE AND INSIGHTS OF MERGE
MERGE demonstrates superior performance. In Table 2 we compare the performance of MERGE against various baseline models across 6 multimodal benchmark tasks. Results (aver- aged over 5 random runs) demonstrate that MERGE consistently achieves superior performance, attaining the best results in nearly all metrics. In particular, for healthcare (MIMIC-IV) and af- fective computing tasks (MOSI and WESAD), MERGE outperforms task-specific fusion methods such as MulT and MISTS by a significant margin. Against multimodal MoE baselines including FuseMoE and 12MoE, MERGE also achieves improved results. These findings highlight that lever- aging interaction dynamics across modalities can effectively enhance model performance, and that MERGE generalizes well across diverse multimodal learning tasks.
Qualitative examples of temporal RUS. To understand the reason behind the improved perfor- mance, we examine the temporal RUS values obtained across different tasks to assess whether they provide rich and interpretable insights. For example, the changing relationship between insulin and furosemide over time in MIMIC-IV, shown in Fig 5 (a). illustrates the interplay between their im- mediate and delayed physiological effects. At the time of simultaneous administration, a synergistic effect emerges, likely driven by their concurrent but opposing influences on blood glucose and potas- sium levels. However, when insulin administered 1 hour earlier or more, its uniqueness becomes much stronger, reflecting its relatively rapid onset of action and peak effect occurring within 2-3 hours. As shown in Fig (b), the time of Furosemide administration, its immediate diuretic ef- fect is unique to the drug and is not yet fully captured in the Blood Urea Nitrogen (BUN) levels. Over time, however, the synergy between Furosemide and BUN increases, reflecting the delayed manifestation of the drug's 's physiological effects in blood chemistry.
For activity recognition on PAMAP2, during activities such as walking, running, or climbing stairs, the natural swinging of the arms is directly coupled with the torso and chest movements. This lationship remains consistent over time, leading to a high redundancy pattern, as shown in Fig. 5 (c). Lastly, examining the dynamics between ECG, respiration, and chest temperature in Fig.
8

---

Published as a conference paper at ICLR 2026
Accuracy vs RUS Length
Effect of Multi-scale BATCH Estimator
Effect of Multi scale BATCH Estimator WESAU Osporunity
90
Difference
Hil
(16
80-
Performance
PAMP2 Opportunity WESAD
Ra ve Miliisale
RUS Length (log scale)
PAMAP2 Oppartunity
"Accueryy
11 Accuracy AUROE Necuracy
(b)
(a)
Figure 6: (a) Increasing temporal RUS length incorporated into MoE training improves performance; (b) discrepancy between the multi-scale BATCH estimator versus step-wise RUS computation; (c) performance differences across tasks after applying the multi-scale BATCH estimator.
Accuracy
Accurssy
Accuracy
Orgmm 3 Tarbiteli Earaide
Orgew Fediide & Eartice U S
Orggnal Furlide Fackide Dotuue
Orgha Eartude R Ear Earude
(a) PAMAP2
(b) MIMIC-IV IHM
(c) MIMIC-IV LOS
(d) Opportunity
Figure 7: Performance change after removing each auxiliary loss term from 1q to show that all interactions contribute effectively to enhancing performance across nearly all settings.
The results
reveals an initial increase in synergy and respiration uniqueness WESAD dataset. This arises because the one-second delay aligns with the time takes for the body's stress response to begin manifesting change in skin temperature. After this brief adjustment period, the relationships sta- bilize, reflecting the steady state of the physiological response. These examples clearly demonstrate that temporal RUS capture unique interaction dynamics for improved model performance.
4.3 IN-DEPTH ANALYSIS OF MERGE
Impact of temporal RUS length. We investigate how the length of temporal RUS incorporated into MoE training affects performance. Our intuition is that longer temporal range captures richer interaction information, thereby improving results. We first increase the maximum time lag of the temporal RUS from to 10. We do not extend it further, as estimating too many steps simultane- ously degrade the performance of the multi-scale BATCH estimator, To further extend sequence length, we repeat temporal RUS segments to form longer sequences. As shown in Fig. 6 (a), both strategies lead to performance gains. Inereasing the maximum time lag provides broader view of the temporal trajectory. Mcanwhile, segment repetition appears to strengthen the model's 's ability to recognize important modality interactions, improve context modeling, and reduce gradient variance. A similar observation that longer sequences with consistent patterns improve generalization has also been discussed prior works (Mustafa et al., 2022 Lai et al. 2018).
Efficiency of multi-scale BATCH estimator. The proposed multi-scale BATCH estimator achieves r-fold speedup temporal RUS estimation while maintaining parameter efficiency. We compare the estimated RUS trajectories against the naive step-wise approach. As shown in Figló (b), report the percentage difference across tasks, averaged over all time lags. Although the magnitudes of individual interactions differ slightly, the discrepancies remain relatively small. Moreover, these differences do not lead to notable performance fluctuations, as demonstrated in (c)
Impact of each auxiliary loss. We further investigate the contribution of each auxiliary loss term to performance across tasks. To this end, we perform an ablation study by removing each auxiliary loss term from Eq 12 14, thereby disabling the corresponding routing mechanism. As shown in Fig

---

Published as a conference paper at ICLR 2026
100
Expert Activation Ratios NIMIC-IM
Expert Activation Rables MIMIC IHM
100-
Expert Activation Ratlos RAM6P2
Expert Activation Ratios PAMAPP
Modalla
Activatton Rate 151
20
CXR
Activation Batn PM]
8
GBB
Activation Hako PM
30
Creet
Activaatonn 1a00 P61
g
Vist-
Visa
Hland
Iaan
Mot
8
Notes
50
Ankic
Ireart Rate
B
400
Kniki
Heart Bace
8
8
15.
250
234
142
Exp drx
Expert hóox
Expert ind
Expert Expert Indox
(a) MERGE
(b) Standard MoE
(c) MERGE
(d) Standard MoE
Figure 8: Comparison of routing distributions between MERGE and standard MoE. The activation ratio is defined by the percentage of tokens from a specific modality that are routed to each expert. In (a) and (c), expert indices highlighted in red denote synergy experts. The plots show that MERGE adheres to the proposed routing principles and provides enhanced interpretability.
uniqueness emerges critical factor for many tasks, while MIMIC-IV IHM and Opportunity primarily synergy-driven, and PAMAP2 shows greater dependence on redundancy.
Expert routing analysis. Finally, explicitly examine how MERGE influences multimodal token routing distributions. We compare MERGE with a standard MoE using top-k routing, as shown in 8 The distributions are generated from models that achieve the best performance for each task. In (a), RUS values show there is strong synergy between CXR and clinical notes, while vitals remain relatively independent. The router captures this by directing corresponding tokens to specialized ex- perts, enabling them to be trained more effectively together. Likewise, in (c), chest and hand signals (with strong redundancy) are routed the same expert, whereas modality pairs with high synergy or uniqueness are also handled according to their interaction type. In contrast, the standard MoE exhibits no such structured routing: experts process arbitrary combinations of modalities, resulting in unorganized assignments that hinder both efficient learning and interpretability.
5 CONCLUSION AND FUTURE WORK
We introduced MERGE, novel MoE architecture that integrates temporal multimodal interactions into model training. By decomposing multi-source directed information into temporal redundancy, uniqueness, and synergy, our approach captures rich evolving dynamics that effectively guide MoE routing in massively multimodal applications. We further developed scalable estimators suitable for multi-time-lag settings. MERGE achieves substantial performance gains while preserving expert- level interpretabilityy making it a compelling design choice for future multimodal foundation mod- els. Looking ahead, promising directions include extending the framework to capture more general temporltemporal dynamics and investigating its applicability to multimodal and multitask learning scenarios. We would also like to extend the design philosophy of leveraging temporal multimodal interaction to guide expert routing to LLM-MoE frameworks. For instance, incorporating multi- modal inputs with known temporal RUS values could potentially improve fine-tuning of MoEebased LLMs by helping the model identify the appropriate experts. Furthermore, we believe the current MERGE design shows even greater potential when applied to large-scale VI.Ms or world-models, where temporal multimodal interaction dynamics remain highly informative and can meaningfully improve real-world performance.
REFERENCES
Pedram Akbarian, Huy Nguyen, Xing Han, and Nhat Ho. Quadratic gating functions in mixture of experts: A statistical insight. arXiv preprint arXiv:2410.11222, 6, 2024.
Benjamin Auffarth, Maite López, and Jesús Cerquides. Comparison of redundancy and relevance measures for fcature selection in tissue classification of ct images. In Industrial conference on data mining, pp. 248-262. Springer, 2010.
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philipee Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2): 423-443, 2018
10

---

Published as a conference paper at ICLR 2026
Nils Bertschinger, Johannes Rauh, Eckehard Olbrich, Jürgen Jost, and Nihat Ay. Quantifying unique information. Entropy, 2014.
Serhat S Bucak, Rong Jin, and Anil K Jain. Multiple kernel learning for visual object recognition: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7):1354 1369, 2013.
Ricardo Chavarriaga, Hesam Sagha, Alberto Calatroni, Sundara Tejaswi Digumarti, Gerhard Tröster, José del R Millán, and Daniel Roggen. The opportunity challenge: A benchmark database for on-body sensor- sensor-based activity recognition. Pattem Recognition Letters, (4(15):2033-204 2013.
JunKai Chen, Zenghai Chen, Zheru Chi, and Hong Fu. Emotion recognition in the wild with feature fusion and multiple kemnel learning. In Proceedings of the 16th International Conference on Multimodal Interaction, pp. 508-513, 2014.
Iman Deznabi, Mohit lyyer, and Madalina Fiterau. Predicting in-hospital mortality by combining clinical notes with time-series data. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4026-4031, 2021.
David Ehrlich, Kyle Schick-Poland Abdullah Makkeh, Felix Lanfermann, Patricia Wollstadt, and Michael Wibral. Partial information decomposition for continuous variables based on shared exclusions: Analytical formulation and estimation. Physical Review E, 110(1):014115, 2024.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
Ross Flom and Lorraine E Bahrick. The development of infant discrimination of affect in multi- modal and unimodal stimulation: The role of intersensory redundancy. Developmental psychol- ogy, 43)1)2238, 2007.
Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Weizadeh2017tenne Xu. Are you talking to machine? Dataset and methods for multilingual image question. In Advances in Neural Information Processing Systems, volume 28, 2015.
Ashutosh Garg, Vladimir Pavlovic, and James Rehg. Boosted learning in dynamic Bayesian networks for multimodal speaker detection. Proceedings of the IEEE, 1355):13551 2003.
Virgil Griffith and Christof Koch. Quantifying synergistic mutual information. In Guided self- organization: inception, pp. 159-190. Springer, 2014.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Xing Han, Jing Hu, and Joydeep Ghosh. Dynamic combination of heterogeneous models for hierar- chical time series. In 2022 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 1207-1216. IEEE, 2022.
Xing Han, Huy Nguyen, Carl Harris, Nhat Ho, and Suchi Saria. Fusemoe: Mixture-of-experts transformers for fleximodal fusion. Advances in Neural Information Processing Systems, 37: 67850-67900, 2024.
Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, and Jie Tang. Fastmoe: A fast mixture-of-expert training system. ArXiv preprint, abs/2103.13262, 2021. URL https://
arxiv.org/abas wbwp://g/abb/////// 2tw..././/21.........
Jack Hessel and Lillian Lee. Does my multimodal model learn cross-modal interactions? it's
harder to tell than you might think! In Proceedings of the 2020 Conference on Empiri- cal Methods in Natural Language Processing (EMNLP), pp. 861-877, Online, 2020. Associ- ation for Computational Linguistics. doi: 10.18653/w1/2020.comm main.62. URL https: a/aclannthooogy..rrtr
emn 1-mpamnnn..m 62

---

Published as a conference paper at ICLR 2026
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-llcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 590-597, 2019.
Jan Ittner, Lukasz Bolikowski, Konstantin Hemker, and Ricardo Kennedy. Feature synergy, redun- daney, and independence in global model explanations using shap vector decomposition. ArXiv preprint, abs/2107.12436 2021. URL https:::p///o. http://///avvvvvooao org/ abs 2107 12436
Albert Q Jjang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088. 2024.
Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew Lun- gren, Chih-ying Deng, Roger G Mark, and Steven Horng. Mimic-cxr, de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019a.
Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger Mark, Seth Berkowitz, and Steven Horng. Mimic-cxr-jpg. large publicly available database of labeled chest radiographs. arXiv preprint arXiv: 2019b.
Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. Mimic-iv, a freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
Tzyy-Ping Jung, Terrence J Sejnowski, et al. Multi-modal approach for affective computing. In the ieee engineering in medicine and biology society
2018 40th annual international conference the (embe), pp. 1994. IEEE, 2018
Swaraj Khadanga, Karan Aggarwal, Shafiq Joty, and Jaideep Srivastava. Using clinical notes with time series data for ICU management. arXiv preprint arXiv:1909.09702 2019.
Philip A Knight. The sinkhorn-knopp algorithm: convergence and applications. SIAM Journal on Matrix Analysis and Applications, 30(1):261-275. 2008.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 4/st international ACM SIGIR conference on research & development in information retrieval, pp. 95-104, 2018.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jae- woo Kang. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234 1240, 2020.
Paul Pu Liang. Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Yufan Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal repre- sentation learning. In Phirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Jeffrey Tsaw, Yudong Liu, Shentong Mo, Dani Yogatama, Louis Philipp Morency, and Ruslan Salakhutdinov. High-modality multimodal transformer: Quantifying modality & interaction heterogeneity for high-modality representation learning. ArXiv preprint, abs/2203.01s
2022. URL https: //aaxivv..///////////eaaa//aaaea .rg. 2203.22332. 01311
Paul Pu Liang. Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, al. Quantifying & modeling multimodal interactions: An information decomposition framework. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
12

---

Published as a conference paper at ICLR 2026
Paul Pu Liang, Chun Kai Ling, Yun Cheng, Alexander Obolenskiy. Yudong Liu, Rohan Pandey, Alex Wilf, Louis- Phiilipp Morency, and Russ Salakhutdinov. Multimodal learning without la- beled multimodal data: Guarantees and applications. In The Twelfth International Conference Learning Representations, 2024a.
Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations trends in multimodal machine learning: Principles, challenges, and open questions. ACM Computing Surveys, 56(10): 1-42, 2024b.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, and Chong Ruan. Deepseek-v3 technical report. arXiv preprint 1IX1v:2412.19437, 2024.
Zhun Liu, Ying Shen, Varun Bharadhwajj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, and Louis-Philippe Morency. Efficient low-rank multimodal fusion with modality- specific factors. In Proceedings of the 56th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pp. 2247-2256, Melbourne, Australia, 2018. URL https://
Association for Computational Linguistics. doi: 10.18653/v 1/P18- 1209.
Sijie Mai, Haifeng Hu, and Songlong Xing. Divide, conquer and combine: Hierarchical feature fu- sion network with local and global perspectives for multimodal affective computing. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 481-4922 Florence, Italy, 2019. Association for Computational Linguistics. doi: 10. 10.18653/w1/P19-11 10446 URL /tttts:://cclaaatt www.anthhooo.o.oooooo wwg.y.rrg/199111111
Emily E Marsh and Marilyn Domas White. A taxonomy of relationships between images and text. Journal of documentation, 2003.
Alessio Mazzetto, Dylan Sam, Andrew Park, Eli Upfal, and Stephen H. Bach. Semi-supervised aggregation of dependent weak supervision sources with performance guarantees. In Arindam Banerjee and Kenji Fukumizu (eds.), The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event, volume 130 of Proceedings of Machine Learning Research, pp. 3196-320 3204. PMLR, 2021. URL httt::/ html
mlr.
Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Mul- timodal contrastive learning with limoe: the language-image mixture of experts. Advances in Neural Information Processing Systems, 35:9564 9576, 2022.
Ara V Nefian, Luhong Liang, Xiaobo Pi, Liu Xiaoxiang, Crusoe Mao, and Kevin Murphy. A coupled HMM for audio-visual speech recognition. In 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 2. pp. 11-2013. IEEE, 2002.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y. N.Ng. Multi- modal deep learning. In Lise Getoor and Tobias Scheffer (eds.), Proceedings of the 28th Interna- tional Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28- July 2, 2011, pp. 689-696. Omnipress, 2011. URL https: /ttpc//mc//////ppae///////// semppaper... pdf
Huy Nguyen, Xing Han, Carl Harris, Suchi Saria, and Nhat Ho. On expert estimation in hierarchical mixture of experts: Beyond softmax gating functions. arXiv preprint arXiv:2410.02935, 2024.
Behnaz Nojavanasghari, Deepak Gopinath, Jayanth Koushik, Tadas Baltrušaitis, and Louis-Philippe Morency. Deep multimodal fusion for persuasiveness prediction. In Proceedings of the 18th ACM International Conference on Multimodal Interaction, pp. 288-288, 2016
Ari Pakman, Amin Nejatbakhsh, Dar Gilboa, Abdullah Makkeh, Luca Mazzucato, Michael Wibral, and Elad Schneidman. Estimating the unique information of continuous variables. In Mare Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wort- man Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Confer- ence Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 20295-20307, 2021. URL https::/proceedingge www.:eidwwiiinnnni......... wwt.uurrpsssccc.oooeeeoooooooeoe wtt.eurrpppcccppppe/ee.epppp paper/ 2021 2ataaahhhaaa111111110 o531 336 Abst htm
13

---

Published as a conference paper at ICLR 2026
Sarah R Partan and Peter Marler. Issues in the classification of multimodal communication signals. The American Naturalist, 166(2):231-24) 2005.
Soujanya Poria, Erik Cambria, and Alexander Gelbukh. Deep convolutional neural network tex- tual features and multiple kernel learning for utterance-level multimodal sentiment analysis. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 2539 2544, 2015.
Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, Amir Zadeh, Chengfeng Mao, Louis-Philippe Morency, and Ehsan Hoque. Integrating multimodal information in large pretrained transformers. In Proceedings of the Conference of the Association for Computational Linguistics, volume 2020, pp. 2359, 2020.
Attila Reiss and Didier Stricker. Introducing a new benchmarked dataset for activity monitoring. In 2012 16th international symposium on wearable computers, pp. 108-109. IEBE, 2012.
Stephan Reiter, Bjorn Schuller, and Gerhard Rigoll. Hidden conditional random fields for meeting segmentation. In 2007 IEEE International Conference on Multimedia and Expo, pp. 639 642. IEEE, 2007.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Su- sano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp- 8583-8598 2021. URL
neurips...........p............s wwtpppppe/c22///aaa.....ee Aactractt.....a.aa.......c html
Jorma Rissanen. Modeling by shortest data description. Automatica, 14(5):465471, 1978.
Natalie Ruiz, Ronnie Taib, and Fang Chen. Examining the redundancy of multimodal input. In Pro- ceedings of the 18th Australia conference on Computer-Human Interaction: Design: Activities, Artefacts and Environments, pp. 389-392, 2006.
Philip Schmidt, Attila Reiss, Robert Duerichen, Claus Marberger, and Kristof Van Laerhoven. In- troducing wesad, a multimodal dataset for wearable stress and affect detection. In Proceedings of the 20th ACM international conference on multimodal interaction, pp. 400-408, 2018.
Xiaojun Shan, Qi Cao, Xing Han, Haofei Yu, and Paul Pu Liang. Mint: Multimodal instruction tuning with multimodal interaction grouping. arXiv preprint arXiv:2506.0233 2025.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture- of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL net htt/://oommmdmileemmmmm
https://openreve/
Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. Mome: Mixture of multi- modal experts for generalist multimodal large language models. Advances in nearal information processing systems, 014222484427777 2024.
Satya Narayan Shukla and Benjamin M Marlin. Multi-time attention networks for irregularly sam- pled time series. arXiv preprint 1rXiv:2101.10316 2021.
Luis R Soenksen, Yu Ma. Cynthia Zeng, Leonard Boussioux, Kimberly Villalobos Carballo, Liangyuan Na, Holly M Wiberg, Michael L Li, Ignacio Fuentes, and Dimitris Bertsimas. In- tegrated multimodal artificial intelligence framework for healtheare applications. NPJ digital medicine, 5(1):149, 2022.
Daria Sorokina, Rich Caruana, Mirek Riedewald, and Daniel Fink. Detecting statistical interac- tions with additive groves of trees. In William W. Cohen, Andrew McCallum, and Sam T. Roweis (eds.), Machine Learning, Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008, volume 307 of ACM International Confer- ence Proceeding Series, pp. 1000-1007 ACM, 2008 doi: (9.114///390155511999222 13902822
URL
https://doi .rrg
14

---

Published as a conference paper at ICLR 2026
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang. J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp- 6558, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/ P19-1656. URL / http://///ccccaoooooooooo..ggg./
Michael Tsang, Dehua Cheng, and Yan Liu. Detecting statistical interactions from neural network weights. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 2018, Conference Track Proceedings. OpenReview.net, 2018. URL BtR..ttps:/ bettpss:/apanreviiie
net fortm?/ormm?mmm informmiidza (immmnddBByfffggFf
Michael Tsang, Dehua Cheng, Hanpeng Liu, Xue Feng, Eric Zhou, and Yan Liu. Feature interaction interpretability: A case for explaining ad-recommendation systems via neural interaction detec- tion. In 8th Internatioal Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL hnttps:///penreviiee net/ forum? d=RkgnhTTtDDS
Thomas F Varley. Decomposing past and future: Integrated information decomposition based on shared probability mass exclusions. Plos one, 18(3):e0222950, 2023.
Thomas F Varley. Generalized decomposition of multivariate information. Plos one, 19(2):
c0297128, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gareett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL http::ttt:///tcttc.ttttlll http:tttt//ppocee..iiieeeeeeeeee neur nww. ips CC httpssccppppppp wwcc/pape//eeeeee httpeer//c1//haaaaee awt.ssaaa-bbsttaccc html
Praveen Venkatesh, Corbett Bennett, Sam Gale, Tamina Ramirez, Greggory Heller, Severine Du- rand, Shawn Olsen, and Stefan Mihalas. Gaussian partial information decomposition: Bias cor- rection and application to high-dimensional data. Advances in Neural Information Processing Systems, 36:74602-744666 2023.
Tsachy Weissman, Young- Han Kim, and Haim H Permuter. Directed information. causal estimation, and communication in continuous time. IEEE Transactions on Information Theory, 59(3):
Paul L Williams and Randall D Beer. Nonnegative decomposition of multivariate information. arXiv preprint 15115 2010.
Jun Xie, Yingjian Zhu, Feng Chen, Zhenghao Zhang, Xiaohui Fan, Hongzhu Yi, Xinming Wang, Chen Yu, Yue Bi, Zhaoran Zhao, et al. More is better: A moe-based emotion recognition frame- work with human preference alignment. arXiv preprint arXiv:2508.0660 066366 2025.
Jiayi Xin, Sukwon Yun, Jie Peng, Inyoung Choi, Jenna L Ballard, Tianlong Chen, and Qi Long. 12moe: Interpretable multimodal interaction-aware mixture-of-experts. arXiv preprint arXiv:2505. 10:00./9999 19/90, 2025.
Bo Yang and Lijun Wu. How to leverage multimodal EHR data for better medical predictions? arXiv preprint ww.iv:2110.15110 2021.
Zequn Yang, Hongfa Wang, and Di Hu. Efficient quantification of multimodal interaction at sample level. arXiv preprint arXiv:2506.1724 2025.
Haofei Yu, Zhengyang Qi, Lawrence Jang, Ruslan Salakhutdinov, Louis-Philippe Morency, and Paul Pu Liang. Mmoe: Enhancing multimodal models with mixtures of multimodal interaction experts. arXiv preprint arXiv:2331.095:: 2023.
Lei Yu and Huan Liu. Efficiently handling feature redundancy in high-dimensional data. In Pro- ceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, 2003.
15

---

Published as a conference paper at ICLR 2026
Lei Yu and Huan Liu. Efficient feature selection via analysis of relevance and redundancy. The Journal of Machine Learning Research, 5:1205-1224, 2004.
Sukwon Yun, Inyoung Choi, Jie Peng. Yangfan Wu, Jingxuan Bao, Qiyiwen Zhang, Jiayi Xin, Qi Long, and Tianlong Chen. Flex-moe: Modeling arbitrary modality combination via the flexible mixture-of-expert. Advances in Neural Information Processing Systems, 37:98782-98805, 2024.
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Mosi: multimodal cor- pus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint 160.::1606.06259, 2016.
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis- Morency. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1103-1114, Copenhagen, Denmark, 2017. Association for Computational Linguistics. doi: 10. 10.181631//D17-1115 URL https:
Shengzhe Zhang, Liyi Chen, Dazhong Shen, Chao Wang, and Hui Xiong. Hierarchical time- aware mixture of experts for multi-modal sequential recommendation. In Proceedings of the ACM on Web Conference 2025, pp. 3672-3682, 2025.
Xinlu Zhang, Shiyang Li, Zhiyu Chen, Xifeng Yan, and Linda Ruth Petzold. Improving medical pre- dictions by irregular multimodal electronic health records modeling. In International Conference on Machine Learning, pp. 41300 41313. PMLR, 2023.
Liangwei Nathan Zheng, Wei Emma Zhang, Mingyu Guo, Miao Xu, Olaf Maennel, and Weitong Chen. Rethinking gating mechanism sparse moe: Handling arbitrary modality inputs with confidence-guidee gate. arXiv preprint arXiv:2505. 19525, 2025.
Jixian Zhou, Fang Dong, Ruijun Huang, Hengjie Cao, Mengyi Chen, Yifeng Yang, Anrui Chen, Mingzhi Dong, Yujiang Wang, Dongsheng Li, et al. Oracle-moe: Locality-preserving routing in the oracle space for memory-constrained large language model inference. In Forty-second International Conference on Machine Learning, 2025.
Yitong Zhu, Lei Han, GuanXuan Jiang, Peng Yuan Zhou, and Yuyang Wang. Hierarchical moe: Continuous multimodal emotion recognition with incomplete and asynchronous inputs. arXiv preprint 03X11:2508.02133, 2025.
16

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORKS
Partial Information Decomposition (PID) (Williams & Beer 2010 Bertschinger et al. 2014 has emerged as a formal way to quantify multimodal interactions by measuring how the total informa- tion between two modalities (21, P22 useful for a task y can be decomposed into redundant (R), unique (U), and synergistic (S) parts. Redundancy measures the common information between two modalities, uniqueness measures the useful information in a modality not present in the others, and synergy measures the new information that arises only when both modalities are fused. Specifically, given unimodal marginal distributions p(z1, y) and p(22, y) over each modality and the multimodal joint distribution 1 22, y). a formal definition is
R max [X11; X2: Y); ,EAp
(15)
U1 min 1x( X2), rEAp
U2 = min 1, ((x; YJ)),, yEAp X2;))
(16)
S Ix(X, X2; Y) min nEA,
(17)
where A = (q E A ((.,,))) y) p(a,,y, Vy E V,2; E X,i E [2]} characterizes the set of marginal-matching joint distributions, and lg is the mutual information (MI) over these joint distribution (Z), 22. y). The key lies in optimizing 9 Ap such that the marginals q(x1,)) y) p(21, v) and g( y) p(22, y) are preserved, while relaxing the coupling between 21 and 22; that is, (2) is not necessarily equal to p(r1,x2). x2). The intuition behind this is that redundancy and uniqueness should be identifiable given access only to p(11, y) and p(r2, y). and thus should depend solely on q E Ap. In contrast, synergy inherently depends on the joint distribution p(æ1, 22), which is reflected in Eq 17 relying the full distribution p.
Early approaches to multimodal fusion employed kernel-based methods Bu- cak et al.] 2013] Chen et al. 22044, Poria al. 2013). graphical models (Nefian et al. 2002 Garg
Multimodal Fusion.
et al., 2003 Reiter et al.] (007). and neural networks (Ngiam et al., 2011 Gao et al. 2015 Noja-
vanasghari et al. 2016). With advances in deep learning, more sophisticated fusion strategies have emerged. Tensor- based methods (Zadeh et al. 2017 Liu al. 2018) perform outer- outer-product product fu- sion to capture multiplicative interactions, while attention- based approaches (Rahman et al., 2020) Yang & Wu (02T) generate cross tross-modal displacement vectors through self-attention mechanisms. The Multimodal Transformer (MulT) (Tsai et al.] (019) introduced cross-modal attention blocks for word-level alignment across vision, language, and audio. In clinical settings, late fusion approaches KKadanga et al., 2019; Deznabi et al. 202I) concatenate embeddings from pre-trained encoders, while Soenksen et al. (2022) developed a generalizable pipeline for electronic health records span- ning four modalities through direct concatenation followed by gradient boosting. More recently, MISTS (Zhang al., 2203) extended cross- cross-modal attention with multi-tim attention modules to handle temporal irregularities.
Table 3: Comparison between Multimodal MoE and MERGE for massively multimodal learning.
Aspect
Multimodal MoE
MERGE
Routing Basis
Expert- token similarity
Temporal RUS Similarity
Modality Scalability
O(M²) ) implicit Implicit
Sub-linear (RUS- guided)
Cross-Modal Awareness
Explicit pairwise temporal interaction
Expert Specialization
Emergent from training
Principled (R/U/S-specific)
Temporal Dynamics
Single time point Black-box Standard sparsity Fixed
Multi-lag temporal interaction
Routing Decision
Interpretable RUS values RUS-optimized sparsity
Computational Efficiency
Fusion Strategy
Adaptive mixture of fusion types
MoE- Multimodal Fusion. MoE architectures have emerged as a promising paradigm for multimodal fusion due to their ability to scale efficiently while enabling expert specialization. LIMoE (Mustafa et al., 2022) pioneered large-scale multimodal MoE by processing both images and text through a shared sparse transformer with contrastive learning, demonstrating that experts naturally specialize in different modalities through entropy-based regularization. FuseMoF. (Han
17

---

Published as a conference paper at ICLR 2026
et al. 2024 addressed the "FlexiModal" setting with inregularly sampled and missing modalities, introducing a Laplace gating function with theoretical convergence guarantees superior to softmax routing. Building this, Flex-MoE (Yun et al. (024) proposed missing modality bank and dual- router design (G-Router for generalized knowledge, S-Router for modality-specifee combinations) to handle arbitrary modality availability. Hierarchical MoE (Nguyen al. 2024) demonstrated that Laplace gating at two bierarchical levels eliminates undesirable parameter interactions, accelerat- ing expert convergence in multimodal tasks. I' (Xin et al. 2025) incorporated interaction-type awareness by assigning separate experts to redundancy, uniqueness, and synergy interactions us- ing weakly supervised losses derived from Partial Information Decomposition. ConfSMoE (Zheng et 2025) addressed expert collapse through confidence-guidee gating that detaches routing scores from task confidence, providing theoretical insights into why experts fail to specialize under stan- dard softmax routing. Despite these advances, existing multimodal MoE approaches face limitations in massively multimodal settings involving dozens to hundreds heterogeneous input streams. We summarize these challenges and compare key aspects of multimodal MoE with MERGE in Table
B
DATASET INFORMATION AND PROCESSING DETAILS
B.I MIMIC-IV
The Medical Information Mart for Intensive Care (MIMIC-IV) ecosystem comprises deidentified clinical databases of patients admitted to the emergency department or intensive care unit (ICU) at Beth Israel Deaconess Medical Center. We use lab measurements, vital signs, and radiology notes from MIMIC-IV and MIMIC-IV-Note (Johnson et al., 2023), along with chest X-rays from MIMIC-CXR (Johnson et al. 2019a b). to construct temporal sequences of ICU stays. Input signals are categorized into three modalities: labs/vitals, clinical notes, and chest X-rays. We extract embed- dings from clinical notes using BioBERT (Lee et al. (2000) and from chest X-rays using DenseNet (Huang et al.) 2017) pre-trained on CheXpert (Irvin et al., 2019). We consider two prediction tasks: in-hospital mortality (IHM) and length-of-stay (LOS). For IHM, stay is labeled as mortality if the patient died after the first 48 hours, otherwise as survival. For LOS, the goal is to predict whether the patient will leave the ICU alive in less than 96 hours. The train, validation, and test splits consist of 32,435, 6,950, and 6,952 stays, respectively.
B.2 CMU-MOSI
CMU-MOSI (Zadeh et al., 2016) multimodal dataset consisting of 2,199 opinion video seg- ments, including speech, visual gestures, and audio, drawn from 93 YouTube vlog videos and an- notated for sentiment intensity. We use the preprocessed version from MultiBench (Liang et al.) 202I) and focus on the binary sentiment classification task (positive negative). The modalities considered are vision, text, and audio. The train, validation, and test splits contain 1,283, 214, and 686 segments, respectively.
B.3 WESAD
WESAD Schmidt et al. 2018) is multimodal dataset for wearable stress and affect detection, collected from 15 participants in a controlled laboratory study. includes physiological and motion data from chest- and wrist-worn devices, covering signals such as electrodermal activity (EDA), electrocardiogram (ECG), acceleration, temperature, blood volume pulse (BVP), and respiration. The task is affect recognition across six states: transient, baseline, stress, amusement, meditation, and unknown. We group the signals into two modalities: chest signals and wrist signals. The dataset is split by subject, with 10, and 3 subjects allocated to the train, validation, and test sets, respectively.
B.4 OPPORTUNITY
The Opportunity dataset (Chavarriaga et al. 2013) comprises a comprehensive human activity recog- nition corpus collected in sensor-rich kitchen environment, featuring four subjects performing activities of daily living (ADL). The dataset contains 20 experimental sessions, with each subject completing ADL runs (S1-ADLI through S4-ADL.5), while drill sessions were excluded from
18

---

Published as a conference paper at ICLR 2026
experimental protocol. Data collection was conducted a 30 Hz sampling frequency, yielding 250 feature columns encompassing 243 sensor measurements and activity labels. The activity recogni- tion task focuses high-level activity classes: Relaxing (101), Coffee time (102), Early morning (103), Cleanup (104), and Sandwich time (105). The preprocessing pipeline implemented a subject- based train/validation/test split protocol to prevent temporal data leakage, allocating subjects 1-3 (runs ADLI-ADL for training (342,535 samples), subjects 1-3 (run ADLS) for validation (76,399 samples), and subject 4 (all runs) for testing (111,644 samples). The 243 sensor features were sys- tematically categorized into seven distinct modalities based on anatomical location and sensor type: Torso (19 sensors), Arms (58 sensors), Legs (52 sensors), Shoes (32 sensors), Objects (35 sensors), Environment (34 sensors), and Location (12 sensors).
B.5 PAMAP2
The PAMAP2 Physical Activity Monitoring dataset (Reiss & Stricker 2012) comprises a compre- hensive multimodal sensor corpus collected from nine participants performing 18 distinct physical activities in naturalistic settings. The dataset incorporates heterogeneous sensor modalities includ- ing three Colibri wireless Inertial Measurement Units (IMUs) positioned at anatomically strategic locations: dominant wrist, chest, and dominant ankle, sampling at 100 Hz, complemented by heart rate monitor operating at approximately 9 Hz. Each experimental session captures 54-dimensional feature vectors encompassing temporal timestamps, activity labels, and 52 attributes of raw sen- sory measurements spanning accelerometry (both 16g and 6g ranges), gyroscopy, magnetometry, and quaternion orientation data. The activity recognition framework targets 18 physical activities including fundamental locomotion patterns (lying, sitting, standing, walking. running), complex motor tasks (cycling, stair navigation), daily living activities (computer work, household tasks), and recreational pursuits (soccer, rope jumping). Similar to the Opportunity dataset, we implemented subject-bases based partitioning strategy to ensure temporal independence and prevent data leakage, allocating subjects 1-6 for training, subject for validation, and subjects 8-9 for testing. The 52- dimensional sensor feature space was systematically organized into four modalities: chest, hand, ankle, and heart rate.
C DETAILED PROCEDURE OF MULTI-SCALE BATCH ESTIMATOR
Algorithm Phase Multi-Task Discriminator Training
Require: Multi-lag datasets (D, J
JK11 lag weights }
1: Initialize shared backbone networks and lag lig-specftic components Naiee do
2: for epoch
3: for step to steps.per.eeoch do
4: Sample lag ~ Categorical((w:))
5: Sample batch (z (b) (b)
2.2.7. (()) ~ D,
6: Forward pass with lag conditioning:
7:
logits, - (2 (..
8:
logits, - (2,0(02..77
(b)
9:
logits;2 < D12 (01220(12)) (1x1 22
Compute discriminator loss:
11: Lrutch +- ()) (b) logits,) + (() logits,) + H ())()), logits
Update via Vg
13: end for
14:
end for
15: return Trained discriminator parameters d
19

---

Published as a conference paper at ICLR 2026
Algorithm 2 Phase 2: Multi-Task Alignment for Q
Require: Multi-lag datasets {D, [D-J/=0 lag weights {wr}. trained discriminators Oaisc
1: Initialize lag-conditioned Alignment module with parameters Oatign
2: Freeze discriminator parameters disc
3: for epoch to Nalign do
4: for step 1 to steps per epoch do
5: Sample lag ~ Categorical (w=))
6: Sample batch (- (b) (b)
$2.77 (b) Dr
7:
Compute lag- lag-condttioned embeddings:
dx. + 01 (10,, ( (b) e(r))
8:
9:
Ox₂ f (22,20 ( e(r))
10:
Compute lag-specific alignment:
11:
align,18.j.A) exp
Vd
12:
Sinkhorn normalization with lag-specific discriminators:
Prix,,7 Softmax 1.0 (I1,T) T)) Pr|x,,7 Softmax (D2 (I2, T)) for k = to C. do
15:
for iter
iier do Q- f- Apply Sinkhorn updates on align, [b i, Check convergence
17:
using Prix,, and PriXai
19:
end for end for
20:
21: Compute alignment loss:
Lalign < AlignmentL.oss(C i..r
1,rt 22g. (0))
(6) (b)
23: Update alignment parameters falggn via V Ong Calign
end for
25:
end for
26: return Trained alignment parameters Galign
Algorithm Phase 3: Multi-Lag RUS Computation
Require: Multi-lag datasets [D> trained discrininators Ouice. trained alignment module Oalign
for (0, I,..., K 1} do
2: Compute mutual information (MI) terms using trained discriminators:
3: MI,(Y; X2) <- EstimateMMI 1,0am Sase (: (, D-)
4: MI,(Y;
EstimateMM D2.0am 8a0 (,,T) D
5:
MY; X2) +- EstimateMM 12. Ron.. (:,
D,)
6: Compute optimal alignment distribution:
7: Q: < CompputeOptimalAAlggnnee
(Qalign: Dr,7)
8: Mlo:
<- EstimateMI((Q: D,)
9:
Decompose into RUS components:
R, MY; MI, (Y; Ml (Y; X2) (Redundancy)
11:
UL. t Mlg: (Y; X1, X²) -Y;+ X₂
Uniqueness X1)
12:
U2.r Mlg: MY;= X2) -Y; (Uniqueness X₂
13: S, + (Y; - -Mle; (Y;X,, X2) {Synergy)
end for
15: return Temporal RUS components (R,, U1. U2 S-1K-1
20

---

Published as a conference paper at ICLR 2026
D MODEL ARCHITECTURE DETAILS
D.1 MODALITY-SPECIHT ENCODER
Component
Configuration
Input Projection Layer
Linear transformation from modality dimension Dm to model dimension dmodel
Feature Scaling
Input embeddings scaled by ddmoddt for training stabil- ity
Temporal Convolution (Optional)
Two 1D convolutional layers with residual connections for temporal feature extraction
Convolutional Architecture
ID convolution: dmrodet - dmodet with kernel size k Applied after each convolutional layer
Batch Normalization
Activation Function
ReLU activation for non-linearity
Self-Attention Layers
Multi-layer transformer encoder
Output Normalization
Layer normalization applied final embeddings
Table 4: Modality- -Seeciii- Encoder Architecture Components
Parameter
Value
Description
Encoder Layers (Lene)
Number of transformer encoder layers Multi-head attention heads per layer
Attention Heads (H)
4
Feed-Forwand Forward Dimension (dgr)
256
Inner dimension of position-wise FFN Dropout probability for regularization Temporal convolution kernel size
Dropout Rate (parop)
0.1
Convolution Kernel (k:)
Table 5: Modality-Speecifi Encoder Hyperparameters
D.2 EXPERT NETWORKS
Layer
Transformation
Input Layer
Linear transformation: IR Amodde -> Redepeett
Activation
ReLU non-linear activation function
Regularization
Dropout with probability Párop
Output Layer
Linear transformation: IR Redespert + I gemodda
Table 6: Feed-Forward Expert Network Architecture
Component
Description
Multi-Head Self-Attention
Scaled dot-product attention with H parallel heads Residual connection around attention sub-layer
Attention Residual
Post-Attention Norm
Layer normalization after attention residual
Position-wise FFN
Two-layer feed-forward network with ReLU
FFN Residual
Residual connection around feed-for ferwwrr sub-layer
Final Normalization
Layer normalization after FFN residual
Dropout Regularization
Applied to attention output and FFN output
Table Synergy Expert Network Architecture
21

---

Published as a conference paper at ICLR 2026
Parameter
Default
Description
Expert Hidden Dimension (dexpert)
128
Internal processing dimension Attention heads for synergy experts
Synergy Expert Heads (H syn)
4
Expert Dropout (Peapert)
0.1
Dropout rate within expert networks
Synergy Experts (N (Nagn)
2
Number of iitention-based synergy experts Total number of expert networks
Total Experts (N (Nexpert)
8
Table 8: Expert Network Hyperparameters
D.3 RUS-AWARE ROUTER
Component
Description
Token Processing
Linear projection: I Rdmodcl + R Rdioken with ReLU activa- tion Projects processed tokens to attention query space: Rdokken + d2
Query Generation
RUS Key Projection
Maps pairwise RUS values attention keys: R + Rd Maps pairwise RUS values to attention values: R Rdd.
RUS Value Projection
Temporal RUS Encoder
Single-layer GRU processes concatenated uniqueness and attention context
Routing MLP
Two-layer network outputs expert routing logits
Table 9: RUS-Aware Routing Network Architecture
Parameter
Value
Description
GRU Hidden Dimension (dgrw)
64
Temporal context encoding dimension Intermediate token representation size
Token Processing Dimension (chaken)
64
Attention Key Dimension (dx)
32
Query-key attention space dimension
Attention Value Dimension (d,)
32
RUS value projection dimension
Table 10: Routing Network Hyperparameters
E CHOICE OF
IN AUXILIARY LOSSES
In this section, we discuss the roles of the hyperparameters and À that appear in the auxiliary loss terms in Eq Bq 14 and provide guidance how tune them for improved performance in multimodal applications. The parameters (TR, Ts) serve as threshold values that determine when activate diffferent routing behaviors based on the RUS scores. Conceptually, they control the selectivity of the routing mechanism. If threshold is set too low, almost all interactions will trigger standard MoE regime and effectively
specialized routing: if set too high, the routing reverts ignores RUS BUS-guided specialization. Empirically, synergistic interactions tend to be much rarer than redundancy or uniqueness real-world multimodal data, so we typically set Ts lower to avoid overly suppressing synergy-driven routing and to maintain balanced distribution of routing patterns.
The parameters (An, Au, As) regulate the strength of the auxiliary loss terms corresponding to each interaction type. These can be selected based domain knowledge, e.g., whether dataset is domi- nated by synergistic patterns, highly independent modalities, etc., or adjusted to explicitly encourage certain forms of interaction. They also be made adaptive during training, with validation perfor- mance guiding the adjustment.
Although and À encode different aspects of the system, they related in effect: increasing threshold т reduces how often particular interaction pattern is activated, which analogous to decreasing the corresponding X, thereby weakening its influence in training.
22

---

Published as a conference paper at ICLR 2026
In practice, find that because the empirical distribution of RUS values is highly non-uniform, the thresholds т (typically set as specific percentiles their respective RUS distributions) are relatively insensitive to fine- aineggained tuning. Therefore, precompute and fix based dataset statistics and only tune the values during model training.
ADDITIONAL. EXPERIMENTS
F.1 SYNTHETIC TEMPORAL RUS
Synthetic data generation To validate our multi-scale BATCH estimator framework for computing temporal RUS, created synthetic benchmark with known R/U/S values. This synthetic dataset consists of three time series: two independent source variables X, and X2, and a target variable Y. Both X₁ and X2 were independently sampled from N(0, 1). Y was constructed with explicit temporal causal influences from both sources, where X1 influences Y with a time lag of (T = 1) and X2 influences
2). At each time point t, (t) was computed Y(t) = (t - 1) + a (t 2) + 1 e(t). where <(t) is noise sampled from N0, 1). We followed the definition direction information (Weissman al. 2012) and information decomposi- tion (Bertschinger et al. 2014) to compute temporal RUS values. Figure 9 demonstrates an example
with time lag of
of the temporal RUS values we obtained for the synthetic experiments.
PID Components vs Time Lag
Information (bits)
Total DI
0:30
Redundancy
0:25
Unique
0:20
Unique
0.15
Synergy
0.10
0.08
Time Lag
Stacked PID Components
Information (bits)
Redundancy
1.30
Unique
1:2
Unique
0.20
Synergy
Total DI
6.15
6000
15
1.
35
42
Time Lag
Figure 9: Ground truth temporal RUS of synthetic dataset: 1, uniqueness of Xı dominates the total directed information; at = 2, uniqueness of Xa dominates; from = 3 and afterwards, synergy between X2 and X₂ dominates the total directed information.
We compare the proposed multi-scale BATCH estimator with the step-wise temporal RUS compu- tation using data with known RUS. Our evaluation spans multiple aspects, including varying sample sizes, different feature dimensions for X1/X2, encoder capacity within the BATCH estimator, and the number of Sinkhorn iterations used during optimization. Figure I0 to Figure [3] demonstrate these results: all experiments are averaged across random runs with different time lag configura- tions; the reported R/U/S values are also aggregated across time lags.
F.2 EFFECT OF BAD R/U/S ESTIMATIONS
We conducted additional experiments to assess how sensitive MERGE to inaccuracies in the temporal RUS estimates, particularly in scenarios where the RUS signals are weak noisy. To this end, we evaluated two variants of the model: (1) Noisy RUS: we injected Gaussian noise into the computed temporal RUS sequences to simulate unstable estimation. (2) Sparse weakened RUS:
23

---

Published as a conference paper at ICLR 2026
Redundancy (R)
Unique X1 (U1)
0.10
0.20
Information (bits)
Information (bits)
0.05
0.15
0.10
0.00
0.05
Ground Truth
0.00
Ground
-0.05
BATCH
BATCH
-0.05
Multi-Scale BATCH
Multi-Scale BATCH
-0.10
0
5000
10000
15000
20000
5000
10000
15000
20000
Sample Size
Sample Size
Unique X2 (U2)
Synergy (S)
0.20
0.25
Information (bits)
0.15
Information (bits)
0.20
0.15
0.10
0.10
0.05
0.05
0.00
Ground Truth
0.00
Ground Truth
BATCH
BATCH
-0.05
Scale BATCH
-0.05
Multi-Sed Scale BATCH
-0.00
-0.0
5000
10000 15000
20000
5000
10000
15000
20000
Sample Size
Sample Size
Figure 10: Ablation study of the multi-scale BATCH estimator under different sample sizes: we tested both methods on [500, 1000, 5000, 10000, 20000] samples. Results show that the multi-scale BATCH estimator leads to similar results with step-wise BATCH estimator. As sample size in- creases, both two methods achieve better approximations to ground truth R/U/S values.
zeroed out majority of the temporal RUS sequence values, degrading them into sparse interaction patterns. Figure [4] demonstraaess the results.
F.3 TRAINING TIME & MAX GPU MEMORY
We summarize the total training cost, including both temporal RUS estimation and MERGE train- ing, in terms of wall-clock training time and maximum GPU memory consumption across all datasets evaluated in the paper. Figure presents scatter plots showing the distribution of train- ing time versus peak GPU memory for each dataset.
F.4 EFFECT OF CLASS IMBALANCE IN RUS ESTIMATION
For datasets with sequence-level labels such as MIMIC-IV and MOSI, the prediction task is per- formed at the full-sequence level. For example, in MIMIC-IV IHM, the model uses the first 48 hours of ICU measurements to predict in-hospital mortality. Accordingly, we compute temporal RUS the final time step (i.e., sweeping lags backward from the end of the 48-hour window), since these values are the relevant values aligned with the prediction target. However, in situations such as class imbalance and label noise can influence both temporal RUS estimation and downstream rout- ing behavior. As discussed in Section 4 each patient contains multiple measurements D; E RdxT, but associated with only a single label Y, E 10, ..)....... Thus, meaningful estimation of the joint distribution (Xx, X2, YY) requires a batch of patient trajectories with sufficient label diversity; otherwise the joint distribution becomes degenerate.
We use MIMIC-IV IHM (in-hospital mortality) as concrete example to study the importance of class balance and label quality for meaningful temporal RUS estimation. In-hospital mortality is binary classification task where the label indicates whether the patient dies after ICU discharge, we designed three conditions to evaluate how class imbalance and label quality affect both RUS estimation and model performance:
24

---

Published as a conference paper at ICLR 2026
Redundancy (R)
Unique X1 (U1)
0.10
Information (bits)
Information (bits)
0.05
0.00
Ground Truth
Ground Truth
BATCH
BATCH
-0.05
Multi-Scale BATCH
Multi-Scale BATCH
-0.10
20
40
60
80
100
0
20
40
60
80
100
Dimension
Dimension
Unique X2 (U2)
Synergy (S)
0.20
0.25
Information (bits)
Information (bits)
0.20
0.15
0.15
0.10
0.10
0.05
0.00
Ground Truth
Ground Truth
0.00
BATCH
BATCH
-0.05
Acale BATCH
Multi Scale BATCH
-0.10
-0.10
100
20
40
60
80
100
Dimension
Dimension
Figure 11: Ablation study on feature dimensionality. We evaluated the multi-scale BATCH estimator under varying feature dimensions 5, 10, 50, 100). Note that the boound-truth R/U/S values natu- rally change as the dimensionality of the features increases. Across all tested dimensions, both the multi-scale BATCH estimator and the step-wise computation produce consistent R/U/S estimates.
A batch containing only patients with outcome 0 or only outcome 1. In this case, temporal RUS cannot be meaningfully computed under the information decomposition framework because X2, Y) becomes degenerate.
2. A batch containing both labels but with > 95% belonging to class O. RUS can be computed, but the resulting temporal RUS sequence is substantially less informative, with the modality interaction over time remaining almost constant.
3. A batch with approximately balanced labels and sufficiently rich histories. This corre- sponds to the original temporal RUS used in the main experiments.
In Figure 16, we use the temporal RUS values of the "Insulin - Furosemide" modality pair as an illustrative example. Such results provide us insights on the importance of class balance and label quality for meaningful temporal RUS estimation, and demonstrate how degraded RUS estimation can affect the final model performance.
25

---

Published as a conference paper at ICLR 2026
Redundancy (R)
Unique X1 (U1)
0.10
Information (bits)
Information (bits)
0.15
0.05
0.10
0.00
Ground Truth
0.00
Ground Truth
-0.05
BATCH
BATCH
00.5
Multi-Scale BATCH
Multi-Scale BATCH
-0.10
0.10
Low
Mid
High
Low
Mid
High
Encoder Capacity
Encoder Capacity
Unique X2 (U2)
Synergy (S)
0.25
0.20
Information (bits)
0.15
Information (bits)
0.15
0.10
0.10
0.05
0.05
0.00
Ground Truth
0.00
Ground Truth
BATCH
BATCH
Multi-Scale BATCH
-0.05
Multi-Scale BATCH
-0.10
-0.0
Low
Mid
High
Low
Mid
High
Encoder Capacity
Encoder Capacity
Figure 12: Ablation study on encoder capacity. We further tested the effect of encoder capacity in the BATCH estimator by evaluating configurations: low, medium, and high capacity. In the low- capacity setting, the MI.P encoder uses a hidden dimension of 16 with layer; the medium-capacity setting uses hidden dimension of 32 with layers; and the high-capacity setting uses hidden dimension of 64 with 3 layers. Across all tested scenarios, we observe that the low- and medium- capacity encoders generally sufficient for accurate synthetic R/U/S estimation, while increasing the capacity yields only marginal improvements.
26

---

Published as a conference paper at ICLR 2026
Redundancy (R)
Unique X1 (U1)
0.10
0.20
Information (bits)
Information (bits)
0.15
0.05
0.10
0.00
0.05
Ground Truth
0.00
Ground Truth
-0.05
BATCH
BATCH
-0.05
Sulti-Scale BATCH
Multi-Scale BATCH
-0.10
100
200
300
400
500
100
200
300
400
500
Sinkhorn Iteration Steps
Sinkhorn Iteration Steps
Unique X2 (U2)
Synergy (S)
0.20
0.25
Information (bits)
0.15
Information (bits)
0.15
0.10
0.10
0.05
0.05
0.00
Ground Truth
Ground Truth
BATCH
BATCH
-0.0
Multi-Scale BATCH
Multii-calee BATCH
-0.10
-0.00
100
200
300
400
500
100
200
300
400
500
Sinkhorn Iteration Steps
Sinkhorn Iteration Steps
Figure 13: Ablation study on the number of Sinkhorn iteration steps. evaluated both methods using range of Sinkhorn iteration counts: [100, 300, 400, 500). Across all settings, the two methods produce close R/U/S estimates, and the estimator seems relatively insensitive to the choice of iteration count.
NINICA
Oppertunty
Ausurey
Acnunay
Original
Nata:
Deadine
Oriinnl
Deac
Deadine
Original
Wesak
Daadee
(a) PAMAP2
(b) MIMIC-IV IHM
(c) MIMIC-IV LOS
(d) Opportunity
Figure 14: Effect of inaccurate R/U/S estimation on model performance. Across the four evalu- ated datasets, we investigated how degrading the quality of the temporal R/U/S estimates impacts MERGE's performance. We observed that adding Gaussian noise to the R/U/S sequences has mild impact on downstream performance. However, making the R/U/S sequences weak or sparse leads substantial performance degradation, often falling below baseline models. We speculate that noisy R/U/S values still preserve the overall structure of modality interactions, enabling the router to make approximately correct decisions. However, sparse or near-zero R/U/S sequences fail to pro- vide meaningful interaction signals. As result, the model receives misleading guidance and may adopt incorrect or inconsistent routing patterns.
27

---

Published as a conference paper at ICLR 2026
TOSAAEE TUC
Toll Than Thee 01
VGTS
Tram Teee
Toaa Than
Tambrre
Mes GPL Minnory (161)
baex GPU Derrayy (M81))
(MIT)
(a)
(b)
(c)
Oppartuntty
WESSO
MOSI
TMF AMF
THM Mec
Y:07 puaning fend
Ruserc
VISTS
Toual Trakeee
MESTC
Tosu Trakso
V.41
MeT
Tondorne
Tunsooner
Tondoreer
Wi Max CPU Meretyyye (48)
Mix GPL, Nankey (183)
Via cnU Monery (98)
(d)
(c)
(f)
Figure 15: Comparison of training time and maximum GPU memory usage across all methods and datasets. Although computing and incorporating temporal RUS values into MoE training introduces certain amount of additional computation and training time, the overall resource requirements of MERGE remain comparable to baseline multimodal Transforner/MoE architectures, such as FuseMoE (Han et al. 2024) and MISTS (Zang al.] (023), while being more efficient than I2MoE (Xin et al. 2025). Note that the sequence length of the temporal RUS values varies depending on the task and dataset; longer sequences naturally increase memory consumption. As a result, the relative positioning of MER HERGE in the training-time v.s. memory plot varies across datasets. Models such as the standard multi-head self-attention Transformer, mTAND, and MulT represent carlier-generation iicettionbbase architectures with lower computational requirements, and thus appear on the lower end of the resource spectrum.
Insulin + Furosemide
Insulin Furosemide
0.35
U2
.R. (RedUndancW
U1
R (Redundancy)
Information (bits)
0.30
Us (Unique X2)
(Unique X1)
Information (bits)
0.4
Us lUnique X2V
U. jUnique
0.25
S (Synergy)
S (Synergy) [Tatal]
(Total)
0.3
0.20
0.15
0.2
0.10
0.1
0.05
0.00
Time Lag (hours)
Time Lag (hours)
Highly imbalanced batch
Balanced batch
We use the "Insulin + Furosemide" modality pair as an example to compare temporal RUS estimations obtained under imballnebe settings versus those computed under the regular (balanced) setting. We fed RUS sequences from both highly imbalanced batch and balanced batch into MERGE training. We observed clear performance degradation for the imbalanced batch: AUROC decreased from 85.40 to 82.35 and F1 score decreased from 84.97 to 81.73.
Figure 16:
28


Reviews:
[{"review_id": "5EX6wnLJDs", "reviewer": "Reviewer_UcoF", "summary": "This work introduces TIME-MOE, which routes tokens in a mixture-of-experts using time-aware interaction cues between modalities. The model estimates redundancy, uniqueness, and synergy (RUS) via a directed-information flavored partial information decomposition, producing sequences over different lags. These sequences condition the router, while auxiliary losses encourage consistent behavior: co-route redundant inputs, diversify for unique inputs, and prefer dedicated synergy experts when cross-modal effects appear.", "questions": "1)\tEstimator validation: Could You please provide a synthetic benchmark where ground-truth RUS over lags is known, to calibrate the multi-scale estimator vs. a step-wise computation, including error bars across data sizes? How sensitive are RUS estimates to encoder capacity and the Sinkhorn regularization?\n2)\tLabel choice: For MIMIC-IV and MOSI, have you tried alternative targets (e.g., intermediate pseudo-labels or weak supervision) to compute more informative temporal RUS sequences? Any performance deltas?\n3)\tComputationals : Please report GPU-hours for TIME-MOE vs. other MoE baselines across datasets, and the incremental cost attributable specifically to RUS estimation and auxiliary losses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1)\tTemporal interaction modeling: The paper builds on PID with directed information to model time-lagged redundancy, uniqueness, and synergy between modalities. The derivation reads cleanly and fits the needs of multimodal time series.\n2)\tArchitecture: The RUS-aware router plus synergy experts turn the interaction signals into concrete routing decisions. The activation patterns look modality-consistent and more structured than a standard MoE.\n3)\tEfficient multi-lag estimation: The multi-scale estimator reuses computation across lags, staying close to step-wise estimates while delivering roughly τ-fold speedups. \n4)\tEmpirical results: Strong empirical results & breadth: TIME-MOE wins on most metrics across 6 tasks, including clinical (MIMIC-IV) and affective datasets (MOSI, WESAD); results averaged over 5 runs.\n5)\tReproducibility: Anonymous code link and detailed appendices and hyper params are provided.", "weaknesses": "1)\tEstimator guarantees and identifiability: The construction of optimal Q*_{tau} via Sinkhorn-normalized alignment tensors is elegant but lacks formal guarantees about convergence to the PID-consistent minimizer and the induced bias in R/U/S under finite data and high-dimensional encoders. A short theorem or calibration study (e.g., on synthetic systems with known RUS) would make the claims much stronger.\n2)\tDependency on label availability and distribution: For sequence-level labels (MIMIC-IV, MOSI), RUS is computed at a single time step while sweeping lags from the sequence end. That’s reasonable, but it could be brittle under class imbalance or label noise. Please analyze sensitivity and discuss how label quality affects RUS and routing.\n3)\tScalability and cost: There is qualitative speedup from the multi-scale RUS estimator, but the paper could better quantify end-to-end training cost (GPU-hours) and router overhead vs. a standard MoE for larger expert counts and more modalities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VbaLyICvVH", "reviewer": "Reviewer_RmCG", "summary": "This paper presents the study on Temporal Interaction-guided Mixture-of-Experts (TIME-MoE). This is a modified MoE architecture that explicitly leverages time-varying multimodal interactions to guide expert routing. In conventional MoE routers the tokens are navigated based solely on content similarity, while for some tasks and inputs, the temporal interaction between tokens is crucial. TIME-MoE's router is interaction-aware, it considers the redundancy, uniqueness, and synergy (RUS) between modalities over time when deciding which expert processes which token. By using quantified interaction dynamics as a prior, the model encourages experts to specialize in processing particular interaction patterns (redundant, unique, or synergistic information) rather than only learning task-specific features. \nThe authors testes their methods on various multimodal benchmarks, mostly connected to the medical domain, and found out that their method outperforms alternative MoEs approaches, and also improves interpretability of the method.", "questions": "1. Please clarify the optimizer, learning-rate schedule, batch size, number of epochs, and gating top-k used during TIME-MoE training. Were modality encoders frozen or fine-tuned?\n2. Is RUS estimation performed as a fully offline pre-processing step (aligned with train/validation/test splits), or is there any joint optimization of the RUS estimator with TIME-MoE training?\n3. Did you observe any expert collapse (e.g., one expert processing the majority of tokens)?\n4. How sensitive is the final model’s performance to the accuracy of the RUS estimation? If the interaction estimator produces noisy or weak signals, does performance degrade significantly?\n5. Could you report wall-clock training times, GPU type/count, and memory usage for both RUS estimation and TIME-MoE training, relative to baseline MoE models (e.g., FuseMoE, I2MoE)?\n6. Do you plan to extend TIME-MoE to large-language-model MoE architectures? Such integration could yield interpretable and context-aware routing in large-scale generative models.\n\nIf the authors address my questions, I am willing to increase the final score.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strengths of the research are the following:\n\n1. Novel interaction-aware MoE for multimodal data. The authors integrate an information-theoretic framework with modern deep learning to guide Mixture-of-Experts routing using temporal redundancy, uniqueness, and synergy (RUS). This design yields an MoE that is both temporally and modality-grounded, leading to more specialized and interpretable experts.\n2. Unlike prior approaches that rely on static cross-modal correlations, TIME-MoE explicitly models time-lagged multimodal effects, which are especially important in medical and physiological domains.\n3. Thorough and diverse evaluation. The approach is tested on a broad set of multimodal benchmarks — PAMAP2, MIMIC-IV (IHM/LOS), MOSI, WESAD, and Opportunity — covering various modality mix. The authors validate their mechanism through component ablations and routing-pattern analyses, confirming both effectiveness and interpretability.\n4. Comprehensive ablation studies. Each component (redundancy, uniqueness, synergy guidance) is evaluated separately, demonstrating its distinct contribution to the final performance.", "weaknesses": "I would define the following weaknesses:\n\n1. Pairwise-only interaction modeling. The RUS framework focuses on pairwise modality interactions; higher-order (three-way or more) synergies are not explicitly modeled. While this ensures scalability, it limits the expressiveness of the model.\n2. Although the authors claim a τ-fold speedup in RUS computation and improved parameter efficiency, the paper does not provide GPU hours, or memory-usage comparisons. Including such measurements would make the efficiency claims more convincing.\n3. Integration with large-scale LLM-MoE frameworks remains unexplored. Given the rapid development of Large Language Model MoEs (e.g., Mixtral, DeepSeek-MoE, etc.), it would be highly valuable to discuss how TIME-MoE could scale to these architectures. The paper does not explore or speculate on integrating its RUS-guided routing into LLM-scale systems, which could significantly broaden its impact.\n4. While Appendix E describes architecture and hyperparameters, and Appendix D details the RUS estimation algorithms, the paper lacks a concise description of the training setup (optimizer, learning rate, batch size, epochs, etc.).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Sh5IVcUbY", "reviewer": "Reviewer_z7XH", "summary": "This paper introduces TIME-MoE (Temporal Interaction-guided Mixture of Experts), a novel multimodal architecture that integrates temporal interaction dynamics into the MoE routing process. The core idea is to quantify multimodal interactions (redundancy, uniqueness, synergy – RUS) over time, and use these dynamics to guide expert routing. The authors design a RUS-aware router and corresponding auxiliary losses that encourage experts to specialize in particular interaction types. Empirical results on multiple benchmarks (MIMIC-IV, PAMAP2, WESAD, MOSI, Opportunity) show that TIME-MoE outperforms state-of-the-art fusion and MoE-based baselines (e.g., FuseMoE, I2MoE), achieving both higher predictive accuracy and more interpretable expert activation patterns over temporal dimension.", "questions": "1. How sensitive is TIME-MoE to errors in RUS estimation, especially when temporal dependencies are weak or noisy?\n2. Can the RUS-guided router be learned end-to-end without explicit precomputation of RUS values?\n3. Could the same framework generalize to non-temporal multimodal fusion (e.g., static image–text tasks) or to LLM-scale architectures?\n4. What is the computational overhead compared to FuseMoE and I2MoE in FLOPs and training time?\n5. How does the proposed directed-information-based temporal RUS compare with simpler correlation-based interaction metrics?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Novel theoretical grounding: Extends Partial Information Decomposition (PID) to the temporal domain using directed information, leading to a measurable RUS metric that captures delayed effects.\n2. Principled architecture: The RUS-aware router and auxiliary loss design directly encode theoretical insights into model training.\n3. Empirical generality: Demonstrates improvements across heterogeneous datasets (clinical, affective, activity, physiological).\n4. Interpretability: Expert routing visualizations reveal meaningful alignment with modality interactions (e.g., redundancy between chest and hand motion over the temporal dimension).\n5. Comprehensive evaluation: Includes ablations, sensitivity to RUS sequence length, and comparison with multiple baselines.", "weaknesses": "W1. Computing temporal RUS and training the router could be computationally expensive, which may limit scalability for large multimodal models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "43xBTTiijn", "reviewer": "Reviewer_sito", "summary": "This paper presents TIME-MoE, a Temporal Interaction-guided Mixture-of-Experts framework that leverages information-theoretic decomposition (RUS) to guide expert routing.", "questions": "1. The proposed multi-scale estimator is interesting, but the overall computation involving PID and Sinkhorn alignment still seems heavy, especially for long temporal sequences or high-dimensional inputs. It would be helpful if the authors could comment on the practical computational cost and scalability of TIME-MoE in such settings.\n\n2. The model appears to rely on several threshold and weighting hyperparameters (τ_R, τ_U, τ_S, λ_R, λ_U, λ_S). How sensitive are the results to these choices? Any guidance or heuristics for tuning them would be useful.\n\n3. Most of the experiments are on medium-scale datasets. Have the authors tried (or do they see a clear path to) applying TIME-MoE to larger-scale setups, e.g., vision-language or multimodal LLM tasks?\n\n4. The framework is built on information-theoretic principles, but in practice RUS estimation depends on neural approximations. How stable are these estimations during training, and is there any quantitative sense of their bias or variance?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper introduces a novel information-theoretic framework for guiding expert routing in multimodal MoE systems.", "weaknesses": "The proposed RUS estimation and routing involve substantial computational overhead and complex hyperparameter tuning. Scalability to large-scale multimodal models remains to be shown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)