$\texttt{I$^2$MoE}$: Interpretable Multimodal Interaction-aware Mixture-of-Experts
Authors: Jiayi Xin
Sukwon Yun
Jie Peng
Inyoung Choi
Jenna L. Ballard
Tianlong Chen
Qi Long
Venue: ICML 2025 poster (2025)
Keywords: Multimodal Learning
Mixture of Experts
Biomedical Analysis
Primary area: deep_learning->other_representation_learning
TL;DR: We proposed a Mixture-of-Experts framework for modeling multimodal interactions in a data-driven and interpretable way.

Abstract: Modality fusion is a cornerstone of multimodal learning, enabling information integration from diverse data sources. However, existing approaches are limited by $\textbf{(a)}$ their focus on modality correspondences, which neglects heterogeneous interactions between modalities, and $\textbf{(b)}$ the fact that they output a single multimodal prediction without offering interpretable insights into the multimodal interactions present in the data. In this work, we propose $\texttt{I$^2$MoE}$ ($\underline{I}$nterpretable Multimodal $\underline{I}$nteraction-aware $\underline{M}$ixture-$\underline{o}$f-$\underline{E}$xperts), an end-to-end MoE framework designed to enhance modality fusion by explicitly modeling diverse multimodal interactions, as well as providing interpretation on a local and global level. First, $\texttt{I$^2$MoE}$ utilizes different interaction experts with weakly supervised interaction losses to learn multimodal interactions in a data-driven way. Second, $\texttt{I$^2$MoE}$ deploys a reweighting model that assigns importance scores for the output of each interaction expert, which offers sample-level and dataset-level interpretation. Extensive evaluation of medical and general multimodal datasets shows that $\texttt{I$^2$MoE}$ is flexible enough to be combined with different fusion techniques, consistently improves task performance, and provides interpretation across various real-world scenarios. Code is available at https://github.com/Raina-Xin/I2MoE.

I I'MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts
Jiayi Xin
Sukwon Yun? Jie Peng
Inyoung Choi Jenna L. Ballard
Tianlong Chen
Long'
Abstract
Image Modality
Language Modality
Modality fusion is a cornerstone of multimodal learning, enabling information integration from diverse data sources. However, vanilla fusion methods are limited by (1) inability to account for heterogeneous interactions between modalities and (2) lack of interpretability in uncovering the multimodal interactions inherent in the data. To this end, we propose I MoE (Interpretable Mul- timodal Interaction-aware Mixture of Experts), an end-to-end MoE framework designed to en- hance modality fusion by explicitly modeling diverse multimodal interactions, as well as pro- viding interpretation
STUDEL
Uniqueness Image
beautiful countess Uniqueness
"A student
RRAGUE
hocomes übsessed La
Genres
surcerei makes deal wilh
"Drama"
young
give
Fantasy
Redundancy
fabulons wmallth and anything be wants, if he will sign his contract. The
"Horzor",
of'both
"Romance"
student lurriedly signs contract. but doesn't lanone what ho's for."
Synergy
PAUL WEGENG
Figure I. illustrative example modality interaction. The poster and plot are taken from the IMDB dataset.
local and global level. First, I'M utilizes different interaction experts with weakly supervised interaction losses to learn multimodal interactions in data-driven way. Sec- ond, I deploys a reweighting model that as- signs importance scores for the output of each interaction expert, which offers sample-level and dataset-level interpretation. Extensive evalua- tion of medical and general multimodal datasets shows that I'MoF. is flexible enough to be com- bined with different fusion techniques, consis- tently improves task performance, and provides interpretation across various real-world scenar- ios. Code is available at https: bttps://githhe thg............. con/ htn/Raina-oe wwn.iiinnniioo
ing diverse data sources such as text, images, audio, and sensor data, modality fusion enables the capture of intri- cate relationships across modalities, which is especially erucial in fields like healthcare, where accurate decision- making relies on multimodal insights (I (Liang et al., 2022b; Kline et al., 2022; Teoh et al., 2024). Although recent ad- vancements in neural architectures, such as transformers (Vaswani et al., 2017; Tsai et al., 2019) and sparse mixture- of-experts (Shazeer al., 2017: Fedus et al., 2022; Jin et al., 2024), have significantly improved the modeling of modality interactions, important yet underexplored area is the systematic understanding of how modalities influence one another-whete mentary, or even confiicing information (Baltrušaitis et al., 2019; Liang et al., 2022b; 2023).
they provide complementary, supple-
Understanding modality interaction essential for advanc- ing multimodal machine learning (Baltrušaitis al., 2019; Liang al., 2022b). An ieformation-theorette framework called Partial Information Decomposition (PID) (Wollstadt et al., 2023; Liang et al., 2023) offers a theoretical foun- dation for understanding modality interactions. PID de- composes information into four distinct types: uniqueness for the first modality (information specific to modality 1), uniqueness for the second modality (information specific to modality 2), synergy (emergent information arising from the combination of two modalities), and redundancy (shared information across two modalities).
1. Introduction
A core challenge in multimodal learning is modality fu- sion-the integration of information from multiple modali- ties to improve predictive performance (Baltrušaitis et al., 2019; Barnum et al., 2020; Lv et al., 2021). By leverag-
UUniveritty of Pennsylvania, PA, USA University of North Carolina at Chapel Hill, NC. USA University of Science and Technology of China, Anhui, China. Correspondence to: Qi Long <qlong upenn.edu>> edu>, Tianlong Chen <tianlong@es udu.. Jiayi Xin <jjayixin @ wewssessuuupeee weaas.upenn.dd....
Figure/ illustrates the importance of carefully modeling different types of multimodal interactions. For instance, the unique information provided by the image modality
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Interpretable Multimodal Interaction-aware Mixture-of-Expers
(ming contributes predicting the Horror genre through distinct visual cues absent in the language modality (Mangg while the unique information from the language modality offers critical textual context for identifying the Romance genre. Redundant information refers to shared informa- tion present in both modalities, such as recognizing the Fantasy genre through the blurry figure in the poster and mentioning a "sorcerer" in the plot. Accurately classify- ing the movie Drama, however, requires modeling syn- ergistic information between the two modalities: visual elements such as clothing and facial expressions in Ming- complement the narrative details from Maagg. From this example, systematic modeling of multimodal interactions is needed to make accurate predictions.
nuanced understanding of multimodal data.
We enhance interpretability by providing both sample- level and dataset-level insights into model decisions, offering a deeper understanding of how interaction experts contribute to predictions.
I'MOE is highly flexible and can be seamlessly inte- grated with existing modality fusion methods, demon- strating its versatility in improving vanilla multimodal fusion backbones.
* Extensive experiments on five diverse real-world mul- timodal datasets validate the efficacy of MMM,, show- casing significant performance improvements (up to 5.5% in accuracy) and interpretability benefits over vanilla modality fusion methods.
While the PID framework provides valuable theoretical in- sights into the proportions of different modality interactions within a dataset, its practical application is limited, lacking integration into end-to-end and interpretable deep learn- ing frameworks. Most existing multimodal fusion methods do not explicitly model multimodal interactions (Liu et al., 2018; Tsai et al., 2019; Xue & Marculescu, 2023). Notable efforts to address this gap, such as (Wörtwein et al., 2022: Yu et al., 2024; Dufumier et al., 2024), exhibit key limita- tions: they either focus exclusively on pairwise modality interactions (Wörtwein et al., 2022), require separate esti- mates for each interaction type (Yu et al., 2024), or lack sufficient interpretability (Dufumier et al., 2024). The op- portunity to directly leverage PID for improving both task performance and model interpretability within multimodal fusion frameworks remains largely unexplored.
2. Related Work
Modality Interaction is theoretically grounded in the Par- tial Information Decomposition (PID) framework (Liang et al., 2023), which analyzes heterogeneous interactions but lacks an end-to-end learning framework. Prior works attempt to model interactions but are either restricted to spe- cific interaction types (Zhang et al., 2023; Kim et al.), fail to quantify interactions in the data (Wörtwein et al., 2024; Liang et al., 2024; Long et al., 2024; Dufumier et al., 2024), or are limited to only two modalities (Wörtwein et al., 2022;
Fan et al., 2024). Our approach bridges this gap by directly modeling and quantifying modality interactions within a unified MoE-based fusion architecture, enabling effective and interpretable multimodal learning.
In contrast to earlier works, we propose I'MOE, an end- to-end mixture-of-e (MoE) framework designed to enhance task performance while improving interpretability. Our approach incorporates separate parameters and weakly- supervised interaction losses, enabling the mixture of in- teraction experts to effectively model diverse interactions between modalities. To further enhance interpretability, we introduce re-weighting model that assigns importance scores to each interaction expert, providing insights into decision-making at both local (sample-level) and global (dataset-level) scales. I'MoE is backbone-age aggnostcc and can be seamlessly integrated with any modality fusion ap- proach. We evaluate the effectiveness of I'M on two medical datasets and three real-world multimodal datasets, demonstrating its ability to consistently improve perfor- mance while offering interpretable insights into the model's docision-making process for individual samples.
Multimodal Fusion integrates data from multiple sources to enhance prediction tasks. Existing methods often rely concatenating input modalities using off-the-sheef archi- tectures (Liu et al., 2018; Tsai et al., 2019; Xue & Mar- culescu, 2023; Shazeer et al., 2017; Fedus et al., 2022). Mfxture--ffFxxerts (MoF) offers a natural architecture for modeling interactions via expert specialization (Jacobs et al., 1991; Chen et al., 1999; Yuksel et al., 2012). Several re- cent works (Mustafa et al., 2022; Lin al., 2024; Yu et al., 2024) explore MoE for multimodal learning. Among them, only MMoE (Yu et al., 2024) explicitly models different types of modality interactions by using a mixture of interac- tion experts on sentiment analysis. However, MMoE treats modality interaction modeling as a preprocessing step rather than integrating it into an end-to-end learning framework, limiting fiexibility and interpretability.
Our contributions are summarized as follows:
Multimodal Interpretation has gained traction re- searchers seek to explain makiingnn in multimodal AI systems. Prior studies either focus on isolating the effect of individual modalities while overlooking inter-modal in- teractions (Ismail et al., 2022; Ghosh et al., 2023; Swamy et al., 2024b), provide human- intereiaterrretable rationales but
We introduce I'MOE, a novel mixture-of-expert framework designed to explicitly model diverse modal- ity interactions through specialized parameters and weakly-supervisee interaction losses, enabling a more
2

---

Interpretable Multimodal Interaction-aware Mixture-of-Expert
fail to quantify interaction contributions (Park et al., 2018; Zadeh et al., 2018 8; Dominici et al., 2023), or lack explicit categorization of interaction types (Tsai et al., 2020; Chefer et al., 2021; Lyu et al., 2022; Liang et al., 2022a; Wenderoth et al., 2024). As no prior work has explored interpretation from a modality interaction perspective, our contribution to systematically quantify multimodal interactions while maintaining interpretability.
modality inputs. During inference, single forward pass is performed using the complete modality input. The final output is weighted sum of the interaction expert prediction with the weights produced by the re-weighting model (Equa- tion 1). We provide a detailed explanation of I with two input modalities in Section 3.3, describe its extension to a higher number of modalities in Section 3.4, and ex- plain how obtain multimodal interaction interpretation in Section 3.5.
3. Interpretable Multimodal Interaction-aware Mixture-of-Experts
3.3. MoM with Two Input Modalities
3.3.1. I IoM ARCHITECTURE
3.1. Preliminary and Notation
Figure 2(b) illustrates the I I'MOz architecture for modeling different types of modality interactions in two input modal- ities. We employ MoE comprising four fusion models, referred to as interaction experts: Funil, F Funi2, F Fsyu. and F Fred. Each interaction expert specializes in capturing a specific type of interaction: Funil models the unique infor- mation contained in modality mj; Funi2 models the unique information contained in modality ma; Fayu captures the synergistic information between m, and m2; and F Frod mod- cls the redundant information between m, and m2.
Problem Setup. Let M {m1, , m, denote set of n input data modalities, and let y represent the target variable for given task. For classification tasks, y is expressed as a one-hot encoded vector corresponding to the class label. For regression tasks, y is a real-valued scalar. The objective is twofold: (1) to improve the performance of predicting the ground truth target y by effectively modeling the interactions between modalities in M, and (2) to provide meaningful interpretations of these multimodal interactions.
Vanilla multimodal fusion (Figure 2(a)) utilizes modality- specific encoders& & = {E1, Eg.... En to process M and obtain latent embeddings [ = {e1, e2, j. where each embedding is computed as c; = E, (m,) for E (1..... n). We define the fusion method as F, which operates on the set of latent embeddings L and produces a fused embedding x, expressed as: F(C)=
Each interaction expert processes the latent embeddings of the two modalities, e₁ and 02, and produces fused em- beddings, represented as x; = (i where i E [unil, uni2, syn, red}. These fused embeddings are then passed through a prediction head within each interaction expert, generating predictions for the corresponding interac- tion type as ý: (x;),, where i E {unil, uni2, syn, red}. To combine the predictions from the four interaction ex- perts, we introduce a re-weighting model W, which as- signs importance scores to the predictions of each expert. The model W takes the latent embeddings e and ee as inputs and outputs a set of soft weights: W(e1, e2) = [Wanil, Wani2, Wayn Wred]. The final prediction is obtained by combining the predictions from all experts using these weights, expressed as:
A prediction head H maps the fused embedding to the final prediction, expressed as: H(x)= = ý. However, this naive modality fusion approach does not explicitly account for the heterogeneous interac- tions present between M.
3.2. Algorithm Overview of I'MOE Framework
I'MoE is mixture of interaction experts, where each ex- pert specializes in modeling a specific type of multimodal interaction. The predictions from individual interaction ex- perts are weighted by a re-weighting model to produce the final prediction. During the training phase, we first per- form forward pass using the intact input of all modalities to estimate the multimodal prediction. Next, additional for- ward passes are conducted, where one modality is replaced by random vector in each pass. These perturbed inputs serve as weak supervision signals to help train the inter- action experts to specialize different types of modality interactions. We designed dual-objective loss, encourag- ing the interaction experts to specialize effectively without degrading task performance. The task loss is calculated us- ing the re- re-weighted output from the interaction experts with the complete modality input, while the interaction loss is computed from the outputs generated with the perturbed
ÿ =
Σ Wi yi, iE {unill uni2, red].. (1)
3.3.2. I2MGE LEARNING OBJECTIVE
The loss function consists of two components. The first component is the rask loss, which encourages the predictions of I ý, to closely match the ground truth target The second component, termed the interaction loss, ensures that the initially identical fusion models within I'MoE specialize into interaction experts by capturing diverse interactions in the dataset.
Following Yu et al. (2024), we characterize interaction types by comparing unimodal and multimodal predictions: pre- dictions made using only the first modality (y1), predictions
3

---

Interpretable Multimodal Interaction-aware Mixture-of-Expert
No Interaction Modeling
Syngy
Final Prediction
Model Diverse Interactions
No Interpretation
Logi
Provide Interpretation
Uni-1
Rdncy
Uni-2
Final Prediction
Genres
Logits
Uni-1 Expert
Uni-2 Expert
Interaction
Syngy Expert
Rdney Expert
Prediction
Prediction
Weights
Prediction
Prediction
Genres
Logits
Logits
Logits
Logits
Prediction Head
Genres
Genres
Experts
Genres
Genres
Prediction Head
Prediction Head
Prediction Head
Prediction Head
MLP
Fusion Model
Fusion Model
Fusion Model
Fusion Model
Fusion Model
Random
mi Encoder mg Encoder
mi Encoder mg Encoder
(a)
Vanilla Fusion
(b) Ours: I
Figure Comparison between vanilla modality fusion and MMM in the case of movic genre classification with input modalities. Left: Existing modality fusion approaches typically use the same parameters to model all types of interactions between the two modalities. Right: In contrast, we design mixture-of ex-eeess framework that employs four different interaction experts and re-weighting model explicitly capture heterogeneous interactions between the two input modalities.
made using only the second modality (y2), and predictions made using both modalities (y12). For interactions empha- sizing the uniqueness of the first modality, the relationships are defined as y12 = Yı and Y12 F y2- Similarly, for inter- actions emphasizing the uniqueness of the second modality, we have Y12 Y2 and Y12 # y1- For synergistic inter- actions, the condition is Y12 # Y1 and Y12 * y2. For redundant interactions, the relationship is Y12 = y1 = y2-
treated as positive example, while the output with modal- ity I masked, ÿ-1, is treated as negative example. The objective is to encourage ý12 to be maximally similar to ý and maximally different from ý-1. since Funil models the uniqueness information presented m, For the Funi2. y y-2 is treated as a negative example, while y-1 treated as a positive example. Here, the objective to encourage Y12 be maximally similar to y and maximally different from ý --2. since Fumi2 models the uniqueness information presented in m2. For the F F and y-2 are both treated as negative examples. The objective is to ensure that ý is maximally different from both ý-2 and ý-1- capturing interactions that require the combination both modalities. For the Fred, 8-1, and ÿ-2 2 are treated as positive examples. The goal is encourage ÿ12. ÿ-2, and y-1 be as simi- lar possible, modeling information shared between the modalities. We discuss the connection between the proposed interaction loss and the PID formulation in Appendix A and present empirical evidence supporting the design choice of random vector masking.in Appendix B.
To approximate the interaction loss, we simulate the uni- modal scenario by replacing one of the modalities with a random vector. For each interaction expert, a unimodal prediction using only the first modality can obtained by replacing the latent embedding of the second modality with random vector r, represented as:
-- = H. (E,
(2)
where i E {unil, uni2, syn, red]. Similarly, a unimodal prediction using only the second modality can be generated by replacing the latent embedding of the first modality with r. expressed as:
3.4. Extend I M'M to Higher Number of Modalities
-ー1 (,(,,,(,
Increase Uniqueness Interaction Experts. To ex- tend T to support more than two input modalities, we increase the number of interaction experts to the |M] 2. Instead of a combinatorial explosion in the number of inter- action experts, the number of input modalities grows, define m uniqueness interaction experts, one for each input
where i E {unil, uni2, red).
We designed general framework to approximate different types of modality interactions. In all cases, the output using the complete multimodal input, Y12, serves as the anchor. For the F Fumil, the output with modality masked, y-2, is

---

Interpretable Multimodal Interaction-aware Mixture-of-Expets
Algorithm 1 Training and Inference of I'MOE
Appendix D.
Require: Modalities XI...., , Xn, label T
Require: Modality-specific Encoders (Enc;
3.5. Local and Global Interpretation from I I'MoE
Require: Experts (.F.)an, reweighting module W
Require: Expert loss functions [InleractionLosss]]] 1==1 // Training with masked modality input
Local interpretation provides insight into the extent to which different interactions contribute to the final prediction for each individual sample, while global interpretation high- lights the average trends of interaction importance across the entire dataset. For I I'MOE, decisions are made locally for each specific input sample by analyzing the prediction, y.. from each interaction expert Fs, and the importance coefficients, W,, assigned by the MLP-based re-weighting model W. Global interpretation for IoM can be achieved by calculating the statisties of the importance weights w; assigned to each interaction expert across all samples in the test set, thereby capturing the overall trends in feature contributions.
1: Encode modalities: Z; + (X., for = 1,...,7
2: for i == to E do
3: [ ................................ 1 F.mmlli (Z1, (Zi.... Znn
4: Lime + Interaction (0))) .1; (1m))
5: end for
6: [UII,... WE + W(ZI..... Zn)
7: Ú + ΣE Wz. W..9. Di
8: Linsk c ((..T) T)
9: Trotll f- T.rack + Age ΣE Limt
10: Update model parameters to minimize Lintal
11: procedure INFERENCE
12:
Encode modalities: Z, (X.) for i I....,
13:
0:00 + Fi(ZI,... ...,,...., Z..) for = 1, I,..., E
14:
[111 WE| WZ........ Zm)
15:
Ú+ E/=1 ++EE1 Wi 0:
4. Experiment Setup
16:
Store [3i.). w, and prediction y
17:
end procedure
Data Collection and Datasets. We evaluate our method five multimodal datasets, using all available modalities while discarding samples with missing data. Two Medical Multimodal Datasets: & ADNI (Weiner et al., 2010; 2017) consists of 2,380 samples for Alzheimer's Disease classification (Dementia, Cognitively Normal, or Mild Cognitive Impairment). It includes four modali- ties: Image (I), Genetic (G), Clinical (C), and Biospec- imen (B). MIMIC-IV (Johnson et al., 2023) is critical care dataset with 9,003 patient records for one- year mortality prediction (binary classification), utilizing three modalities: Lab (C), Notes (N), and Code (C). Three General Multimodal Datasets: >> IMDB (Arevalo et al., 2017) includes 25, 959 movies for multi-label genre classification across 23 genres, leveraging Image (I) and Language (C) modalities. MOSI (Zadeh et al., 2016) com- prises 2,199 annotated YouTube clips for sentiment analysis (regression with scores E [-3,3] and then map to binary clas- sification), incorporating Vision (V), Audio (A), and Text (T) modalities. & ENRICO (Leiva et al., 2020) contains 1,460 Android app screens for UI design classification into 20 categories, featuring two modalities: Screenshot (S) and Wireframe (W). Detailed dataset preprocessing is provided in Appendix E.
modality, along with a single synergy expert and a single redundancy expert. Each uniqueness expert, Funi,; is re- sponsible for capturing the unique information specific to its corresponding modality, m; E M, where i E {1..... m). The synergy expert, F syyn focuses on modeling global syn- ergistic interactions across all modalities, while the redun- dancy expert, F Fodd, captures globally redundant information shared among the modalities.
Modify Interaction loss. For uniqueness expert i, we con- sider the output of the complete modality as the anchor. The masked modality serves as negative example, while all other perturbed inputs are treated as positive examples. This is because the unique information of modality i is lost when the modality embedding is replaced by random vectors. For the synergy interaction loss, we treat all the output of the perturbed modality as negative examples, as input modality perturbations damage the synergistic information. For the redundancy interaction loss, we consider the output of the perturbed modality as positive example because redun- dant information remains available even when one modality is masked. For classification tasks, we employed Triplet Margin Loss to model uniqueness interactions. For synergy and redundancy interactions, we utilized Cosine Similarity to capture the relationships between modality outputs. For regression tasks, we used the Mean Squared Error (MSE) Loss to measure differences in predictions.
Modality-Speeiffc Encoders and Prediction Heads. The primary objective of our experiments is to evaluate whether the proposed mixture-of-experts framework im- proves modality fusion. To ensure a fair comparison, we control for variations in mpdelity-specifie encoders (E) and prediction models (H) by using the same E and H for both vanilla multimodal fusion and I I'MOE. For further details on the encoder and classification head configurations, please refer to Appendix F.
I'M Algorithm and Complete Objective. We present the training and inference pipeline of I'M in Algorithm 1. The complete learning objective is provided in Appendix C. We analyze computational overhead and scalability in
5

---

Interpretable Multimodal Interaction-aware Mixture-of-Expers
Baseline Fusion Methods. To validate the effectiveness of M'M in enhancing multimodal learning, com- pare it to various widely used fusion techniques. We be- gin with fundamental approaches, including early fusion (EF) (Baltrušaitis et al., 2019), late fusion (L.F (Baltrušaitis et al., 2019), low-rank multimodal fusion (LRMF) (Liu et al., 2018), and multimodal transformers (MulT) (Tsai et al., 2019). We then implement more advanced fu- sion methods, including interpretable conditional compu- tation (InterpretCC) (Swamy et al., 2024a), the Switch Transformer (SwitchGate) (Fedus et al., 2022), and sparse mixture-of-experts (MoE++) (Jin et al., 2024). In both SwitchGate and MoE++, the MLP layer in MulT is replaced with a sparse MoE layer that incorporates the respective routing function.
On the MIMIC dataset, I'MoE achieves notable AUROC improvements of 4.43% when combined with Interpret-CC. highlighting its ability to capture complex interaction in multimodal patient data. However, accuracy decreases (- 0.56% to -1 -11.11221111 1.82%) are observed, which can be attributed to dataset imbalance. In such cases, the model becomes less overfitted to the majority class, leading to a decrease in accuracy but a corresponding increase in AUROC, re- flecting improved performance in distinguishing between classes overall. @ I"MoE consistently enhances multimodal learning, achieving improvements in Mícro F1 on IMDB (2.45%), sentiment analysis accuracy on MOSI (4.76%), and design classification accuracy on ENRICO (5.14%) when integrated with MoE++ and SwitchGate. b RRsuults with different fusion backbones emphasize the generaliz- ability and effectiveness of I
The dataset is partitioned into training, validation, and testing sets, with 70% allocated for training, 15% for validation, and the remaining 15% for testing. Each experiment is run three times with different random seeds and the results are averaged. To ensure a fair comparison with other baselines, we utilize the optimal hyperparameter settings provided in the original studies. If a dataset does not have reported optimal parameters, we perform grid search over the key hyperparameters of the baseline methods. The re-weighting model (W) is implemented as a multilayer perceptron (MLP). For a detailed description of the hyper- parameter settings, we refer the reader to Appendix G.
Implementations.
5.3. M' Offers Local Interpretation
To illustrate the interpretability provided by I'M on the individual sample level, we present a qualitative example from the IMDB test set where I'MoE-MulT makes cor- rect prediction (Figure 3). This example showcases how different interaction experts contribute to the final prediction through visualized logits and assigned weights, offering a clear decomposition of the decision decision-making process. The ground truth genres of this movie include Animat ion. In Figure 3()), the logits produced by each interaction expert are shown. Notably, the uniqueness expert for the image modality and the redundancy expert generate positive log- its, while the synergy expert yields a negative logit. This aligns with the visual content of the image, which features cartoon characters uniquely contributing to the prediction in Figure 3(d). Figure 3(b) depicts the weights assigned by the reweighting mechanism. Higher weights are given to the uniqueness expert for the image modality and the redundancy expert. As shown in Figure 3(c), the final weighted logits for the Animation genre become positive, enabling the correct prediction. This example demonstrates how I'MoE leverages different interaction patterns to make accurate predictions. We provide human evaluation of lo- cal interpretation in Appendix H and additional qualitative examples in Appendix I.
5. Performance and Interpretability of I'MOE
5.1. I I'MOE Demonstrates Superior Task Performance
In Table 1, compared the performance of I I'M com- bining with MulT (I'MoE- Mull) with other vanilla fu- sion methods across five datasets: O Compared to vanilla MulT, I yields a significant accuracy improvement of 5.5% for ADNI and 3% for MOSI, demonstrating its ability to enhance the performance of existing transformers. @ Across all datasets, I'MoE outperforms advanced base- lines such SwitchGate and MoE++, with notable gain of 2.5% accuracy, 1.5% AUROC ADNI, and 1.4% im- provement in Macro F1 for IMDB. & These results illustrate the benefit of I'MoE in tackling the challenges of modality interaction to achieve superior task performance.
5.4. I MMM Enables Global Interpretation
5.2. Generalization Across Different Fusion Methods
We analyze the weight assigned by the reweighting model to each interaction expert across all test samples. Figure 4 illustrates the weight variation across datasets, offering in- sights into dataset-level interaction patterns. The reweight- ing model demonstrates the ability to adaptively assign dis- tinct weights to interaction experts, reflecting its capacity to capture dataset-specific nuances. In the ADNI dataset, weights are relatively uniform, with a subtle bias toward cer- tain experts, indicating balanced contributions from all inter-
To evaluate the generalizability of I'MoE across various fusion backbones, we integrate it with three fusion archi- tectures, including MoE++, SwitchGate, and Interpret-CC, and assess the combined models on all datasets (Table 2): O For the ADNI dataset, T yields significant perfor- mance gains, with up to 5.23% improvement in accuracy and 2.12% in AUROC when combined with SwitchGate.
6

---

Interpretable Multimodal Interaction-aware Mixture-of-Expers
Table Comparison of Accuracy, AUROC, and F1 scores across different fusion methods and datasets. The upper panel lists vanilla fusion methods, while the last row presents the proposed I'MoE framework combined with MulT fusion method.
Dataset
ADNI
MIMIC
IMDB
MOSI
ENRICO
Metrics
Accuracy
AUROC
Accuracy
AUROC
Micro F1
Macro F1
Accuracy
Accuracy
EF
=0.01
+1.6
67.63+1166
67.75=0.93
56,10=0.27
41.12++1.0
72.16+0.60
42.35=00.81
LF
50.79
18.70
67.11 11.06
67.58 80.88
56.22_0.03
45.27:10.4
44.20 11.11
LRMF
50.79
69.37.13.15
70.17
65,45
56.22_o.0s
45.27
76.63
46. 1.01
InterpreICC
54.53+3344
72.34
61.93=2.63
58.00=0.25
48.68+0.11
75.85++0.07
1.56
SwitchGate
62.28-11.17
79.70
70.98:t0.88
68.26
55.92=0.07
47.33-20.47
72.35:+0:27
43.95
MoE++
58.08+2..2
75.18 11.95
72.51::2.09
68.50=2.15
58.15
50.49
70.85+t0.83
47.83
MulT
59.57=00.66
77.21 +0.51
72.42++2.53
68. 68.79=3.31
59.68
19.68=0.19
51.41+0.01
68.80 68.80+0.78
47.37+1.82
I'MoE-MMll
65.08=1.52
81.09+0.02
69.78+0..9
68.81=0.99
61.00=0.11
52.38+00448
71.91+2.20
48.22
model to address variance among individual patients. For the IMDB dataset, the weight variation is less pronounced compared MIMIC, aligning with its more homogeneous characteristics. The MOSI dataset shows evenly distributed weights, reflecting equal contributions from all interaction experts. Finally, the ENRICO dataset demonstrates a con- centrated weight distribution with dominant experts for the screenshot modality.
Table Comparison of metrics across datasets using different fu- sion methods for I'M Performance improvements are indicated in blue, and decreases are indicated in red.
Dataset
SwitchGate
InterpretCC
MoB++
ADNI
Accuracy
67.51 (5.23)
56.02 (1.49)
59.01 (0,93)
AUROC
81.82 (2)
73.36 (1.18)
75.69 (0.51)
Accuracy AUROC
70.42 (-0.56)
69.85 (-2.49)
60.69 (-11.82)
MIMIC
69.08 (0.82)
66.36 (4.43)
69.15 (0.65)
IMDB
Micro F1
57.43 57.45 (1.51)
58.32 (0.32)
60.60 (2.45)
Macro
(0.44)
49.21 (0.53)
50.73 (0.24)
6. In-depth Analysis of I'MoE
MOSI
Accuracy
73.86 (1.51)
76.14 (0.29)
75.61 (4.76)
ENRICO
Accuracy
49.09 (5.14)
49.09 (1.49)
47.83 (0)
6.1. Accuracy of Individual Experts
To further analyze the effectiveness of I'MOE, we compare its task performance against individual interaction experts across different datasets, shown in Figure 5. The re- sults highlight the following insights: Across all datasets, the overall performance of I line) consistently surpasses that of any individual interac- tion expert expert, with performance gains of 2. 2.2%, 1.3%. 7.1%, 0.6%, and 2.6% for ADNI, MIMIC, IMDB, MOSI, and ENRICO, respectively. This underscores the ad- vantage of leveraging a mixture-of-expere approach over single-expert methods. The proposed method exhibits the largest performance gains in datasets with high inter- action importance distribution variability, such as MIMIC and ENRICO. While for more uniform datasets like MOSI, the performance of individual experts is closer that of the overall model, indicating that the ensemble effeet may be less pronounced in these cases. > This suggests that the fusion of multiple experts becomes particularly beneficial in datasets with complex and heterogeneous multimodal interactions.
Ecarrt
3.5
Exprett Weighting
5.0
2.5
0.0
9:2
(red horizontal
5.0
Comedy
Adviotto
fontasy
Family
Artrustion
01994
(a)
"Witcn e groen ogre" numed Stuck swamped with ail soeta of tairytale creatures the scheming Land Farcuaad, sets very donkey side 'fenuusiia" Tlanpooul give Stock his swamp book...
3.0
2.5-
2.8
2--
Camedy
Adkestore
Festany
Fanddy
Animation
SHReK
(c)
Figure 3. Qualitative example of local interpretation on the IMDB
dataset provided by I MulT. Ground truth labels Comedy, Adventure, F Fantasy, Family ly, and Animatis ion. (b) Weighting assigned by the reweighting model. (c) Contribution of each in- teraction expert to the final weighted logit. (d) Raw image and language modalities used for prediction.
(a) Logits output by different interaction experts.
6.2. Interaction Expert Diversification
To analyze the diversification of different interaction ex- perts, we evaluate the ratio of expert agreement to disagree- ment and assess the corresponding accuracy of I²M A
action experts to the model's performance. Conversely, the MIMIC dataset displays pronounced variability in weight assignments, emphasizing I MMM. 's reliance on reweighting
7

---

Interpretable Multimodal Interaction-aware Mixture-of-Expere
Weills Burrrautinnn
4
Wegth Distituuton
Wanght Distltution
0.200
Weath OHstr Buntin
20203
0.
00000
03795
0.6
2
1925
Uni1
Synsy
Adocyy
059-
Sy1eY Rdrcy
une1
ueb2
syaa
ndncy
Unii
0n97
Uni-
Syngy
Rdncs
Unii
Symas
MOSI
Figure 4. Visualization of interaction weight distributions across all test samples for five datasets. Black bars indicate the median, mean, and extreme values.
8.79
65 55%
3.00
0.725
6.63
0.700
40.2%%
6.43
Uni1
Umi:2
Uni3
Uni-4
Syngy
Rdacy
6.400
Unk3
Unl-2
Uni-3
Symgy
Rancy
unl-1
Unl3
Syngs
Unit1
Unl-3
Uni-3
Syngy
Rdncy
Uns-3
Synoy
Edssy
Figure 5. Comparison hetween the task performance of MMRR-Mu 1 (red horizontal line) and each individual interaction expert across
different datasets.
high proportion of disagreement among experts indicates greater diversity, which is essential for capturing distinct interaction patterns. Furthermore, when experts disagree, we expect I'MOE to still maintain high level of accuracy, demonstrating its ability leverage diverse expert opinions effectively.
Table 3. Interaction experts agreement analysis on test set for all datasets. "Disagree" or "Agree" indicates whether all expert pre-
diction is the same. / ("Correct") or X ("Incorrect") refers to the correctness of I' I'MoE's prediction.
% of Data
ADNI
MIMIC
IMDB
MOSI
ENRICO
Disagree. /
48.74
63.51
15.85
37.80
46.85
Table 3 presents the proportion of cases where experts disagree or agree, along with the corresponding accu- racy of I'MoE across five datasets: MIMIC datasets, the proportion of disagreement among experts is relatively high (81% % and 85%, respectively), while T2M achieves correct predictions in substan- tial portion of these cases. @ On the IMDB and EN- RICO datasets, the proportion of disagreement is very high (99.99% and 98%), yet M'M achieves significantly fewer correct predictions when experts disagree (15.85% % Correct, 84.14% Wrong and 46.85% Correct, 51.44% % Wrong). @ For the MOSI dataset, the disagreement proportions (59%) highlight moderate diversity among experts. No- tably, I'M maintains relatively high accuracy when perts disagree (37.80% Correct for MOSI. These results indicate a potential need for better handling of disagree- ment in complex datasets, and how dataset characteristics influence the diversification and effectiveness interaction experts.
Disagree,
32.40
21.39
84.14
21.97
$1.44
Agree, /
16.34
6.37
0.00
34.11
1.37
Agree, x
2.52
8.73
0.01
0.34
For ADNI and
interaction loss is removed, resulting in a simple mixture-of- experts model without explicit encouragement for learn- ing diverse multimodal interaction among experts. (2) Latent SeeeettConrrrttte directly to the latent embeddings produced by each interac- tion expert instead of their outputs. (3) Simple-We igh: The MI M----a reweighting model is replaced by a shared, learnable global weight that does not adapt to individual samples. (4) Less-Porward Perturbation reduced by randomly masking only two modalities per sample instead of perturbing all modalities.
The interaction loss is applied
(5) ryynergy-Reunddnee Only synergy and redundancy experts are included, omitting uniqueness experts.
From Table 7: No- Mo-Interrattoo: Removing the in- teraction loss results in significant performance degradation across all datasets (c.g.. -6.35% accuracy on ADNI and -3.99% AUROC), confirming that explicitly encouraging diversity among experts is crucial for capturing complemen- tary modality interactions. @ Iatent-Contrass ive: Applying the interaction loss to latent embeddings instead of expert outputs causes a noticeable performance drop (e.g.. -6.9 1% accuracy on ADNI). This highlights the importance of applying the interaction loss at the output level to di-
7. Ablation Studies
To validate the effectiveness of I I'MOE, perform exten- sive ablation studies by systematically removing or mod- ifying key components of the model. Each variant is de- signed to assess the contribution of specific design choices to the overall performance: (1) No- Int intteracionn iin: The
8

---

Interpretable Multimodal Interaction-awary -ixpurs-o--Expert
Acknowledgements
Table Ablation study results on three datasets (ADNI, MOSI, ENRICO), showing the impact of removing or modifying key components of I'MoE. Each row corresponds a variant the model with a specifie component ablated. Performance drops (in red) are reported relative to the full model.
This work was supported part by NIH grants, RFIAG063481, ROLAG071174, and U01CA274576. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. We would like to thank the anonymous reviewers for their in- sightful feedback.
Dataset
ADNI
MOSI
ENRICO
Ablation
Accuracy
AUROC
Accuracy
Accuracy
(1)
(-6.3))
77.10 (-3.99)
69.49 (-2.42)
47.63 (-0.59)
(2)
58.17 (-6.91)
75.40 (-5.69)
69.68 (-2.23)
47.50 (-0.72)
Impact Statement
59.29 (-5.79)
74.55 (-6.54)
68.46 (-3.45)
47.49 (-0.73)
(4)
59.76 (-5.32)
76.81 (-4.28)
69.89 (-2.02)
46.92 (-1.30)
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
(5)
56.77 (-8.31)
74.30 (-6.79)
70.12 (-1.79)
47.49 (-0.73)
rectly guide expert specialization. @ Simple-Weight:
References
Replacing the sample-specific reweighting model with a global weight reduces performance (e.g.. -5.32% accuracy on ADNI and -1.00% ENRICO), demonstrating the value of adaptive reweighting for leveraging diverse expert out- puts effectively. O Less-Forwarde Reducing modality perturbations leads to reduced accuracy (e.g., -5.79% on ADNI and -3.45% on MOSI). This suggests that generat- ing sufficient negative examples through extensive pertur- bation is essential for capturing diverse interactions. Synergy-Redundancyy Limiting the experts to only synergy and redundancy results in the largest performance drop (e.g., -8.31% % accuracy on ADNI). This emphasizes the importance of uniqueness experts in modeling compre- hensive modality interactions. & The ablation study demon- strates that each component of IMM is vital for its success.
Arevalo, J., Solorio, T., Montes-y Gómez, M., and González, F. A. Gated multimodal units for information fusion. arXiv preprint /rXiv:/702.0//3 2017.
Baltrušaitis, T., Ahuja, C., and Morency, L.-P. Multimodal machine learning: A survey and taxonomy. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 41(2):423 44444233-443, 2019, URL https: /ttp:s///eeeeeeoreee//// eeee /eeee.rrg/eeee.ee......ee..eeeee www.oorg/ooooooooo http///cceeeette///// 88669866.
Barnum, G., Talukder, S., and Yue, Y. On the benefits of early fusion in multimodal representation learning, 2020, URL https:tp///// http/vxxivvvv./// orr//iooooooo/oooooooo..oooeee abs /20p////bbb/111..1111.. 07191.
Bertschinger, N., Rauh, J., Olbrich, E., Jost, J., and Ay, N. Quantifying unique information. Entropy, 16(4):2161- 2183, 2014.
8. Conclusion
We introduced I'MOE, a novel MoE framework designed to enhance multimodal task performance and interpretabil- ity by explicitly capturing heterogeneous modality interac- tions. Extensive experiments on five real-world datasets demonstrated the superiority of IoMoE in improving perfor- mance across diverse multimodal scenarios. By leveraging mixture-of-exxpets design with adaptive reweighting and specialized interaction losses, our approach systematically models and quantifies modality interactions. Additionally, we analyzed the distribution of interaction weights, pro- viding meaningful insights at both the sample and dataset levels, which enhances the interpretability of the model's predictions. We also conducted ablation studies to evalu- ate the impact of each design component and demonstrated the flexibility of I Io to generalize across various fusion methods. For future work, alternative forms of interaction loss could be explored to further improve performance. Ad- ditionally, integrating feature attribution methods to analyze the contributions of individual features within interaction experts can offer deeper interpretable insights.
Chefer, H., Gur, S., and Wolf, L. Generic attention-model explainability for interpreting bi-modal and encoder- decoder transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 397- 406, 2021.
Chen, K., Xu, L., and Chi, H. Improved learning algorithms for mixture experts in multiclass classification. Neural networks, 12(9):1229-122 1252, 1999.
Dominici, G., Barbiero, P., Magister, L. C., Liò, P., and Simidjievski. N. Shares: Shared concept space for explainable multimodal learning. arXiv preprint arXiv:2307.00316, 2023.
Doshi, J., Erus, G., Ou, Y., Resnick, S. M., Gur, R. C., Gur, R. E., Satterthwaite, T. D., Furth, S., Davatzikos, C., Initiative, A. N., et al. Muse: Multi-atlas region segmen- tation utilizing ensembles of registration algorithms and parameters, and locally optimal atlas selection. Neuroim- age, 127:186-1881 195, 2016.

---

Interpretable Multimodal Interaction-aware Mixture-of-Expeer Expertss
Dufumier, B., Castillo-Navarro, J., Tuia, D., and Thiran, J.PP. What to align in multimodal contrastive learning? arXiv preprint arXiv:2409.07402, 2024.
Forty-first International Conference on Machine Learn- ing.
Kline, A., Wang, H., Li, Y., Dennis, S., Hutch, M., Xu, Z.., Wang, F., Cheng. F., and Luo, Y. Multimodal machine learning in precision health: A scoping review. npj Digital Medicine, 5(1):171, 2022.
Esmaeilzadeh, S., Belivanis, D. I., Pohl, K. M., and Adeli, E. End-to-end alzheimer's 's disease diagnosis and biomarker identification. In Machine Learning in Medical Imaging: 9th International Workshop, MLMI 2018, Held in Con- junction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings 9, pp. 337-345. Springer, 2018.
Leiva, L. A., Hota, A., and Oulasvirta, A. Enrico: A dataset for topic modeling of mobile ui designs. In 22nd Interna- tional Conference Human- Human-Computer Interaction with Mobile Devices and Services, pp. 1-4, 2020.
Fan, C., Zhu, K., Tao, J., Yi, G., Xue, J., and Lv, Z. Multi- level contrastive learning: Hicrarchical alleviation of het- erogeneity in multimodal sentiment analysis. JEEE Trans- actions on Affective Computing, 2024.
Liang, P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., Chen, L. Y., Wu, P., Lee, M. A., Zhu, Y., et al. Multi- bench: Multiscale benchmarks for multimodal represen- tation learning. In Thirty-fifth Conference on Neural In- formation P Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling trillion parameter models with simple and ef- ficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
Liang, P. P., Lyu, Y., Chhablani, G., Jain, N., Deng, Z., Wang. X., Morency, L.-P., and Salakhutdinov, R. Multi- viz: Towards visualizing and understanding multimodal models. arXiv preprint arXiv:2207.0000 2022a.
Ghosh, S., Yu, K., Arabshahi, F., and Batmanghelich, K. Dividing and conquering blackbox to mixture of inter- pretable models: route, interpret, repeat. In Proceedings of the... International Conference on Machine Learning. International Conference on Machine Learning, volume 202. pp. 11360, 2023.
Liang, P. P., Zadeh, A., and Morency, L.-P. Founda- tions and trends in multimodal machine learning: Prin- ciples, challenges, and open questions. arXiv preprint arXiv:2209.03440 2022b. URL https:/ /stps://arxiv.. org aa/aabb/22200 03430.
Goldberger, A. L., Amaral, L. A., Glass, L.,, Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., and Stanley, H. E. Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. circulation, 101(: e215 e220, 2000.
Liang, P. P., Cheng, Y., Fan, X., Ling, C. K., Nie, S., Chen, R., Deng, Z., Allen, N., Auerbach, R., Mah- mood, F., Salakhutdinov, R., and Moreney, L.-P. Quan- tifying & modeling multimodal interactions: An in- formation decomposition framework. arXiv preprint arXiv:2302. /2247, 2023. URL httts:://arxt org/
Ismail, A. A., Arik, S. Ö., Yoon, J., Taly, A., Feizi. S., and Pfister, T. Interpretable mixture of experts for structured data. arxiv preprint arXiv:2206.02100 2022.
Jacobs, R. A., Jordan, M. I,, Nowlan, S. J., and Hinton, G. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 87, 1991.
Liang, P. P., Deng. Z., Ma, M. Q.. Zou, J. Y., Morency, L.-P., and Salakhutdinov, R. Factorized contrastive learn- ing: Going beyond multi-view redundancy. Advances in Neural Information Processing Systems, 36, 2024.
Jin, P., Zhu, B., Yuan, I.,, and Yan, S. Moe++: Accelerat- ing mixture-of-experts methods with zero-computation experts. arXiv preprint arXiv:2410.07348, 2024.
Lin, X. V., Shrivastava, A., Luo, L., Iyer, S., Lewis, M., Ghosh, G., Zettlemoyer, L., and Aghajanyan, A. Moma: Efficient carly-fusion pre-training with mixture of modality-awar experts. arXiv preprint arXiv:2407.22 2407.217777 2024.
Johnson, A., Bulgarelli, L., Pollard, T., Gow, B., Moody, B., Horng, S., Celi, L.,, and Mark, R. Mimic-iv (version 3.1). physionet, 2024.
Liu, Z., Shen, Y., Lakshminarasimhan. V. B., Liang, P. P.P Zadeh, A., and Morency, I.-P. Efficient low-rank mul- timodal fusion with modality-specific factors. arXiv preprint arXiv: 00064, 2018
Johnson, A. F., Bulgarelli, L.,, Shen, L., Gayles, A., Sham- mout, A., Horng, S., Pollard, T. J., Hao, S., Moody, B., Gow, B., al. Mimic-iv, freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
Long, L., Cui, J., Zeng. P., Li, Y., Liu, Y., and Wang, Y. Mugi: Multi goatii-granulartty interactions of heterogeneous biomedical data for survival prediction. In International
Kim, C., van der Schaar, M., and Lee, C. Discovering fea- tures with synergistic interactions in multiple views. In
10

---

Interpretable Multimodal Interaction-aware Mixture-of-Expert
Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 490-500. Springer, 2024.
Tsai, Y.-H. H., Ma, M. Q. Yang, M., Salakhutdinov, R., and Moreney, L..P. Multimodal routing: Improving local and global interpretability of multimodal language analysis. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empir- ical Methods in Natural Language Processing, volume 2020, pp. 1823. NIH Public Access, 2020.
Lv, E., Chen, X., Huang, Y., Duan, L.,, and Lin, G. Progres- sive modality reinforcement for human multimodal emo- tion recognition from unaligned multimodal sequences. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 2554 2562, June 2021.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, I.., Gomez, A. N., Kaiser, L.., and Polosukhin, I. Attention is all you need. nips' 17. In Proceedings of the 31st Inter- national Conference on Neural Information Processing Systems December, pp. 6000-6010, 2017.
Lyu, Y., Liang, P. P.. Deng, Z., Salakhutdinov, R., and Morency, L.-P. Dime: Fine-graine Fine-grained interpretations of multimodal models via disentangled local explanations. In Proceedings of the 2022 AAAI/ACM Conference AI, Ethics, and Society, pp. 455-467, 2022.
Weiner, M. W., Aisen, P. S., Jack Jr, C. R.,, Jagust, W. J., Trojanowski, J. Q., Shaw, I.., Saykin, A. J., Morris, J. C., Cairns, N., Beckett, L. A., et al. The alzheimer's ' disease neuroimaging initiative: progress report and future plans. Alzheimer's & Dementia, 6(3):202-211 211, 2010.
Mustafa, B., Riquelme, C., Puigcerver, J., Jenatton, R., and Houlsby, N. Multimodal contrastive learning with limoe: the language-image mixture of experts. Advances in Neural Information Processing Systems, 35:9564-9576, 2022.
Weiner, M. W., Veitch, D. P., Aisen, P. S., Beckett, L. A., Cairns, N. J., Green, R. C., Harvey, D., Jack Jr, C. R., Jagust, W., Morris, J. C., et al. The alzheimer's 's disease neuroimaging initiative 3: Continued innovation for clini- cal trial improvement. Alzheimer's & Dementia, 13(5):
Park, D. H., Hendricks, L., A., Akata, 7., Rohrbach, A., Schiele, B., Darrell, T., and Rohrbach, M. Multimodal explanations: Justifying decisions and pointing to the evidence. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8779-8788, 2018.
561-571, 2017.
Wenderoth, L., Hemker, K., Simidjievski, N., and Jamnik, M. Measuring cross-modal interactions in multimodal models. arXiv preprint arXiv:2412.15828 2024.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint 1rXiv:1701.0633 2017.
Wibral, M., Priesemann, V., Kay, J. W., Lizier, J. T., and Phillips, W. A. Partial information decomposition as unified approach to the specification of neural goal functions. Brain and cognition, 112:25-38, 2017.
Swamy, V., Montariol, S., Blackwell, J., Frej, J. A., Jaggi, M.. and Käser, T. Interpretce: Intrinsic user-centric in- terpretability through global mixture of experts. arXiv preprint arXiv:2402.02933 2024a.
Williams, P. L. and Beer, R. D. Nonnegative decom- position of multivariate information. arXiv preprint ar04v:1004.25111 2010.
Swamy, V., Satayeva, M., Frej. J., Bossy, T., Vogels, T., Jaggi, M., Käser, T., and Hartley, M.-A. Multi- modn-mullimone multimodal, multi-task, interpretable modular networks. Advances in Neural Information Processing Systems, 36, 2024b.
Wollstadt, P., Schmitt, S., and Wibral, M. A rigorous information-theorete definition of redundancy and rele- vancy in feature selection based on (partial) information decomposition. Journal of Machine Learning Research, 24(131)::144, 2023.
Teoh, J. R., Dong, J., Zuo, X., Lai, K. W., Hasikin, K., and Wu, X. Advancing healthcare through multimodal data fusion: a comprehensive review of techniques and applications. PeerJ Computer Science, 10:e2298, 2024.
Wörtwein, T., Sheeber, L.,, Allen, N., Cohn, J., and Morency, L.-P. Beyond additive fusion: Learning non-additive multimodal interactions. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 4681- 1- 4696, 2022.
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., and Salakhutdinov, R. Multimodal transformer for unaligned multimodal language sequences. In Proceed- ings of the conference. Association for computational linguistics. Meeting, volume 2019, pp. 6558. NIH Public Access, 2019.
Wörtwein, T., Allen, N. B., Cohn, J. F., and Morency, L.- P. Smurf: Statistical modality uniqueness and redun- dancy factorization. In Proceedings of the 26th Interna- tional Conference on Multimodal Interaction, pp. 339 349, 2024.

---

Interpretable Multimodal Interaction-aware Mixture-of- Experts
Xue, Z. and Marculescu, R. Dynamic multimodal fu- sion, 2023. pttttpps//aaii..orggggoo 00102.
Yu, H., Qi, Z.,, Jang, L.,, Salakhutdinov, R., Morency, L.-P., and Liang, P. P. Mmoo: Enhancing multimodal models with mixtures of multimodal interaction experts. In Pro- ceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 10006-10030, 2024.
Yuksel, S. E., Wilson, J. N., and Gader, D. Twenty years of mixture of experts. IEEE Transactions on Neural Networks and Learning Systems, 23(8): 2012. doi: 111.TNNNS.201..2222...
Yun, S., Choi, I., Peng. Wu, Y., Bao, J., Zhang, Q., Xin, J., Long, Q., and Chen, T. Flex-moe: Modeling arbitrary modality combination via the flexible mixture-of-experts arXiv preprint wrxiv:2410.08244 2024.
Zadch, A., Zellers, R., Pincus, E., and Morency, L.-P. Mosi: multimodal corpus sentiment intensity and subjectiv- ity analysis in online opinion videos. arXiv preprint arXiv: 1600160066000 062.59 2016.
Zadeh, A. B., Liang, P., Poria, S., Cambria, E., and Morency, L.-P. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fu- sion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( Volume 1: Long Papers). pp. 2236-2246, 2018
Zhang, Y., Doughty, H., and Snoek, C. Learning unseen modality interaction. Advances in Neural Information Processing Systems, 36:54716-54700 2023.
12

---

Interpretable Multimodal Interaction-aware Mixture-of-Expers
A. The Connection between Interaction Loss and PID
link our perturbation-based losses to components Partial Information Decomposition (PID), following Bertschinger al. (2014):
X1, X2) (ed(T;; X2) (T; X2) + UT; Xi) Syn(T; X1, X2)
(4)
In the two-modality scenario, our model learns four experts, each trained to specialize in a PID component using corrupted modality inputs.
Unique Information.
Experts Funil and Funi2 are trained on inputs where the other modality is replaced with noise:
Cami Funil (X1, X2) T|,
Cum2
Funi2 (x,
T|
(5)
Assuming X, contains no task-relevant information, these losses approximate:
uuiil & Unq(T; X1 \ X2), Lumi2 & Unq(T; {1)
(6)
This aligns with unique information defined by conditional information under fixed marginals (Bertschinger et al., 2014; Wollstadt al., 2023).
Redundant Information. Expert Fied is trained match predictions from either single-modality input:
[IFra (X,, T| + Frea (X1, - -T))
Lid
(7)
This loss encourages Fred to extract information shared by both X2 and X2, approximating:
Cred x Red(T; X2)
(8)
It aligns with redundancy defined via shared informativeness (Williams & Beer, 2010; Wollstadt al., 2023).
Synergistic Information. Expert Fsy is trained rely on both modalities jointly. It is penalized for performing well any partial view:
Cayn = (1 - T - || [X - -T'l (X1, - 2Tl)
(9)
This loss isolates information that emerges only through joint modality interaction:
Layn o Syn(T;
(10)
This formulation reflects the formal synergy component defined in Williams & Beer (2010); Wibral et al. (2017).
By explicitly constructing perturbed input views that suppress or preserve specific modality contributions, each expert trained to model distinct PID component. This forms a contrastive approximation to the constrained information projections discussed in prior work (Bertschinger et al., 2014; Williams Beer, 2010).
B. Empirical Evidence for the Random Vector Masking
The use of random vector replacement for modality dropout may appear ad hoc. However, our design is motivated by the need to fully suppress information from the dropped modality during interaction supervision. In contrast, alternatives such as mean or zero vector replacement risk preserving residual signals, which can undermine disentanglement of unique and redundant information pathways.
13

---

Interpretable Multimodal Interaction-aware Mixture-of-Expers
This decision is further supported by findings from CoMM (Dufumier et al., 2024), which highlight the regularization benefits and improved robustness of full modality dropout.
To assess this empirically, we conducted an ablation comparing three masking caattegies-raddoon mean, and zero vector uplacements-acree across five datasets. The results (Table 5) show that random vector masking consistently yields stronger performance on most metrics and tasks.
Table Performance comparison across different modality masking strategies (Random, Mean, Zero). Metries: Accuracy (Acc). AUROC, Micro/Macro FI. Numbers are reported as mean + standard deviation.
Dataset
ADNI
MIMIC
IMDB
MOSI
ENRICO
Metric
(3)
AUROC
(2)
AUROC
Micro FI (23)
Macro (23)
Acc (2)
Ace (20)
Random
65.08 1.52
= 0.02
69.78 = 0.91
68.81
61.00
52.38 0.48
71.91 2.20
48.22 1.61
Mean
59.85 & 3.52
76.40 + 2.84
70.00 # 1.27
67.96 :: 1.3
59.36 0. 0.14
50.82 & 0.46
68.95 # 2.37
50.00 = 1.94
Zero
59.48 # 1.61
77.06 # 0.60
69.80 = 0.97
64.62 # 1.39
60.57 # 0.07
51.16 = 0.76
70.41 = 0.66
48.63 = 1.28
These results support our use of random vector masking as more effective strategy for isolating and supervising interaction- specific information flow in multimodal learning.
C. Complete Training Objective
Let {Ei) denote the B = n interaction experts: n uniqueness experts, one synergy expert, and one redundancy expert. For each expert F,, we obtain outputs from (1 n) forward passes (one full input and one for each modality replaced):
{ Fufoorarrdmmliipeeexx... Xn)
The main prediction is computed as:
B
Wi Di where [Wi,.... wB M..ReeeeightXX, X,)
1=1
The task loss is defined as:
Lunk =(ÿ, T)
We define the expert-specific interaction losses as follows:
Uniqueness loss for each F; (i 1,...,n): n):
C (8)
1
TripletLoss (v.0) (1) 0.
Synergy loss (Fn+1):
(n+1)
1
CosSim (eermalize([g.) normalize((!))
int
n
Redundancy loss (Fx+2):
(n+2)
(1 CosSim (normalize(b00)) normalize(() ze(
We then average the interaction loss over all experts:
Li
1
(i) int
B
im
14

---

Interpretable Multimodal Interaction-aware cesture-of-Experrs
The final training objective is:
Liotal L Aim Lime
Model parameters are updated to minimize Lioul-
D. Computational Overhead and Scalability
In theory, I'MOE scales linearly with the number of input modalities. Specifically, the fusion overhead increases by approximately (Numer of modalities +2) times, corresponding one uniqueness expert per modality, plus one redundancy and synergy expert.
To quantify the overhead of our method, we compare I MoEEMMll' with the MulT baseline across three key metrics: training time per epoch (in seconds), inference latency (in seconds), and parameter count. As shown in Table 6, I?Mo introduces moderate increases in compute- rrouhly proportional to the number of modalities plus two (accounting for synergy and redundancy experts). All experiments were run on a single NVIDIA A A100 GPU. Despite this additional cost, the model yields consistent improvements in interpretability and predictive performance, justifinng the added overhead.
Table 6. Comparison of MulT and MiMo training time, inference latency, and model size across datasets.
Train / epoch (s)
Inference (s)
# Params
Dataset
Modalities
MulT
MMMoE-MuTT
MulT
2 1 MulT
MulT
1 "MBEMMulI
ADNI
I, G.G.C. B L, N, C L
8.98 + 0.04
16.82 + 0.02
1.34 + 0.00
2.29 + 0.00
1,072.131
6,696,728
MIMIC
2.24
33.67
0.15 0.00
0.91 0.00
268,034
1,390,095
IMDB
3.62 0.00
0.59
0.53 # 0.00
3.23
1,068,567
4,423,008
MOSI
V. T
0.70
0.00
0.09 0.00
0.48 0.00
134,402
673,935
ENRICO
S, W
1.38 0.02
0.03
0.20 0.00
0.00
538,644
2,352,724
E. Details for Dataset Preprocessing
We followed the same preprocessing procedure of the ADNI dataset and MIMIC dataset, as described in Flex-MoE (Yun al., 2024).
E.1. Detailed Data Preprocessing in ADNI
Imaging, Genetic, Biospecimen, Clinical Modalities. The Alzheimer's Disease Initiative (ADNI) longitudinal multi- center observational study containing multi-modal data from subjects diagnosed cognitively normal (CN), mild cognitive impairment (MCI), and Alzheimer's dementia (AD) (Weiner et al., 2010; 2017). In our experiments, we utilized imaging, genetic, biospecimen, and clinical modalities. The imaging data consisted of magnetic resonance images (MRIs) which were preprocessed using field intensity inhomogeneity correction, gray tissue matter segmentation via MUSE (Multiatlas Region Segmentation Utilizing Ensembles of Registration Algorithms and Parameters) (Doshi et al., 2016), and voxel-wise volumetric mapping of tissue regions. The genetic data consisted of SNP (single nucleotide polymorphisms) data from the ADNI 1, GO/2, and studies. These were preprocessed via alignment to unified reference, followed by aligning strands based on the 1000 Genome Project phase 3, linkage disequilibrium (LD) pruning, and imputation. The resulting data consisted of 144, 746 SNPs. The biospecimen modality included CSF A,31-42 and A,81-40, Total Tau and Phosphorylated Tau, Plasma Neurofilament Light Chain, and ApoE genotype. Clinical data included medical history, neurological exams, patient demographics, medications, and vital signs. Data columns directly containing Alzheimer's Disease diagnosis information were excluded. For both biospecimen and clinical data, numerical data was scaled using a MinMax scaler to a range of 1 to 1, while categorical data was one-hot encoded. Missing values, were imputed using the mean for numerical fields and the mode for categorical fields.
E.2. Detailed Data Preprocessing in MIMIC
Lab, Notes, Codes Modalities. The MIMIC dataset was extracted from the Medical Information Mart for Intensive Care IV (MIMIC-IV) database, which contains de-identified health data for patients who were admitted to either the emergency
15

---

Interpretable Multimodal Interaction-aware Mixture-of-Expert
department or stayed in critical care units of the Beth Israel Deaconess Medical Center in Boston, Massachusetts (Johnson al., 2024; 2023; Goldberger et al., 2000). MIMIC-IV excludes patients under 18 years of age. We take subset of the MIMIC-IV data, where each patient has at least more than visit in the dataset as this subset corresponds to patients who likely have more serious health conditions. For each datapoint, we extract ICD-9 codes, clinical text, and labs and vital values. Using this data, we perform binary classification on one-year mortality. We drop visits that occur at the same time as the patient's s death.
F. Details for Modality-specifie Encoder and Classification Head
ADNI Dataset: For the image modality, employed customized 3D-CNN (Esmaeilzadch et al., 2018) with hidden dimension of 256 as the encoder. For the genomics, clinical, and biospecimen modalities, we used a one-hidden-layer MLP with hidden dimension of 256 as the encoder.
MIMIC Dataset: For all lab, note, and code modalities, we utilized an LSTM with a hidden dimension of 256 as the encoder.
R MOSI Dataset: Gated Recurrent Unit (GRU) with hidden dimension of 256 was used as the encoder for the vision, audio, and text modalities.
4 ENRICO Dataset: For both the screenshot image and wireframe image modalities, we used VGG11 1 from the torchvision library with a hidden dimension size of 16 as the encoder.
IMDB Dataset: For the image modality, a VGG-16 model was applied as the feature extractor. For the language modality, features were extracted using the pretrained Google Word2vec model. Additionally, we employed VGG1 from the torchvision library with hidden dimension size of 16 the encoder and used MaxoutLinear unimodal encoders. following current work (Liang al., 2021).
b Classification Head: For all models and all datasets, we use linear classification head to output the corresponding prediction.
G. Details for Hyperparameter Setting
To improve reproducibility, the tables below provide a summary the hyperparameters used in our experiments. For hyperparameters of other baseline fusion methods, please refer to the scripts in the GitHub repository httpt::///gt com/Raine Raain--Xini I2MoB/ http:///tteeemmaaaa///tttts/////
Table 7. Hyperparameter Configuration for I'Mo -Mul on Different Datasets
Hyperparameter
ADNI
MIMIC
IMDB
MOSI
ENRICO
Learning Rate (1x)
0.0001
0.0001
0.0001
0.0001
0.0001
Temperature for Reweighting (temperature_ro))
2
2.0
2.0
2.0
Hidden Dimension for Reweighting
256
128
256
256
256
Number of Layers in Reweighting souu_layer.nn
2
2
3
3
3
Interaction Loss Weight (interaction peaeetioonncsss.o wetghht)
0.5
0.01
0.5
0.005
0.5
Modality (modality)
IGCB
LNC
LI
TVA
SW
Training Epochs reepohh))))
50
30
40
30
50
Batch Size (batch.size)
32
32
32
32
32
Number of Experts (num.experts)
8
4
4
4
Number of Layers Encoder (nun.leyers.enc)
1
Number of Layers Fusion (nun_layers_fuus
2
2
1
Number of Layers in Prediction (num layers pred)
2
2
2
1
2
Number of Attention Heads (num heads)
4
1
4
1
4
Hidden Dimension (hidden dim)
256
128
256
256
256
Number of Patches (num.patches)
16
4
16

---

Interpretable Multimodal Interaction-aware Mixture-of-Expert
Table 8, Hyperparameter Configuration for "xxoE-SwitchGat Different Datasets
Hyperparameter
ADNI
MIMIC
IMDE
MOSI
ENRICO
Learning Rate (1x)
0.0001
0.0001
0.0001
0.0001
0.0001
Temperature for Rewcighting (cemperature_rv
2
2
2.0
2.0
Hidden Dimension for Reweighting (hidden_dim_rg)
256
256
256
128
128
Number of Layers Reweighting (num.layer.rw)
2
2
1
Interaction Loss Weight (interaction weight) Modality modalltty)
0.5
0.001
IGCB
LNC
LI
TVA
SW
Training Epochs (trshn.epochs)
30
30
40
50
30
Batch Size (batch_s1f ze)
8
64
64
32
8
Number of Experts (num.expert.s)
16
16
4
Number of Layers in Encoder
www.llyerreee.. enc)
2
1
Number of Layers in Fusion (num layers fus)
2
2
2
1
Number of Layers in Prediction (num. layers pred)
2
2
2
1
Number of Attention Hcads (num_heada)
4
4
Hidden Dimension (hidden_dim)
256
128
Number of Patches (num.pat ches)
8
16
16
Table 9. Hyperparameter Configuration for I'MoE- I'Moe-InneeppetCCC Different Datasets
Hyperparameter
ADNI
MIMIC
IMDB
MOSI
ENRICO
Learning Rate (1x)
0.0001
0.0001
0.0001
0.0001
0.0001
Temperature for Reweighting (temperature.uk
2
2
2.0
1.5
4.0
Hidden Dimension for Reweighting (bidden.alim.rw)
128
128
256
256
256
Number of Layers Reweighting (aum_layer.rw)
2
Interaction Loss Weight ( interct ght)
0.5
0.1
0.01
0.001
Modality (moda ity)
IGCB
LNC
1.I
TVA
SW
Tau (7)
1.0
0.7
1.0
1.0
0.5
Threshold (threahold)
0.5
0.5
0.6
0.6
0.4
Train Epochs (train epochs)
30
40
Batch Size
32
128
32
32
64
Hidden Dimension (hidden_dim)
128
256
256
128
256
Hard (hard)
True
True
True
True
True
Table 10. Hyperparameter Configuration I$MoE I*MOE-MOE++ on Different Datasets
MoE++
Hyperparameter
ADNI
MIMIC
IMDB
MOSI
ENRICO
Learning Rate (1x)
0.0001
0.0001
0.0001
0.0001
0.0001
Temperature for Reweighting (semperaturee.v)
2
Hidden Dimension for Reweighting (bidden.
256
256
256
128
256
Number of Layers in Reweighting (num_layer.rw)
3
2
2
2
2
Interaction Loss Weight (Enteractioon_loss.eelegg
0.5
0.5
0.5
0.001
0.5
Modality (modal lit)
IGCB
LNC
L.I
TVA
SW
Training Epochs (t.ra (t.rain.epo... epochs))
50
40
Batch Size (batch size)
64
32
32
32
32
Number of Experts (num.experts)
8
4
4
8
8
Number of Layers Encoder (nun.layers.enc)
2
2
2
Number of Layers Fusion (num_l fus))
1
Number of Layers Prediction (num layers layers.pred)
2
2
2
2
Number of Attention Heads (num, heads)
4
4
4
4
4
Hidden Dimension (hidden_ding
256
128
256
64
64
Number of Patches (num.patchees)
4
4
H. Human Evaluation for Local Interpretation
To strengthen evidence for the local interpretability of our model, we conducted a human evaluation study involving 15 participants. Each participant was shown 20 movie examples, resulting a total 300 interaction expert weight evaluations. Participants were asked to assess how reasonable the model's assigned expert weights were, choosing from a 5-point Likert scale: "Completely makes sense," "Mostly makes sense," "Neutral," "Makes little sense," and "Makes no sense all."
17

---

Interpretable Multimodal Interaction-aware Mixture-of-Expert
Overall, 70.4% of responses were positive (i.e., "Mostly makes sense" "Completely makes sense"), while only 9% negative. Notably, just 0.7% of ratings selected the lowest option. These results suggest that the model's ' expert weight assignments are broadly viewed as reasonable and interpretable by human evaluators.
The questionnaire and de-identified responses are available at httpps:/ggiee com Beon/Raina-Ban Haam/Raina-iiinaaae /ttraa////////////teee/eaaaaaaa asset s/ h/wuman_evaaaa
Table Distribution of human ratings local interaction expert weights (n = 300).
Response Option
Percentage of Responses
Completely makes sense
19.4%
Mostly makes sense
51.0%
Neutral
19.7%
Makes little sense
9.0%
Makes sense at all
0.7%
I. More Qualitative Examples for Local Interpretation
We present a comprehensive visualization of all 23 classes in the IMDB dataset, illustrating local interpretability for individual examples. All examples are correctly predicted by I²MoE.
0.5
Expert Welghting
Logic Values
-5
Weighted
Expert
-10
Undi
Urs-
-15
Synsy
Rdmay
0.0
Uni-L
Unil
Syngy
Rdncy
Experts
The Care Bears live in a country high in the clouds, where they have lot of fun together. But they also care human children on Earth, through huge teleseopes from the sky, and come to help whenever there is need. Nikolas, magician's apprentice,
Final Logits
Beae
they watch
-2
danger of getting under the influence of had spirit, which resides in ancient spell book. The siblings Kim and
-6-
don't anyone anymore after being disappointed once top often. The Care Bears take them into their wonderland where they experience eveiting dangerous adventures together quickly hecome good friends.
Decision Threshold
Figure 6. IMDB example (ID: 0088885).
18

---

Interpretable Multimodal Interaction-aware Mixture-of-Expee
10
5
Expert Weighting
Logit Values
Weightzd Logit
0
-2
-5
Expert
10
Unit
15
Raney
Syngy Experts
DUR MEDIC KRIREER
The history of first vietim modern artillery and its moving agony, amidst conspiracies and betrayals powerful. Life and death of Giovanni De Medici, young, brave captain against Pope, in the first half of 1500.
Fina Logis
Docision Threshald
Figure IMDB example (ID: 0245276).
10
Logit Values
Expert Weighting
0.3
Expert
10
0.2
Sange Rdney
0.1
0.0
UnH-
Uni-1
Syngy
Rdncy
Experts
pretending to his lost unele, brings the boy to magical place which hides the dangerous underground cave. There, Haseem asks of Aladdin to find but simple lamp which contains genic.
Hinal Logits
Aladdin refuses to come outside of the with the lamp and eventually discovers secret. A penie inside lamp then helps his life change and marry love.
Decision Threshold
Figure IMDB example (ID: 0827999).
19


Reviews:
[{"review_id": "15SsFMJuQi", "reviewer": "Reviewer_KHiC", "summary": "The paper introduces $I^2MoE$, an end-to-end mixture-of-experts framework that explicitly models heterogeneous interactions between input modalities. By deploying specialized interaction experts (e.g., uniqueness, synergy, redundancy) and a reweighting module, $I^2MoE$ not only improves task performance—demonstrated by accuracy gains on datasets like ADNI, CMU-MOSI, and MM-IMDB—but also provides both local and global interpretability of multimodal fusion. The method leverages weakly supervised interaction losses based on modality perturbation to guide expert specialization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive feedback. Point-to-point responses below.\n\n>Q1. The connection between interaction loss and PID\n\n\nWe would appreciate any insights from the reviewer on this point. Below, we attempt to connect each expert trained on the perturbed input views to a distinct PID component. This might form a contrastive approximation to the constrained information projections discussed in [1, 2]:\n\n\\\\[\nI(T; X_1, X_2) = \\mathrm{Red}(T; X_1, X_2) + \\mathrm{Unq}(T; X_1 \\\\setminus X_2) + \\mathrm{Unq}(T; X_2 \\\\setminus X_1) + \\mathrm{Syn}(T; X_1, X_2)\n\\\\]\n\nIn the two-modality scenario, our model learns four experts, each trained to specialize in a PID component using perturbed modality inputs.\n\n**Unique Information [1, 3].** Experts \\\\( F\\_{\\text{uni}1} \\\\) and \\\\( F\\_{\\text{uni}2} \\\\) are trained on inputs where the other modality is replaced with noise:\n\n\\\\[\n\\\\mathcal{L}\\_{\\text{uni}1} = \\\\| F\\_{\\text{uni}1}(X_1, \\\\tilde{X}\\_2) - T \\\\|, \\quad\n\\\\mathcal{L}\\_{\\text{uni}2} = \\\\| F\\_{\\text{uni}2}(\\\\tilde{X}\\_1, X_2) - T \\\\|\n\\\\]\n\nAssuming \\\\( \\\\tilde{X}\\_i \\\\) contains no task-relevant information, these losses approximate:\n\n\\\\[\n\\\\mathcal{L}\\_{\\text{uni}1} \\\\propto \\mathrm{Unq}(T; X_1 \\\\setminus X_2), \\quad\n\\\\mathcal{L}\\_{\\text{uni}2} \\\\propto \\mathrm{Unq}(T; X_2 \\\\setminus X_1)\n\\\\]\n\n\n**Redundant Information [2, 3].** Expert \\\\( F\\_{\\text{red}} \\\\) is trained to match predictions from either single-modality input:\n\n\\\\[\n\\\\mathcal{L}\\_{\\text{red}} = \\\\frac{1}{2} \\\\left( \\\\| F\\_{\\text{red}}(X_1, \\\\tilde{X}\\_2) - T \\\\| + \\\\| F\\_{\\text{red}}(\\\\tilde{X}\\_1, X_2) - T \\\\| \\\\right)\n\\\\]\n\nThis loss encourages \\\\( F\\_{\\text{red}} \\\\) to extract information shared by both \\\\( X_1 \\\\) and \\\\( X_2 \\\\), approximating:\n\n\\\\[\n\\\\mathcal{L}\\_{\\text{red}} \\\\propto \\mathrm{Red}(T; X_1, X_2)\n\\\\]\n\n\n**Synergistic Information [2, 4].** Expert \\\\( F\\_{\\text{syn}} \\\\) is trained to rely on both modalities jointly. It is penalized for performing well on any partial view:\n\n\\\\[\n\\\\mathcal{L}\\_{\\text{syn}} = \\\\frac{1}{2} \\\\left( \\\\| F\\_{\\text{syn}}(X_1, X_2) - T \\\\| - \\\\| F\\_{\\text{syn}}(\\\\tilde{X}\\_1, X_2) - T \\\\| - \\\\| F\\_{\\text{syn}}(X_1, \\\\tilde{X}\\_2) - T \\\\| \\\\right)\n\\\\]\n\nThis loss isolates information that emerges only through joint modality interaction:\n\n\\\\[\n\\\\mathcal{L}\\_{\\text{syn}} \\\\propto \\mathrm{Syn}(T; X_1, X_2)\n\\\\]\n\n>Q2. Random vector replacement for modality dropout\n\nPlease see our response to Reviewer LfMU Q3.\n\n>Q3. Strategies to mitigate accuracy degradation in imbalanced datasets like MIMIC?\n\nIn imbalanced settings like MIMIC, threshold-independent metrics such as AUROC provide a more reliable measure of performance. We mitigate class-imbalance by applying class weighting (0.25 for negative, 0.75 for positive) during training across all models.\n\nTo clarify performance across classes, we report per-class accuracy, balanced accuracy, and AUROC below. I2MoE-MulT achieves higher balanced accuracy and AUROC than MulT, indicating improved positive class recognition without sacrificing majority class performance. \n\n| Metrics       | Acc (+) | Acc (-) | Bal. Acc | Avg. Acc | AUROC |\n|---------------|---------|---------|----------|----------|--------|\n| **MulT**       | 20.87   | **90.31**   | 55.59    | **74.64**    | 65.61  |\n| **i2MoE-MulT** | **50.49**   | 75.43   | **62.96**    | 69.80    | **69.44**  |\n\n\n>Q4. A discussion on computational overhead and scalability is encouraged\n\nPlease see our response to Reviewer aVYo Q2.\n\n>Q5. How does I2MoE scale when extending to more than two modalities or when applied to other multimodal tasks (e.g., video–audio fusion)?\n\nMost real-world multimodal datasets ([5, 6]) involve fewer than four modalities. I2MoE performs well in such settings—for example, ADNI includes imaging, genetics, clinical tests, and biospecimens. We also evaluated I2MoE on video–audio fusion using CMU-MOSI (text, video, audio), where I2MoE improves the performance of four different baseline fusion methods (Table 1 and Table 2).\n\n>C1. I2MoE contrasts with conventional fusion methods and recent MoE-based models\n\nThanks for the comment. We will expand Section 2 to position I2MoE within the broader MoE literature you mentioned more explicitly.\n\n[1] Bertschinger et al., 2014. *Quantifying Unique Information*. Entropy.  \n[2] Williams and Beer, 2010. *Nonnegative Decomposition of Multivariate Information*. arXiv.  \n[3] Wollstadt et al., 2023. *A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decomposition*. JMLR.  \n[4] Wibral et al., 2017. *Partial information decomposition as a unified approach to the specification of neural goal functions*. Brain and Cognition.  \n[5] Liang et al., 2021. *MultiBench: Multiscale Benchmarks for Multimodal Representation Learning*. NeurIPS.   \n[6] Liang et al., 2022. *High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning*. arXiv:2203.01311.", "claims_and_evidence": "The authors claim that:\n- Modeling distinct multimodal interactions via dedicated experts improves predictive performance and interpretability.\n- The reweighting mechanism provides sample-level and dataset-level insights.\n\nAblation studies support the necessity of each design component. \n\nThese claims are backed by extensive experiments (including ablations) and visualizations (e.g., Figure 3 and Table 4). However, the rationale behind the specific random vector perturbation strategy is more heuristic than theoretically grounded.", "methods_and_evaluation_criteria": "The dual-objective loss (task loss plus interaction loss) and the use of a reweighting model are central to the design. Experiments across five datasets—with comparisons to both vanilla fusion and advanced baselines (e.g., SwitchGate, MoE++)—demonstrate consistent performance gains. One concern is the sensitivity of the method to hyperparameters, particularly in imbalanced datasets (e.g., MIMIC), where accuracy drops occur despite AUROC improvements.", "theoretical_claims": "The paper is motivated by the Partial Information Decomposition (PID) framework; however, the theoretical connection between the proposed weakly supervised interaction loss and formal PID measures is only loosely established.", "experimental_designs_or_analyses": "The experimental setup is comprehensive:\n- **Datasets:** Experiments span diverse domains (medical imaging, sentiment analysis, movie genre classification) demonstrating the model’s generality.\n- **Ablation Studies:** Detailed ablations confirm the contribution of the interaction loss, reweighting module, and perturbation strategy.\n- **Interpretability Analysis:** Visualizations of local expert contributions and global weight distributions validate the interpretability claims.", "relation_to_broader_scientific_literature": "The paper is well situated within the multimodal fusion literature. It contrasts with conventional fusion methods and recent MoE-based models. For instance:\n- **MMoE (Yu et al., 2024):** Similar in spirit to I2MoE, this work also decomposes multimodal interactions into specialized experts.\n- **MoMa (Jiang et al., 2024):** Introduces modality-aware expert routing for early fusion efficiency, offering insights into modality-specific parameter allocation\n- **Interpretable Mixture of Experts for Structured Data (Ismail et al., 2022):** Although targeting tabular and time-series data, its inherently interpretable design may offer complementary perspectives for multimodal settings.\n- **Dividing and Conquering a BlackBox (Ghosh et al., 2023):** Presents a divide-and-conquer approach for extracting interpretable models, highlighting expert specialization strategies .\n- **LIMoE (Mustafa et al., 2022):** Uses contrastive learning in a MoE framework for vision–language tasks, emphasizing organic emergence of modality-specific experts .\nIntegrating these discussions would further contextualize I2MoE’s contributions.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "**Strengths:**\n- **Novel Architecture:** Explicitly modeling heterogeneous interactions via specialized experts.\n- **Interpretability:** Provides both local and global explanations, supported by qualitative and quantitative analyses.\n- **Comprehensive Evaluation:** Extensive experiments and ablation studies across diverse datasets substantiate the claims.\n\n**Weaknesses:**\n- **Theoretical Grounding:** The connection to PID is mainly heuristic; more formal theoretical analysis would be beneficial.\n- **Perturbation Method:** The use of random vector replacement for modality dropout is ad hoc and might affect stability.", "other_comments_or_suggestions": "A discussion on computational overhead and scalability with an increased number of modalities is encouraged.", "questions_for_authors": "1. Could you elaborate on strategies to mitigate accuracy degradation in imbalanced datasets like MIMIC?\n2. How does I2MoE scale when extending to more than two modalities or when applied to other multimodal tasks (e.g., video–audio fusion)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DRSyNVOqF2", "reviewer": "Reviewer_zoTN", "summary": "In this paper, the authors introduced I2MOE, a novel multimodal model that trains a different set of experts to model each type of multimodal interaction between modalities. Each expert is trained with a different interaction loss specifically designed for the type of interaction it has to deal with, in addition to the main task objective. The model is evaluated on 5 different multimodal tasks, where the model achieves top performance on all of them. The authors also performed additional analysis or demonstrations to show that I2MOE can generalize to different fusion backbones, offers local and global interpretability, and works with more than 2 modalities. There is also ablation studies to justify the design choices of I2MOE.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive feedback. Point-to-point responses below.\n\n> Q1.The support for \"local interpretation\" is only backed by qualitative samples from one task\n\nThanks for suggesting to strengthen the evidence for local interpretability. We conducted a human evaluation with 15 participants on 20 movie examples (300 total ratings), asking how reasonable the model’s assigned interaction expert weights were. Participants chose from five options, ranging from “Makes no sense at all” to “Completely makes sense.” \n\n70.4% of responses were positive (Mostly or Completely makes sense), while only 9% were negative, and just 0.7% selected the lowest rating. These results suggest that the model’s expert weights are broadly viewed as reasonable and interpretable by human evaluators. \n| **Response**                     | **Percentage of all responses (n=300)** |\n|-----------------------------|--------------------------------------|\n| 'Completely makes sense'    | 19.7%                                |\n| 'Mostly makes sense'        | 51%                                  |\n| 'Neutral'                   | 19.7%                                |\n| 'Makes little sense'        | 9%                                   |\n| 'Makes no sense at all'     | 0.7%                                 |\n\n\nLink to the questionnaire: [Link](https://anonymous.4open.science/r/I2MoE-rebuttal-8308/AR-zoTH/Q1_human_eval/Human%20Evaluation%20of%20I2MoE%20Interpretability.pdf)\n\nLink to deidentified response: [Link](https://anonymous.4open.science/r/I2MoE-rebuttal-8308/AR-zoTH/Q1_human_eval/human_eval_deid.csv)\n\n\n>Q2. The font size of the tables and figures are tiny, making them hard to read.\n\nThank you for the helpful suggestion. We have increased the font sizes in all figures and tables and will include them in the final version. \n\nUpdated tables and figures: [Link](https://anonymous.4open.science/r/I2MoE-rebuttal-8308/AR-zoTH/Q2_font_size/table_4.png)\n\n>Q3. Writing down the complete objective as a mathematical expression (or maybe an algorithm block for the entire training objective)\n\n**Algorithm block** for I2MoE training and inference forward pass: [Link](https://anonymous.4open.science/r/I2MoE-rebuttal-8308/AR-zoTH/Q3_objective/i2moe_algorithm.png)\n\nBelow we explain the complete objective in math expression:\nLet \\\\( \\\\{F_i\\\\}_{i=1}^{B} \\\\) denote the \\\\( B = n + 2 \\\\) interaction experts: \\\\( n \\\\) uniqueness experts, one synergy expert, and one redundancy expert. For each expert \\\\( F\\_i \\\\), we obtain outputs from \\\\( (1 + n) \\\\) forward passes (one full input and one for each modality replaced):\n\n\\\\[\n[\\\\hat{y}\\_i^{(0)}, \\\\hat{y}\\_i^{(1)}, \\\\dots, \\\\hat{y}\\_i^{(n)}] = F\\_i\\\\mathrm{.forward\\\\_multiple}(X\\_1, \\\\dots, X\\_n)\n\\\\]\n\n\nThe main prediction is computed as:\n\n\\\\[\n\\\\hat{y} = \\\\sum\\_{i=1}^{B} w\\_i \\\\cdot \\\\hat{y}\\_i^{(0)}, \\\\quad \\\\text{where } [w\\_1, \\\\dots, w\\_B] = \\text{MLPReWeight}(X\\_1, \\\\dots, X\\_n)\n\\\\]\n\nThe task loss is defined as:\n\n\\\\[\n\\\\mathcal{L}\\_{\\\\text{task}} = \\\\ell(\\\\hat{y}, T)\n\\\\]\n\nWe define the expert-specific interaction losses as follows:\n\n**Uniqueness loss** for each \\\\( F\\_i \\\\) (\\\\( i = 1, \\\\dots, n \\\\)):\n\n\\\\[\n\\\\mathcal{L}\\_{\\\\text{int}}^{(i)} = \\\\frac{1}{n - 1} \\\\sum\\_{j \\\\ne i} \\\\text{TripletLoss}\\\\left( \\\\hat{y}\\_i^{(0)},\\\\; \\\\hat{y}\\_i^{(j)},\\\\; \\\\hat{y}\\_i^{(i)} \\\\right)\n\\\\]\n\n**Synergy loss** (\\\\( F\\_{n+1} \\\\)):\n\n\\\\[\n\\\\mathcal{L}\\_{\\\\text{int}}^{(n+1)} = \\\\frac{1}{n} \\\\sum\\_{j=1}^{n} \\\\text{CosSim}\\\\left( \\\\text{normalize}(\\\\hat{y}\\_{n+1}^{(0)}),\\\\; \\\\text{normalize}(\\\\hat{y}\\_{n+1}^{(j)}) \\\\right)\n\\\\]\n\n**Redundancy loss** (\\\\( F\\_{n+2} \\\\)):\n\n\\\\[\n\\\\mathcal{L}\\_{\\\\text{int}}^{(n+2)} = \\\\frac{1}{n} \\\\sum\\_{j=1}^{n} \\\\left( 1 - \\\\text{CosSim}\\\\left( \\\\text{normalize}(\\\\hat{y}\\_{n+2}^{(0)}),\\\\; \\\\text{normalize}(\\\\hat{y}\\_{n+2}^{(j)}) \\\\right) \\\\right)\n\\\\]\n\nWe then average the interaction loss over all experts:\n\n\\\\[\n\\\\mathcal{L}\\_{\\\\text{int}} = \\\\frac{1}{B} \\\\sum\\_{i=1}^{B} \\\\mathcal{L}\\_{\\\\text{int}}^{(i)}\n\\\\]\n\nThe final training objective is:\n\n\\\\[\n\\\\mathcal{L}\\_{\\\\text{total}} = \\\\mathcal{L}\\_{\\\\text{task}} + \\\\lambda\\_{\\\\text{int}} \\\\cdot \\\\mathcal{L}\\_{\\\\text{int}}\n\\\\]\n\nModel parameters are updated to minimize \\\\( \\\\mathcal{L}\\_{\\\\text{total}} \\\\).", "claims_and_evidence": "The claims made in the submission is supported by clear and convincing evidence from the experiments.\n\nThe only minor problem is that, the support for \"local interpretation\" is only backed by qualitative samples from one task (mm-imdb) in both main text and appendix. The support for the local interpretability would be stronger with either a human evaluation of interpretability (i.e. whether the interaction attributions generated by I2MOE makes sense) or by including more qualitative samples from tasks other than mm-imdb.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria makes sense for the problem.", "theoretical_claims": "No theoretical claim.", "experimental_designs_or_analyses": "I have checked the soundness and validity of experimental designs, including the main experiment with 5 tasks, and additional analysis and ablations. The experimental designs looks valid and sound to me.", "relation_to_broader_scientific_literature": "Compared to existing works in multimodal machine learning and multimodal interaction quantification/interpretability, the key contribution of this paper is that the proposed method creates one single end-to-end model that (1) achieves high performance in tasks, (2) inherently offers multimodal interaction quantification and interpretability, (3) applies to tasks with more than 2 modalities, and (4) generalizes well to different types of multimodal fusion. While previous works have proposed models or methods that can achieve some of the above, the proposed method seem to be the first that can achieve all of them within one single end-to-end model.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Additional Strength:\n\n- The presentation quality of the paper is good. The methodology is easy to follow and the intuition behind each decision is clearly explained.\n\n- There is ablation studies that clearly demonstrates the need for each design choice of the proposed method.\n\nAdditional Weakness:\n\n- The font size of the tables and figures are tiny, making them hard to read.\n\n- The paper did not specify the complete objective (i.e. the final combination of all task loss and interaction losses). Maybe writing down the complete objective as a mathematical expression (or maybe an algorithm block for the entire training objective) would make things more clear. For example, currently it is not clear how different losses are combined, or whether they are weighted during the combination.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Can you clarify how different interaction losses are combined with the main task objectives? If they are added together, how is each loss weighted?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o1ibUlTP3A", "reviewer": "Reviewer_LfMU", "summary": "This paper addresses multimodal learning using mixture-of-experts, where dedicated experts learn distinct information from input modalities. The authors introduce a reweighting model to interpretably assign weights to the experts, facilitating understanding of their individual importance. The proposed approach is evaluated on five multimodal benchmarks covering various modalities and tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful feedback. Point-to-point responses below. **All supplementary on [GitHub](https://anonymous.4open.science/r/I2MoE-rebuttal-8308/README.md)**.\n\n>Q1. I2MoE lower scores on MM-IMDB?\n\nThis is primarily due to **differences in experimental setups**:\n1. Evaluation Setup: Our experiments follow the setup in MultiBench [1]. Within this framework, I2MoE achieves Micro-F1: 61.00 and Macro-F1: 52.38, outperforming the SOTA (Micro: 59.3, Macro: 50.2; see Table 23 in [1]).\n2. Reimplement: While [2–4] report higher MM-IMDB scores, they adopt different experimental setups and do not release code for exact reproduction. We reimplemented CentralNet and ViLT under the MultiBench setting and observed performance drops compared to their original reports.\n3. I2MoE Improvement: We further evaluated I2MoE as a drop-in framework applied to CentralNet and MulT. As shown below, I2MoE consistently improves performance across both Micro and Macro F1 under the MultiBench setup. \n\n| Model | **Micro F1**  | **Macro F1** |\n|--------------------|------------------|------------------|\n| CentralNet | 58.57 ± 0.58 | 49.90 ± 0.24  |\n| I2MoE-CentralNet | 58.72 ± 0.13  | 50.13 ± 0.21 |\n| ViLT  | 58.38 ± 0.44 | 48.31 ± 0.32 |\n| I2MoE-ViLT  | 59.53 ± 1.84     | 49.66 ± 2.23 |\n| MulT  | 59.68 ± 0.19     | 51.41 ± 0.04 |\n| **I2MoE-MulT**   | **61.00 ± 0.44** | **52.38 ± 0.48** |\n\n>Q2. MM-IMDB result in Figure 5 appears different from Table 1 \n\nThanks for catching this–we forgot to update the MM-IMDB result in Figure 5. Updated figures can be found on GitHub.\n\n>Q3. …motivation for using random vectors to mask modalities? ..modality mean vectors..?\n\nWe use random vectors to completely remove information from the masked modality so that any observed uniqueness and synergistic interaction from the masked modality cannot be attributed to leakage. \n\nWhile mean vectors could be less noisy, they are also dynamic in our setting. Since I2MoE is trained end-to-end with modality-specific encoders, the mean vectors evolve across training epochs. To mitigate this instability, we maintain a running average of mean vectors (see GitHub), but this introduces additional complexity and still does not guarantee full information removal.\n\nFurther, we compare random / mean / zero vectors across five datasets. As shown below, random vectors consistently outperform mean vectors in nearly all settings. \n\n| | ADNI | | MIMIC | | IMDB | | MOSI | ENRICO |\n|--------------|---------------|--------------|---------------|--------------|----------------|--------------|---------------|----------------|\n| | **Acc** | **AUROC** | **Acc** | **AUROC** | **Micro F1** | **Macro F1** | **Acc** | **Acc** |\n| **Random** | **65.08 ± 1.52** | **81.09 ± 0.02** | 69.78 ± 0.91 | **68.81 ± 0.99** | **61.00 ± 0.44**  | **52.38 ± 0.48** | **71.91 ± 2.20**  | 48.22 ± 1.61   |\n| **Mean** | 59.85 ± 3.52  | 76.40 ± 2.84 | **70.00 ± 1.27** | 67.96 ± 1.43 | 59.36 ± 0.14  | 50.82 ± 0.46 | 68.95 ± 2.37  | **50.00 ± 1.94** |\n| **Zero** | 59.48 ± 1.61 | 77.06 ± 0.60 | 69.80 ± 0.97 | 64.62 ± 1.39 | 60.57 ± 0.07  | 51.16 ± 0.76 | 70.41 ± 0.66  | 48.63 ± 1.28   |\n\n>Q4. Interpretation of Figure 3b\n\nFigure 3b shows a local (sample-level) decomposition for a specific MM-IMDB example, where the language modality contributes less unique information for that instance. This does not reflect global trends in the MM-IMDB dataset. For dataset-level insights, we refer the reviewer to Figure 4, which summarizes expert weights across the entire test set.\n\n>Q5. (I2MoE) could significantly increase computational costs\n\nPlease see our response to Reviewer aVYo Q2.\n\n>Q6. The studied baselines such as VGG and LSTM seem somewhat outdated\n\nWe clarify that VGG and LSTM are used only as modality-specific encoders, not as fusion architectures. This setup follows widely adopted practice in multimodal learning benchmarks such as [1], where lightweight encoders are used to ensure a fair comparison of fusion methods. We agree that powerful modality-specific encoders could further boost performance. Due to time constraints, we plan to incorporate them in the final version.\n\n>C1. Some related studies might have been overlooked\n\nWe note that the two works mentioned focus on generative modeling, but we focus on interpretable discriminative modeling. We consider these lines of research orthogonal but will clarify this distinction and cite these works in our related work section.\n\n>C2. Some dataset names appear inconsistent (e.g., \"MM-IMDB\" vs. \"MMIMDB\").\n\nWe will standardize all occurrences to “IMDB” in the final version.\n\n[1] Liang et al., 2021. *MultiBench: Multiscale Benchmarks for Multimodal Representation Learning*. NeurIPS.  \n[2] Pérez-Rúa et al., 2019. *MFAS: Multimodal Fusion Architecture Search*. CVPR.  \n[3] Vielzeuf et al., 2018. *CentralNet: A Multilayer Approach for Multimodal Fusion*. ECCV Workshops.  \n[4] Ma et al., 2022. *Are Multimodal Transformers Robust to Missing Modality?*. CVPR.", "claims_and_evidence": "The related work section (lines 93-94) mentions only one previous work applying mixture-of-expert (MoE) to multimodal learning. However, some related studies might have been overlooked, such as [1][2].\n\n[1] Wu, Mike, and Noah Goodman. \"Multimodal generative models for scalable weakly-supervised learning.\" Advances in neural information processing systems 31 (2018).\n\n[2] Shi, Yuge, Brooks Paige, and Philip Torr. \"Variational mixture-of-experts autoencoders for multi-modal deep generative models.\" Advances in neural information processing systems 32 (2019).", "methods_and_evaluation_criteria": "Could the authors clarify their motivation for using random vectors to mask modalities? There is concern that using random vectors might add noise to the fused representations. Would modality mean vectors potentially offer a less noisy alternative? Additional clarification on your design choice here would strengthen the paper.", "theoretical_claims": "All provided equations appear correct.", "experimental_designs_or_analyses": "Overall, the experimental design and analyses appear sound, but several aspects could benefit from clarification:\n+ In Table 1, the reported results on MM-IMDB (Micro F1: 61.00; Macro F1: 52.38) seem notably lower compared to previous literature such as MFAS [3] (Macro F1: 55.70), CentralNet [4] (Micro F1: 63.90; Macro F1: 56.10), and ViLT [5] (Micro F1: 64.70; Macro F1: 55.30). Given that these methods employ relatively straightforward fusion strategies (e.g., early fusion), could the authors discuss why their proposed MoE approach achieves lower scores?\n+ The reported MM-IMDB result in Figure 5 (49.21) appears different from the value in Table 1 (52.38). Could the authors clarify this inconsistency?\n+ In Figure 3(b), the unique information learned from language modality appears to have lower weights. Does this suggest language is less important for MM-IMDB? It would be valuable to have more insight or discussion on this observation.\n+ The proposed method requires multiple runs during training, which could significantly increase computational costs. It would be helpful if the authors could report metrics related to computational efficiency, such as training time. \n+ Additionally, the studied baselines such as VGG and LSTM seem somewhat outdated. Incorporating more recent baseline methods would strengthen the evaluation.\n\n[3] Pérez-Rúa, Juan-Manuel, et al. \"MFAS: Multimodal fusion architecture search.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\n[4] Vielzeuf, Valentin, et al. \"Centralnet: a multilayer approach for multimodal fusion.\" Proceedings of the European Conference on Computer Vision (ECCV) Workshops. 2018.\n\n[5] Ma, Mengmeng, et al. \"Are multimodal transformers robust to missing modality?.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.", "relation_to_broader_scientific_literature": "The idea of interpretable multimodal fusion is valuable, especially in safety-critical domains such as healthcare.", "essential_references_not_discussed": "Please see references noted in the \"Claims and Evidence\" section above.", "other_strengths_and_weaknesses": "Strengths:\n+ The paper is generally well-organized and clear.\n+ The problem of interpretable multimodal fusion is meaningful and relevant to a broad audience.\n\nWeaknesses:\n- Please refer to the comments provided above for potential improvements.", "other_comments_or_suggestions": "Some dataset names appear inconsistent (e.g., \"MM-IMDB\" vs. \"MMIMDB\").", "questions_for_authors": "The following clarifications would help strengthen the manuscript:\n+ Could you provide insights on why your approach yields lower performance compared to simpler fusion methods in existing literature?\n+ Can you explain the inconsistency between the MM-IMDB results reported in Table 1 and Figure 5?\n+ Please further elaborate on your insight of using random vector masking for modalities. What are the potential limitations or implications of this design choice?\n+ For more questions, see the above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FIBfXdNnNh", "reviewer": "Reviewer_aVYo", "summary": "The paper introduces a new mixture-of-experts framework designed to explicitly model diverse modality interactions and multi-modality fusion, through specialized parameters and weakly-supervised interaction losses. The proposed method is validated on five multimodal datasets, across different modalities, showing state-of-the-art performances.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your encouraging feedback. Point-to-point responses below.\n\n>Q1. For using triplet margin loss to model uniqueness interactions, what is the margin used in the work, why is the margin loss used here?\n\nWe use triplet margin loss to uniqueness interactions, as it naturally aligns with the modeling of uniqueness—there are both positive and negative examples. For example, for uniqueness expert for modality 1, we treat the full input as the anchor, the one input with modality 1 being masked as a negative, and all other perturbed inputs as positives. \n\nTriplet margin loss maximizes the distance between the anchor and a negative and minimizes the distance between an anchor and a positive. We set the margin to 1.0, selected based on validation performance. A sweep across three datasets showed that margin = 1.0 performs best in nearly all cases (see table below). \n\n| Dataset | ADNI        |        | MIMIC       |        | ENRICO        |\n|--------|-------------|--------|-------------|--------|---------------|\n| **Margin** | **Acc (3)**     | **AUROC**  | **Acc (2)**     | **AUROC**  | **Acc (20)**  |\n| 0.2 | 64.05 ± 1.97 | 80.47 ± 0.95 | 68.07 ± 2.07 | 69.42 ± 1.65 | 46.58 ± 1.75 |\n| 0.5 | 63.49 ± 1.08 | 81.05 ± 0.83 | 69.21 ± 0.61 | 68.53 ± 0.65 | 47.72 ± 1.78 |\n| **1 (Current)** | **65.08 ± 1.52** | **81.09 ± 0.02** | **69.78 ± 0.91** | **68.81 ± 0.99** | **48.22 ± 1.61** |\n| 2 | 62.56 ± 2.06 | 79.49 ± 1.82 | 68.76 ± 1.16 | 69.97 ± 0.42 | 46.80 ± 1.71 |\n| 5 | 63.96 ± 0.95 | 80.47 ± 1.25 | 68.12 ± 1.86 | 68.07 ± 1.34 | 47.95 ± 1.01 |\n\n\n>Q2. The author should discuss the computation overhead for the proposed method, compared with its counterparts, such as I2MoE-MulT vs. MulT\n\nWhile I2MoE introduces additional forward passes—scaling linearly with the number of modalities—the method remains fully end-to-end and weakly supervised. The fusion overhead increases by approximately (#modalities + 2), corresponding to the number of specialized experts.\n\nTo quantify the overhead, we compare I2MoE-MulT to MulT across training time per epoch, inference latency, and parameter count (see table below). All experiments are conducted on a single A100 GPU. Despite moderate increases, we find the added cost justified by the significant gains in interpretability and predictive performance.  \n\n| Dataset | ADNI | MIMIC | IMDB | MOSI | ENRICO |\n|---------|------|--------|----------|-----------|--------|\n| **Modality** | I,G,C,B | L,N,C | L,I | V,A,T | S,W |\n| **Train per epoch (s)** | | | | | |\n| MulT | 8.98 ± 0.04 | 2.24 ± 0.01 | 3.62 ± 0.00 | 0.70 ± 0.00 | 1.38 ± 0.02 |\n| I2MoE-MulT | 16.82 ± 0.02 | 33.67 ± 0.67 | 44.20 ± 0.59 | 4.47 ± 0.01 | 6.17 ± 0.03 |\n| **Inference (s)** | | | | | |\n| MulT | 1.34 ± 0.00 | 0.15 ± 0.00 | 0.53 ± 0.00 | 0.09 ± 0.00 | 0.20 ± 0.00 |\n| I2MoE-MulT | 2.29 ± 0.00 | 0.91 ± 0.00 | 3.23 ± 0.00 | 0.48 ± 0.00 | 0.44 ± 0.00 |\n| **# Parameters** | | | | | |\n| MulT | 1,072,131 | 268,034 | 1,068,567 | 134,402 | 538,644 |\n| I2MoE-MulT | 6,696,728 | 1,390,095 | 4,423,008 | 673,935 | 2,352,724 |\n\n\n>Q3. MMoE is a very close work to the proposed method, but it is not well compared in the experiment, are there any special concerns about not comparing with it?\n\nWhile both MMoE and I2MoE aim to model heterogeneous interactions, they differ substantially in design and applicability, making direct comparison non-trivial:\n1. **End-to-End vs. Preprocessing Dependency**: I2MoE is fully end-to-end, with interaction specialization emerging via weak supervision. MMoE relies on a seperate preprocessing step to cluster training data by interaction type, breaking end-to-end training.\n2. **Local Interpretability**: I2MoE provides instance-level interpretability by quantifying expert contributions per sample. MMoE lacks this capability, limiting its utility in settings requiring explanation of individual predictions.\n3. **Generalizability to Higher Modalities and Complex Domains**: MMoE is tailored to vision-language tasks with pretrained LLM/VLM backbones. Its extension to domains like healthcare—where pretrained models for structured data or multi-way modality combinations are lacking—is unclear. In contrast, I2MoE operates without modality-specific pretraining and supports >2 modalities, as demonstrated on ADNI and MIMIC datasets. \n\n>Q4. In section 6.2, how to determine the agreement/disagreement between experts, do the authors use any thresholds?\n\nWe define expert agreement based on predicted labels. For single-label classification tasks (i.e., ADNI, MIMIC, ENRICO), each expert outputs a logit vector, and predictions are obtained via argmax. For multi-label classification tasks (i.e., MM-IMDB), predictions are thresholded at 0.5 after sigmoid activation. For the regression task (i.e., CMU-MOSI), each expert outputs a real number the threshold is set at 0. Agreement occurs when all experts predict the same label(s); disagreement is defined as any mismatch between expert outputs.", "claims_and_evidence": "1. For using triplet margin loss to model uniqueness interactions, what is the margin used in the work, why is the margin loss used here?\n2. The author should discuss the computation overhead for the proposed method, compared with its counterparts, such as I2MoE-MulT vs. MulT.\n3. MMoE is a very close work to the proposed method, but it is not well compared in the experiment, are there any special concerns about not comparing with it? \n4. In section 6.2, how to determine the agreement/disagreement between experts, do the authors use any thresholds?", "methods_and_evaluation_criteria": "The proposed methods and evaluation look sound", "theoretical_claims": "There are no issues regarding the theoretical claims", "experimental_designs_or_analyses": "Yes, the experimental design and analysis are sound", "relation_to_broader_scientific_literature": "It is a very interesting topic to achive interpretable interaction/fusion on multi-modality data.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a multimodal learning model that trains a different set of experts to model each type of multimodal interaction between modalities.  

After the rebuttal, three reviewers ( KHiC, zoTN, and aVYo) gave positive ratings (3,4,3), recognizing the paper's strong performance and novel ideas.  Reviewer LfMU requested more detailed explanations of certain technique designs, as well as clarifications regarding the experimental design and analysis. The authors provided a thorough response, but Reviewer LfMU did not reply to the feedback.  

The Area Chair (AC) agrees with most of the reviewers and recommends accepting the paper. Additionally, it is strongly suggested that the authors incorporate the content discussed in the feedback to make the paper more comprehensive.
Decision: Accept (poster)