Balancing Multimodal Training Through Game-Theoretic Regularization
Authors: Konstantinos Kontras
Thomas Strypsteen
Christos Chatzichristos
Paul Pu Liang
Matthew B. Blaschko
Maarten De Vos
Venue: NeurIPS 2025 spotlight ()
Keywords: Multimodal Deep Learning
Multimodal Competition
Primary area: deep_learning

Abstract: Multimodal learning holds the promise for richer information extraction by capturing dependencies across data sources. Yet, current training methods often underperform due to modality competition, a phenomenon where modalities contend for training resources, leaving some underoptimized. This raises a pivotal question: how can we address training imbalances, ensure adequate optimization across all modalities, and achieve consistent performance improvements as we transition from unimodal to multimodal data? This paper proposes the Multimodal Competition Regularizer (MCR), inspired by a mutual information (MI) decomposition designed to prevent the adverse effects of competition in multimodal training. Our key contributions are: 1) A game-theoretic framework that adaptively balances modality contributions by encouraging each to maximize its informative role in the final prediction. 2) Refining lower and upper bounds for each MI term to enhance the extraction of both task-relevant unique and shared information across modalities. 3) Proposing latent space permutations for conditional MI estimation, significantly improving computational efficiency. MCR outperforms all previously suggested training strategies and simple baselines, demonstrating that training modalities jointly lead to important performance gains on synthetic and large real-world datasets. We release our code and models at https://github.com/kkontras/MCR.

Balancing Multimodal Training Through Game-Theoretic Regularization
Konstantinos Kontras
Thomas Strypsteen'
Christos Chatzichristos'
Paul Pu Liang
Matthew Blaschko
Maarten De Vos
1 DDepartment of Electrical Engineering, KU Leuven, Leuven, Belgium 2 Department of Development and Regeneration, KU Leuven, Leuven, Belgium MIT Media Lab and EECS, Cambridge, MA, USA
Abstract
Multimodal learning holds promise for richer information extraction by capturing dependencies across data sources. Yet, current training methods often underperform due to modality competition, a phenomenon where modalities contend for training resources leaving some underoptimized. This raises pivotal question: how we address training imbalances, ensure adequate optimization across all modalities, and achieve consistent performance improvements to multimodal data? This paper proposes the Multimodal Competition Regularizer (MCR), inspired by mutual information (MI) decomposition designed prevent the adverse effects of competition in multimodal training. Our key contributions are: 1) game-theae theortht framework that adaptively balances modality contributions by encouraging each to maximize its informative role in the final prediction 2) Refining lower and upper bounds for each MI term to enhance the extraction of both task- relevant unique and shared information across modalities. 3) Proposing latent space permutations for conditional MI estimation, significantly improving computational efficiency. MCR outperforms all previously suggested training strategies and simple baseline, clearly demonstrating that training modalities jointly leads to important performance gains on both synthetic and large real-world datasets. We release our code and models at http://ggituub..ooooottttee
transition from unimodal
Introduction
Exploiting multimodal data has made significant progress, with advances in generalizable representa- tions and larger datasets enabling solutions to previously unattainable tasks [28 33 30
39
45
44
46 52 55 66). However, studies indicate that jointly trained multimodal data often utilized subop- timally, underperforming compared to ensembles of unimodal models, jointly trained modalities, or even the best single modality [56] (64. The expectation that adding new modality should improve performance, assuming independent errors and above-chance predictive power [16], is frequently contradicted in practice.
Huang et al. (20) attribute this issue to modality competition, where one modality quickly minimizes training error, misdirecting and suppressing the learning of others. To counteract this effect, monitor- ing each modality's contribution during training and applying corrective measures is crucial. To this
Coorespondence to: konstantinos. .ontras@kuleuven..b
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Uy: U,CMM = ((x2)
C2 = ((x2;
Accuracy vs. Ratio of U, U,, and
Accuracy (%6
H(X))
H(X2)
+0 Zeg cin
OGH
H(Y)
PCR
13:22
11-23
9-38
Xx)())))) - 1(x)x221V)
U, Us
Figure 1: (Left) Illustration of the conditional mutual information (CMl) terms, CMI I(X1; (XI;Y X2) and CMI2 T(X2;Y Y Xi), representing the unique contributions (U,, U2) each modality. The shared task-relevant information (S) is defined ;( [[x;; YY. (Right) Accuracy on a synthetic dataset designed to induce multimodal competition. We vary the ratio of unique information from modality (U1) to shared information (S), while keeping the contribution of modality (U22 constant. As the imbalance increases (moving right on the x-axis), the performance of most methods drops. The standard Joint Training (Singleloss) approach shows steep decline, highlighting its vulnerability to modality competition where one modality dominates and suppresses the other. In contrast, method, MCR. demonstrates greater robustness by maintaining the highest accuracy and exhibiting the slowest performance degradation. See Section 4.1 for more details.
end, several balancing strategies have been proposed | 6
61
59 69 19 (88. Some ignore a modality's 's contribution beyond its independent unimodal performance, while others address this by measuring output differences under input perturbation, but at the cost of increased sensitivity to these perturbations and significant computational overhead. Moreover, it is crucial examine whether enhancing one modality's influence on the output does come the expense of others, as this could undermine overall performance.
Given these challenges, how and effective learning across modalities?
efficiently regularize multimodal competition ensure balanced
This paper introduces loss function encouraging the exploration of task-relevant information across modalities, the MULTIMODAL COMPETITION REGULARIZZR (MCR). The approach incorporates the following key contributions:
1. MI Bounds:
We decompose joint mutual information into task-relevant shared and unique compo- nents, using refined lower and upper bounds promote informative signals and suppress noise.
2. Game-Theoretic Modality Balancing: We frame modality interaction through game-theoretic framework. allowing each modality to adjust its contribution throughout training.
3. Efficient CMI Estimation: We introduce latent-space perturbations for low-cost conditional MI estimation, avoiding repeated full-model passes.
We extensively evaluate MCR synthetic datasets and several established real-world multimodal benchmarks, including action recognition on AVE [51] and UCF [47]. emotion recognition on CREMA-D [4) human sentiment CMU-MOSI (65), human emotions CMU-MOSEI [67], and egocentric action recognition on Sonething-Somethiin [14). Our results demonstrate that MCR outperforms all previous methods and simple baselines across various datasets and models, improving multimodal supervised training.
2 Problem Analysis and Related Work
Consider a dataset of N independent and identically distributed (i.i.d.) datapoints sampled from distribution D, where each datapoint has M modalities X = (X1...., Xaa) and target Y,. Our goal is to learn a parameterized function f : X; + Y., where 0 denotes all the model's learnable parameters. The unimodal encoder for each modality is defined as fm. Xm Om + Zms encoding input X,, into latent representation Zn, The fusion network sa: [Z....., 7M); G V, predicts Y, from the latent representations, do the unimodal task heads fcn. Zmi Pcm.. - Ym.. Model

---

families are defined as, unimodal models for m [1, M] modalities:
Fum Smm (Xmi Om, Ocm) Sem (Sm (Xmi mm)= Ocm) and for multimodal models:
(1)
F f(x;0) Se (Ifi (X1;01)... fas (XM: ;0))
(2)
For simplicity, we continue our analysis with M = 2, focusing on models with two modalities.
2.1 The limitation of supervised multimodal training
In supervised learning, the goal is to learn representations Z1 fi and Zz = f2 when fused via fa([Z1, Za]). yield accurate predictions. This is achieved by minimizing the task loss or, equivalently, by maximizing the MI between the fused representation and the target:
arg max
[(fc[Z21 Z2)); Yi).
= (X1:1). 2x (Xp:00)
accessible modality, limiting the contribution of others. This leads to mutual information being dominated by one modality, e.g.. I[fa(ZZ) Z2)); s I(Z1; Y) with 7(Z2; Y Zi) 2 0, indicating that Z2 adds little once Zi is learned. See Appendix A.I for an illustrative experiment and Appendix A.2 for a formal definition of the resulting generalization gap.
During training, models often over-rely on the stronger
This kind imbalance is well-known in single-modality learning, where dominant features can overshadow others, harming generalization. Regularization techniques like 4 / l2 penalties and dropout promote balanced feature use [40]48] I but their adaptation to multimodal settings is nontrivial. For example, applying modality-speciin dropout [62] offers limited benefits [42]. The core challenge
remains: how to effectively regulate interaction and competition between modalities.
2.2 Related Work
Prior research has explored various strategies for multimodal learning, ranging from simple unimodal and ensemble-based approaches to more sophisticated methods for balancing modality contributions. Unimodal training optimizes each modality separately, while ensemble methods combine unimodal predictions without additional training. Joint training optimizes all modalities under a single-loss objective but does not explicitly ensure sufficient training for each modality. To address this, Multi- Loss [54 introduces additional unimodal task losses, and MMCosine (63] equalizes modality influence by standardizing features and weights. Pre-trained unimodal encoders are often used, either with frozen weights (Uni-Pre Frozen) or fine-tuned jointly (Uni-Pra Finetuned). Other adaptive strategies include MSLR (64], which adjusts learning rates based on unimodal validation performance, which modulates gradients by comparing unimodal performance across modalities, and MLB 27 which combines unimodal task losses and modulates gradients from both unimodal and multimodal objectives, MMPareto [57] that mitigates gradient conflicts between modalities by equalizing the contribution of unimodal and multimodal gradient and D&R [59] suggest new strategy where modalities that overfit get their part of the network partially reweighted with the initial weights of the training.
OGM 42
Most of these methods assume distributional independence and measure modality contributions through unimodal performance, which can be a limited indicator, missing cases where modality correlation is crucial. Other approaches estimate influence based on prediction differences after perturbations 229 21 10] AGM [29] uses zero- mero-mmskiig Shapley values directly optimizing them as unimodal predictors, Wei et al. [58] use a permutation- based Shapley values and resampling of the training set to affect the training, while other methods address similar problems by introduc- ing perturbations such as Gaussian noise [10] or task-specific augmentations [2 32). However, perturbation-basee approaches increase the network's 's sensitivity to the chosen perturbations and hinder scalability due to their higher computational demands.
A line of work keeps unimodal training as the primary strategy. MLA 69 uses shared task head and dynamic weighted summation during validation, while ReconBoost [9 alternates unimodal updates with agreement and diversity regularization before finetuning the ensemble. However, these approaches avoid multimodal training the earlier steps to mitigate conflicts, yet overlook the potential benefits of direct multimodal interactions in those steps.

---

Multmooall Competition Regularizer
Key Cantribution: Augmentations biinnt space
LUCK LuiDD* Lcon* LEEB
LNIPO LLIIP,, + LLSIpD
Video Encoder
[2
P
Exno, - -J50((
Fusion Network
2 (el
Lussa -15D(V.
2. Audio Encoder
lexal2.23
Figure 2: Multimodal Competition Regularizer (MCR): The diagram illustrates the MCR frame- work, which mitigates modality competition in multimodal learning. Raw data (X₃ and X2) are encoded into latent representations (Z₁ and Z2), which are then permuted to create Zι and 2:2 and the paired combinations. These combinations are passed through the Fusion Network to produce pre- dicted outputs (Y, Y1, Y2). The comparison between predictions reveals each modality's contribution. For example, if Y ~ Y,, it shows that X1 has little impact, and the model relies Xg. The MCR loss includes three components: LMIPD maximize the Jensen-Shannon divergence (JSD) between task output and permuted modality predictions. LCou aligns modality representations, while LCEB penalizes inre-irrelevant information by reconstructing back to the latent space.
3 Multimodal Competition Regularizer
Multimodal competition arises when a model trained on multiple modalities prioritizes one, leading to over-reliance and reducing the contribution of others. This imbalance limits the model's ability to fully utilize all available information. In this section, we introduce LMCR, set of loss components designed to address multimodal competition. Each component of the loss is motivated by the following MI decomposition:
X(x;;2;; I(X;;Y x2) + 1(2;; Ix+++
X2)
X2 1Y)
(4)
Rele-Relevant Unique Information of each moddlitty-&Iirr
Shared Information
Task-Irellvant,
Shared Infermation
This decomposition is illustrated by the Venn diagram in Figure The CMIs I(X;;Y X2) and [(x;; Y [X1) capture mopality-specific information for predicting the target. Maximizing them with the Mutual Information Perturbed Difference (MIPD) loss, LMIDD. which assesses each modality's contribution via output variations under input perturbations (elaborated in Sec. 3.3) and encourages the extraction of modality task-relevant features. The third term, [(x1: quantifies shared information between modalities. Maximizing it with a contrastive loss, CCon. aligns representations and leverages their shared information effectively [22 41 46). The final term, T(X;; X2 Y). represents aask-irrelevant shared information. Penalizing it with the conditional entropy bottleneck (CEB) [ and the corresponding loss LCEB to filter out irrelevant information, focusing the model on features relevant to the downstream task. Each term has corresponding loss, as illustrated in Figure 2 forming the regularizer with three key losses:
LMCR LMIPD + LCon + -CEB
(5)
3.1 Approximating MI Terms
I(X;; Y X2): To approximate each CMI and capture the unique contribution each modality, the MIPD serves surrogate function, measuring how input perturbations affect the model's output. By comparing predictions with and without these perturbations, MIPD estimates how much information each modality provides. If modality is crucial, altering its input should significantly change the output, revealing its importance.
Estimating the CMT directly through T(X1: Xx, = HY X²) intractable. Instead we use the MIPD as a lower bound, defined as:
H(Y X1,X2) is typically
MIPD(X,;Y X²) I(X;; Y X₂) I(X;; Y X2) S I(X1Y (x)).
(6)

---

Game Strategies
Collaborative (Video Encoder: Importance,)
Independent
Vo, Suron 2
Greody
(Vide Encoder, Importancet.)
Key Contribution: Video Encoder benefits
Encoder
Itseff by reducing Pa, Aarsp, le. greedy strategy
Xi
Audio
somuk:
2/3
Fusion Network
Gradient
Y.R
Luroo
-JSD(Y,8)
Encoder
Importanccy
Vo, Carro
Figure 3: This figure illustrates key aspect of training process, showing how competition strategies between modalities are applied. The gradient multiplier adjusts the video encoder's response Audio Importance (Importance,). When k the video encoder enhances Importanceg: at = 0, it remains neutral, and at k = =1, it competes by reducing Importance, to prioritize its own (Importance,). This reflects the principle that increasing the importance of one modality can reduce the importance of the other.
where the perturbed version of modality X, denoted X,, Interpreting MI via entropy, each CMT can be expressed as the difference of the log probabilities with and without the perturbations:
MIPD(X1; X2) = H(Y X2, X1) H(Y X2,
E
E 31-0(=1)
ply
F2, @1)] + log 22)
y--(())
21
Instead of this log-likeling log-likelihood ratio, we use the symmetrically bounded Jensen-Shannon divergence (JSD) 34 prevent training instabilities, leading to the following:
LMIPD, MIPD(XX;Y X2) = EE [JSD(p(V (1..1)) 22, (1)].
w--(0)
(8)
Similarly, LMIPD, can be computed symmetrically.
I(X;; X2) The next MI term measures how much information the two modalities share, capturing the common patterns between the modalities and aligning the representations of these shared aspects. We exploit the available label information employing the supervised contrastive loss Lcom (25
(11,, x
Lcon
E
log Σ (1 x2.
31 (1111) 2 (p(2)1)))
22 <p22
where s is the critic function, which, in our case, is the exponential dot product. Minimizing the LCo., maximizes lower bound on both the MI between the two modalities and the CMI terms:
1(x2; ()) + [(x;; X(22) + 21 (X2; X1) 2 log N - -oo
(10)
As N increases, the bound becomes tighter, while the bound samples (same class datapoints). More details are provided in Appendix B
affected by the number of positive
I(X;;; Y):
The final term captures irrelevant shared information between modalities, and minimizing an upper bound on this ensures the model retains only task-relevant content. For this pur- pose, exploit the idea of Conditional Entropy Bottleneck (CEB) LCEB [7]. targeting superftuous information in multimodal representations via reconstruction loss. A small reconstruction head, (Zi, Z2). predicts back the latent space, effectively filtering out irrelevant content:
h: Y; On.
LCER E (z1), fa(x2) 6,)1
(11)
at #1."2_//ppp1... m2..))
The exact derivation of this loss term can be found Appendix A.6 Penalizing irrelevant information has been shown to enhance calibration and robustness [8), but it must be carefully evaluated, as it can introduce constraints that may hinder overall performance.
--y (v/ E y * y). where y is the set of target labels.
5

---

3.2 The Game of Multimodal Fusion
We adopt game- same-theoretic approach to balance the terms of the proposed LMPD. The key idea is that increasing one modality's 's importance (e.g.. via MIPD,) can inherently reduce the other's s (e.g., M) Thus, an underutilized encoder (with parameters 0.) can boost its relevance both by minimizing LMIPD, and by maximizing LMIPD_.. This twofold strategy helps prevent suppression of weaker modalities. We frame LMPD as a game where each encoder (player) selects a strategy. minimize, maximize, ignore. Figure iillustrates how the video modality, via hyperparameter k. can choose to assist, ignore, or diminish the audio modality. Each encoder applies this logic selectively as formalized below:
Vo, LMIPD AM CMIPD, + & Vo, LMIPD2);
(12)
Voz fMIPD AM CMIPD, + k V LMIPD,
(13)
where XM is a Lagrange multiplier, and & E { 1-10,, 0, 1} sets the modality's 's strategy:
Collaborative (k = 1): All modalities work together to increase each other's contributions. The LMIPd terms are applied across all parameters, resulting in min LMIPD.
Independent (k = 0): Each modality focuses on maximizing its own contribution by optimizing solely its respective LMIPD term, leading min LMIPD,- 8.
Greedy (k: 1)): Each modality seeks to maximize its own contribution by: 1) minimizing its own LMIPD term, and 2) maximizing the LuMPD terms of other modalities, resulting in min-max game, min max LMIPD.
0, 9,
Following the results in Appendix A.8 we adopt the greedy strategy as default, as showed the most consistent performance in our setting.
3.3 Perturbations
To assess the importance of modality X1, we define LMIPD, which captures changes in the model's output when X₁ is perturbed (i.e., (X1, X2) vs. (X1, X2)). Instead of traditional input-space perturbations, which can be computationally expensive and task-dependent, we apply a within-batch permutation de Uniform(P) the latent space, yielding X; = (e())). This approach avoids extra forward passes and reduces computational and memory overhead. Further analysis of this technique and comparisons with prior methods are provided in Appendix A0 and A.14
The complete algorithm is presented in Algorithm | with an extension of LMCR to M modalities described in Appendix A.7 In Appendix A.9 we analyze various combinations of loss components, revealing that penalizing task-irrelevant information benefits models with extensive SSL pretraining but proves detrimental for those without it.
4 Experiments
4.1 Synthetic Dataset
We create a scenario where mutual information varies, showcasing modality competition. While various factors contribute to such phenomenon, we focus modality informativeness imbalance to motivate our approach.
Data: We generate task-irrelevant information for each modality by sampling N1, (O, I) and the 5-class label Y, from a uniform distribution Y, ~ Uniform(5). Each modality is converted into high- nigh-dimensional vector using fixed transformations, similar to Liang et al. (32). We relate both modalities to the label through linear relationship: X₁ = .N + Y, and X2 = N₂ + Y.. Data points distributed in such way that either both modalities contain label information (Shared Information) or only one of the modalities (Unique Information). In cases where only one modality contains label information, the other modality is defined X1 N2 and X₂ N₂ respectively. We
'The parameter set under the loss indicates where backpropagation applies.
'The notation -i refers to the rest of the modalities except

---

Algorithm Multimodal Training with MCR
Input: Training dataset D with modalities X1, X2, ..... XM, labels Y, multimodal model F.
initialized unimodal encoders O,, reconstruction model h. Auni, AM Lagrangian coefficients:
1: for each batch (X1, M,, , Yt) of each epoch do
2: Compute Ltask 1X...... XM),Y, Yi) and Crask Auai Em Feask ( (X Xi)
3: Extract (Z1..., ZM) from f(X1,.., (xX1,..Xx, X
4: Assess the Lcon with Eq. 2 aand CCEB with Eq.
5: Sample de permutations and compute permuted pairs on the latent space
6: Pass each pair through the fusion model fe to get predictions Ym with modality m permuted
7: Compute LVPP using Eq. 13 and selecting by strategy (default: Greedy)
8: Determine LMCR from Eq.
9: Update the model parameters based on
Ltotal = Ltask + Luni + LMCR
10: end for
vary the percentage of data points with shared and unique information to analyze model performance under different conditions.
Figure D shows the performance on synthetic data, comparing our method (MCR) with several baselines. As the shared information S among the modalities decreases, and the unique information of one modality U1 increases while the U2 remains constant, we observe performance drop for all methods. MCR maintains the highest accuracy across all combinations, demonstrating the slowest decline and highlighting its robustness to such imbalance.
Results:
4.2 Real- World Datasets
Datasets:
We explore several real-world datasets, primarily with video, optical flow, audio, and text modalities, that either exhibit significant imbalance among modalities or serve standard multimodal benchmarks. Detailed descriptions are provided in Appendix A.. with brief summaries below:
1. CREMA-D An emotion recognition dataset with 91 actors expressing 6 distinct emotions.
AVE 51 A collection videos with temporally aligned audio-visual events across 28 categories.
UCF An action recognition dataset of real-life YouTube videos.
4. CMU-MOSEI [67 Multimodal sentiment analysis dataset with 23k monologue clips.
5. CMU-MOSI 65 Multimodal sentiment analysis dataset with over 2k YouTube video clips.
SomeethingSSomething (V2) [4: 220k clips of individuals performing 174 hand actions.
Models: We employ variety of models and backbone encoders to examine the behavior of both smaller-scale models trained from scratch and larger, more complex models pretrained with self- supervised learning (SSL). This combination demonstrates that method is effective in both limited data scenarios without pretraining and in cases with ample data where the goal is fine-tuning. We utilize ResNet-18 and small-scale Transformers (from thousand to 20M parameters) alongside state- of-the-art models such as Swin-T'F (35) and Conformer [13], incorporating backbone encoders like Wav2Vec2, HuBERT, and ViViT, resulting in model sizes approaching 200M parameters. Detailed model configurations for each dataset are provided in Appendix A.4 and experimental details in These choices aim to bridge the gap between theoretical work and practical application.
Appendix A.5
Results: Table reports accuracy comparisons across baseline methods on our evaluation datasets. We highlight two key observations:
a) Most prior methods, including recent multimodal approaches, fail outperform simpler alterna- tives such as Ensembles or unimodal encoders (either frozen or finetuned). While some of these methods partially address modality competition, they often fall short of effectively leveraging multimodal data.
b) MCR is the only method that consistently surpasses all baselines across datasets, model archi- tectures, number of modalities, and task types (classification and regression), with an exception AVE with Conformer model. This consistent ad advantage highlights MCR's ability to balance
7

---

Table 1: Performance comparison of MCR against prior multimodal training methods across six datasets. MCR consistently achieves top results across various modality combinations (V-A, V-T, V-A-T, V-OF) and model architectures. MOSI and MOSEI use three modalities and are trained as regression tasks (converted to binary accuracy); the rest are classification tasks. All baselines were rerun under our evaluation protocol to ensure fair comparison and address data leakage issues identified in previous evaluation setups.
CREMA-D
AVB
UCF ResNet V-A
MOSI
MOSEI
Stb-Sth Swin-TF V-OF
ResNet
Conformer V-A
ResNet V-A
Conformer V-A
Transformer V-A-T
Method
V-A
V-T
V-
V-A-T
54
V: 64.8.
Unimodals
A: 60.6.n
.5.
A: 76.0..
4..
62.2.6..
45.7
75,5,
A: 76.5..
A: 30.3...
53.
A: 64
OF: 50.8
4:6
T: 72.
T: 78.9u
Ensemble
71,7.a
84.6.>
70.5
88.4.
$2.8.
70.5.a:
67.2.m
78.4..
77.2
64.6
Joint Tmining
62.6..
74.6
66.7.t
82.2
47.7us
73.0.::
73.6um
80.5..
80.8..
57.5..
Multi-Loss
69.2...
70.1
86.3.m
51.1-s
72.1.-
23.6...
80.0..
80.2-
61.5..
Uni-Pre Frozen
72.4...
85.0..
72.2
87.2.
53.0...
73.3.,
72.7at
79.9.s.
79.8.n.
64.0..
Uni-Pre Finctuned
73.3...
82.4.ca
72.5.0
86.5...
53,5...
73.1
73.7mm
80.3...
80.3
62.1.az
MSLR [4
56.5..
17.
67.3at
81.0..
50.9.>>
MMCosine
59.3..>
74.0.s>
65.0As.
83.8
47.3.
X
OGM 2N
65.6,..
82.4.
67.3ur
79.7.t
51.8.>
73.9..
79.7...
57.8,
AGM
69.3...
78.5.>
68.4.1:
85.3
51,0..s
74.0...
73.9.
79.3....
80,2
56.6.a
MLB Hr
71.9.m
85.2..>
71.6um
86.7
52.2.
72.4.
74.2
80, x
80.5..
61.6.e
RoconBoost
19
69.0uz
84.8..
68.4..
86.
50.2..
x
56.1.a
MMPareto
57
69.0..-
83.8,,
73.0...
87.4...
51.4.n
73.4.
73.7ux
79.3...
79.5
59.2.
D&R
59
G..
85.0...
72.3.0
91.0..
49.3...
61.7.t:
MCR
76.1
85.7
73.4
88.8.
55.2.
75.2
76.5
80.8
81.1
65.0
x method not applicable to regression tasks; & method not applicable to trimodal inputs; result reported.
modality contributions during training under diverse settings, positioning it as a strong and generalizable approach for multimodal learning.
4.3 Analysis of Multimodal Error
Methodology: To understand how MCR improves over existing methods, we perform post-hoc error analysis by categorizing each sample based on the correctness of the unimodal predictions. Specifically, we consider four groups: (1) both unimodal models are correct, (2) only the first is correct, (3) only the second is correct, and (4) both are incorrect. This breakdown allows us to compare how different multimodal methods behave across these categories and identify whether gains arise from selective reliance unimodal cues or from synergistic integration.
Results:
The error analysis of Figure reveals key strengths and limitations of the proposed method. MCR consistently outperforms other methods in routing information favouring both modalities in the cases that only one of the modalities is correct maintaining competitive performance on both of them and in the case that all modalities correctly predict the label. This demonstrates MCR's 's ability to effectively route information to the appropriate modality, in line with its design choice to model task-relevant information through the mutual
and control training via this information terms.
However, MCR does not exhibit significant gains in capturing synergetic information in the datapoints that all unimodal models fail, underperforming relative to AGM and MLB. This suggests MCR excels in routing decisions but may be less effective at leveraging synergies across modalities when all individual models falter. This trend holds across other datasets (see Appendix A..II, with the exception of MOSI, where MCR improves synergy. Our initial assumption that concurrent modality training would foster synergy thus did not hold in practice.
8

---

CREMA-D ResNet
Ensembre
Jonn Taining
AGM
MLB
Settmmottl soot
Guei
Canet
Cand
Word
Dami
War Garel
Canul Crest
Models
Caveel
Wwng
Cwerd Gueed Cases
CREMA-D Conformer
MAp Modell
Wirr
Wong
4.
Whong
71
11.3
Whr1g
Correct
11.5
Cornet
30.3
Carred
26
50. Carmec
30
46.3
Comrec
16.
53.0
Cells
Balh Sullo
64h Models
Figure 4: Error comparison on the CREMA-D dataset across unimodal and multimodal models (MCR, Ensemble, Joint Training, AGM, MLB). Each matrix summarizes model performance based on unimodal prediction correctness. MCR performs best when least one unimodal branch is correct (brown box), effectively preserving modality-specific signals. However, AGM and MLB outperform MCR when both unimodal predictions fail, in the "Both Wrong" (purple box), indicating stronger synergy in those edge cases. Trends across other datasets are shown in Appendix A.11 with MOSI being a notable exception where MCR also excels synergy.
5
Discussion
This paper examines the challenge of modality competition in multimodal learning, where certain modalities dominate the training process, resulting in suboptimal performance. We introduce the Multimodal Competition Regularizer (MCR). a novel approach inspired by information theory, which frames multimodal learning game where each modality competes to maximize its contribution to the final output. MCR efficiently computes lower and upper bounds to optimize both unique and shared task-relevant information for each modality. Our extensive experiments show that MCR consistently outperforms existing methods and simple baselines on both synthetic and real- world datasets, providing more balanced and effective multimodal learning framework. MCR paves the way for fulfilling the long-standing promise of multimodal fusion methods to achieve performance that surpasses the combined results of unimodal training.
We explored different game strategies and observed that directly encouraging competition between modalities in the overall objective function positively impacts performance, detailed in Appendix A.8 Future work could investigate more refined strategies to enable individualized and adaptive decisions for each modality to unlock greater performance gains.
Lastly, we conduct post-hoc error analysis found both in Section 4.3 and Appendix A. examining overlaps between the errors of multimodal models and their unimodal counterparts. The results show that MCR excels at routing decisions to the correct unimodal information but does not promote synergetic behavior accordingly, compared to previous methods. Our initial assumption that the simultaneous progress unimodal encoders during training would naturally enhance synergy was not supported in practice, highlighting the need for future work to promote this behavior explicitly. Finally, this analysis highlights the potential for performance improvements through enhanced multimodal training, motivating further exploration in this area.

---

Acknowledgement
This material is based upon work partially supported by the FWO Research Project "Task- and device-agnostic Artificial Intelligence (AT) for EEG analysis" (G046925N); C2 "Dissecting agitation in dementia by multimodal sensing (DADS)" (C2M/23/053);
2-00222-TTHHLHH "Artificial intelligence- based Parkinson's 's disease risk assessment and prognosis "A-PPROGNOSIS"" under Grant Agreement No. 101080581; and the Flemish Government (Al Research Program).
References
[1] Cem Anil, James Lucas, and Roger Grosse. Sorting out lipschitz function approximation. In International Conference on Machine Learning, pages 291-301. PMIR, 2019.
[2] Anurag Arnb, Mostafa Dehghani, Georg Hcigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: vidco vision transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6836 6846, 2021.
[3] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for supervid learning of speech representations. Advances in neural information processing systems, 33: 12449-12460, 2020.
[4] Houwei Cao. David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd- sourced emotional multimodal actors dataset. IEEE transactions on affective computing. 5(4):377-390) 2014.
[5] Chenzhuang Du, Jiaye Teng, Tingle Li, Yichen Liu, Tianyuan Yuan, Yue Wang, Yang Yuan, and Hang Zhao, On uni-modal feature learning in supervised multi-modal learning. arXiv preprint arx5v:2305.01233, 2023.
[6] Yunfeng Fan, Wenchao Xu, Haozhao Wang, Junxiao Wang, and Song Guo. Pmr: Prototypical modal rebalance for multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 00039- 2023.
[7] Ian Fischer. The conditional entropy bottleneck. Entropy, 22(9):999, 2020.
[8] Ian Fischer and Alexander A Alemi. Cch improves model robustness. Entropy, 22(10): 1081, 2020.
[9] Naotsuna Fujimori, Rei Endo, Yoshihiko Kawai, and Takahiro Mochizuki. Modality Modaaitt-speciffcc learning rate control for multimodal classification. In Patten Recognition: 5th Asian Conference, ACPR 2019, Auckland, New Zealand, November 26-29, 2019, Revised Selected Papers, Part II 5, pages 412-422. Springer, 2020.
[10] Itai Gat, Idan Schwartz, Alexander Schwing, and Tamir Hazan. Removing bias in multi-modal classifiers: Regularization by maximizing functional entropies. Advances in Neural Information Processing Systems, 33:3197-3208, 2020.
[11] Jort F Gemmeke, Daniel PW Ellis. Dylan Freedman, Aren Jansen, Wade Lawrence, Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and luman-labeled dataset audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776-780. IEEE, 2017.
[12] L. Goncalves, S.-G. Leem, W.-C. Lin, B. Sisman, and C. Busso. Versatile audiovisual learning for handling single and multi modalities in emotion regression and classification tasks. ArXiv e-prints (arXiv:2305.07216 pages 1-14, May 2023. doi: 10.48550WarXi..2355.7222
[13] Lucas Goncalves, Seong- Leem, Wei-Cheng Lin, Berrak Sisman, and Carlos Busso, Versatile audio-visual learning for handling single and multi modalities in emotion regression and classification tasks, arXiv preprint arXiv:2305.0722 072.11 2023.
[14] Raghav Goyal. Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitae et The" something something" vidco database for learning and evaluating visual common sense. In Proceedings of the IEFE international conference on computer vision, pages 5842-5850, 2017.
[15] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang. Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augne augmented transformer for speech recognition. arXiv preprint 2rXiv:2005.08100, 2020.
10

---

[16] Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions puttern analysis and machine intelligence. 12(10):993-1001 1990.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 770-778, 2016.
[18] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451 13460, 2021.
[19] Cong Hua, Qianqian Xu, Shilong Bao, Zhiyong Yang, and Qingming Huang. Reconboost: boosting can achieve modality reconcilement, In Proceedings of the 41 International Conference on Machine Learning, (4. JMI.R.org 2024.
[20] Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang. and Longbo Huang. Modality competition: What makes joint training of multi- modal network fail in deep learning (provalyy......yy.... In International Conference on Machine Learning, pages 9226 9259. PMLR, 2022.
[21] Baijun Ji, Tong Zhang, Yicheng Zou, Bojie Hu, and Si Shen. Increasing visual awareness in multi- modal neural machine translation from an information theoretic perspective. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6755-6764, 2022.
[22] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le. Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916. PMIR, 2021.
[23] Emmanouil Kariotakis, Nikolaos Sidiropoulos, and Aritra Konar. Fairness-regulated dense subgraph discovery. arXiv preprint arXiv:2412.02604 2024.
[24] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vjjaynnarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kineties human action vidco dataset. arXiv preprint or 1705:/705.0055500 2017.
[25] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:186611 18673, 2020.
[26] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1rXiv:1412.6980. 2014.
[27] Konstantinos Kontras, Christos Chatzichristos, Matthew Blaschko, and Maarten De Vos. Improving multimodal learning with multi-loss gradient modulation. arXiv preprint 20Xv:24055.07930. 2024.
[28] Konstantinos Kontras, Christos Chatzichristos, Huy Phan, Johan Suykens, and Maarten De Vos. Core-sleep: A multimodal fusion framework for time series robust imperfect modalities. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2024.
[29] Hong Li, Xingyu Li, Pengbo Hu, Yinuo Lei, Chunxiao Li, and Yi Zhou. Boosting multi-modal model performance with adaptive gradient modulation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 22214-22224, 2023.
[30] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision- vision-language understanding and generation. In International Conference Machine Learning, pages 12888- 12900. PMLR, 2022.
[31] Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslic Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal representation learning. Advances in neural information processing systems, 2DBI(1 2021.
[32] Paul Pu Liang. Zihao Deng, Martin Q Ma. James Y Zou, Louis-Philippe Morency, and Ruslan Salakhut- dinov. Factorized contrastive learning: Going beyond multi-view redundancy. Advances in Neural Information Processing Systems, 36, 2024.
[33] Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations & trends in multimodal machine learning: Principles, challenges, and open questions. ACM Computing Surveys, 14(10):1-42, 2024.
[34] Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145 151, 1991.
11

---

[35] Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[36] Ilya Loshchilov and Frank Hutter. arXiv: 1,00017,1100000,000000
Decoupled
weight decay regularization.
arXiv preprint
[37] Scott M Lundberg and Su-In Lec, A unificd approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
[38] Kevin P Murphy. Probabilistic machine learning: introduction. MIT press, 2022.
[39] Arsha Nagrani, Shan Yang. Anurag Amab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks multimodal fusion. Advances in neural information processing systems, 34:14200-1444 14213, 2021.
[40] Andrew Y Ng. Feature selection, 1 I vs. 12 regularization, and rotational invariance. In Proceedings of the twenty-firs international conference on Machine learning. page 78, 2004.
[41] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint / 2018.
[42] Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal lcarning via on-the-fly gradient modulation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8238-8247, 2022.
[43] Huy Phan, Oliver Y Chén, Minh C Tran, Philipp Koch, Alfred Mertins, and Maarten De Vos. Xsleepnet: Multi-view sequential model for automatic sleep staging. IEFE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5903-5915, 2021.
[44] Gorjan Radevski, Marie-Francine Moens, and Tinne Tuytelaars. Revisiting ppatio-temporal layouts for compositional action recognition. arXiv preprint arXiv:2111.01955 2021.
[45] Gorjan Radevski, Dusan Grujicic, Matthew Blaschko, Marie-Francine Moens, and Tinne Tuytelaars. Multimodal distillation for egocentric action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
[46] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 7748-8763, PMLR, 2021.
[47] Khurram Scomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 1212 2012.
[48] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929-1958, 2014.
[49] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 13/2.6199 2013.
[50] Mingxing Tan and Quoc Lc. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference machine learning, pages 6105-6114, PMLR, 2019.
[51] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio visual event localization in unconstrained videos. In Proceedings of the European conference on computer vision (ECCV), pages 247-263, 2018.
[52] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language soquences. In Proceedings of the conference. Association for computational linguistics. Meeting, volume 2019, page 6558. NIH Public Access, 2019.
[53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems. 30, 2017.
12

---

[54] Valentin Vielzeuf, Alexis Lechervy. Stéphane Pateux, and Frédéric Jurie. Centralnet: multilayer approach for multimodal fusion. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0-0, 2018
[55] Bokun Wang, Yang Yang, Xing Xu, Alan Hanjalic, and Heng Tao Shen. Adversarial cross- modal retrieval. In Proceedings of the 25th ACM international conference on Multimedia, pages 154-162, 2017.
[56] Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard?? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16695-12700 2020.
[57] Yake Wei and Hu. Mmpareto: boosting multimodal learning with innocent unimodal assistance. In Proceedings of the 41st International Conference on Machine Learning, ICML'24, JMLR.org, 2024.
[58] Yake Wei, Ruoxuan Feng. Zihe Wang, and Di Hu. Enhancing multimodal cooperation via sample- level modality valuation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27338-27347, 2024.
[59] Yake Wei, Siwei Li, Ruoxuan Feng, and Di Hu. Diagnosing and re-learning for balanced multimodal learning. In European Conference on Computer Vision, pages 71 86. Springer, 2024.
[60] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault. Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen. Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the tf-t---rtttttttttt natural language processing. In Qun Liu and David Schlangen, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10. 38653//1/0200.mmllppp//
[61] Nan Wu, Stanislaw Jastrzebski, Kyunghyun Cho, and Krzysztof J Geras. Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks. In International Conference Machine Learning, pages 24053-24055. PMLR, 2022.
[62] Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer. Audiovisual slowfast networks for video recognition. arXiv preprint arXiv:2001.088 2020.
[63] Ruize Xu, Ruoxuan Feng, Shi- Shi-Xiong Zhang, and Di Hu. Mmcosine: mudti-modal cosine loss towards balanced aisia- fine-grained learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[64] Yiqun Yao and Rada Mihalcca. -pdelit learning rates for effective multimodal additive late- fusion. In Findings of the Association for Computational Linguisties: ACL 2022, pages 1824-1834. Association for Computational Linguistics, 2022.
[65] Amir Zadeh, Rowan Zellers, Pincus, and Louis-Philippe Morency. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint 1666v:16060062559
2016.
[66] Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis- Phillpp Morency. Tensor fusion network for multimodal sentiment analysis. arXiv preprint 1eX.::1707.0725500 2017.
[67] AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-PPilipee Morency. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting the Asxociation for Computational Linguistics (Volame I: Long Pupers), pages 2236-2246. 2018
[68] Kaipeng Zhang. Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE signal processing letters. 23(10)11999 1503, 2016.
[69] Xiaohui Zhang, Jaehong Yoon, Mohit Bansal, and Huaxiu Yao. Multimodal representation learning by alternating unimodal adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27456 27466, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We believe the abstract and introduction accurately reflect the ideas, contribu- tion and the scope of our presented research.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have conducted series of experiments in the error analysis section that provide the limitations of our method, which are also included in the discussion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: We believe we provide the full set of assumptions and the complete and correct proofs for our methodology in the sections that this applies.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have included all the experimental details needed to reproduce our results and claims.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: In the Appendix we provide a full description with the instructions needed to reproduce our experimental results, from the experimental details, dataset preparation and time needed. We believe these will sufficient but after the review process we will make our code publicly available too.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis. cc/
gublic/guides httpe//uuiiiee/eeeeuiiissse Crode@Sbmissionooll...
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips cc/ htcc/pullcc/aae
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have included all the necessary details the Appendix make sure someone can follow the training and test details. Since we faced issues with previous works, we believe this is an important part of our contribution.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper report error bars in terms of standard error (STE) for the different folds which involve cross validation the datasets without fixed test set and multiple runs with different seeds in the case of fixed test set. We do not report ste in two cases, in Something-Somethiee
dataset, due to its size running multiple folds was computationally prohibitive, and on some instances that we tried due to the size of the test set, the STE was also negligible, the latter reason lead us to skip the reporting of deviation on MOSEI and MOSI. To the rest of our results, always include STE reporting.
16

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the corresponding implementation details section we do report about the maximum computational resources used and the execution time.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't ' make it into the paper).
paper (e.g., preliminary or failed experiments that http:ppbbiiitttiicccccddeeeeeee
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: To the best of our knowledge we align completely with the Code of Ethics of NeurlPS.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
17

---

Answer: [NA]
Justification: We believe our work is foundational research which is not tied to a specific task or use and therefore there are no direct societal impacts the worked performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (c.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We believe that our paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes)
Justification:
We have properly cited and acknowledged the data, code and models used in work.
Guidelines:
The answer NA means that the paper does not existing assets.
18

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing or research with with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for rescarch with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
19

---

Our work does not involve crowdsoureing research with with human subjects.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our work does not include LLM as core method for developing our research.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components. wwtprriissccc/ooooee occ/ooneernccc/22255///////////e
Please refer to our LLM policy for what should or should be described.
20

---

A Supplementary Material
A.1 Evidence of limitations of supervised multimodal training
Identifying the instances where supervised multimodal training collapses is often easier than resolving the issue itself. Nevertheless, is crucial to understand both these limitations and why simple solutions might not suffice. To explore this, we utilize a ResNet-18 18 [17] backbone on the CREMA-D dataset, employing audio and video the two modalities. These modalities are concatenated just before the final linear layer. We measure multimodal performance at the end of each epoch and, simultaneously, perform linear probing on each modality to evaluate their individual contributions throughout training.
In Figure 5 it is clear that the performance of the multimodal model aligns closely with that of the audio modality alone, suggesting that the model heavily relies audio while neglecting the video modality. This lack of exploration results in the video modality remaining at chance-level accuracy throughout training. As a result, the model fails to leverage any information available in the video modality and performs significantly worse than an ensemble of the unimodally trained models.
Accuracy Multimodal Model during Training
wmma
Audo now Priting
200 220
60
Epochs
Figure 5: Accuracy of the multimodal model on the CREMA-D dataset across training epochs, showing the performance of the full multimodal model (blue) and individual modality linear probing for audio (orange) and video (green). The dashed red line represents the accuracy of a unimodal ensemble model, highlighting how the model's over-reliance the audio modality negatively impacts the utilization the video modality
A.2 Multimodal Competition Error
Multimodal competition occurs when the network primarily optimizes for one modality, leading to a decline in generalization. One modality dominates the reduction in training error and limits gradient feedback the other modalities. Huang al. (20) proved that in late fusion models (i.e. 6, = 9), each modality has a probability of causing multimodal competition. They define the phenomenon in terms of the correlation I'm Ea max ( ml W each input modality and the weights, W, of the output layer for each corresponding class cl. If a modality dominates the competition, it causes the other modality to maintain its initial (i.c. before training) correlation levels. Such correlation levels do not have specific target, unlike other problems where equality is the goal (23), making the solution less straightforward. We proceed with this analysis on late-fusion models by introducing the generalization error resulting from the effects of multimodal competition.
Definition A.1. Definition of Multimodal Competition Error (MCE): Let D2 and T2 represent the correlations of modalities Z₁ and Z: with the output. If, after training, T2 >>> T2, and modality X₂ has predictive power greater than randomness (R(X2) < R(random)), while making emrors independent from those of X, (Eie) - e2] = O, where é and e2 are the errors of X, and X2, respectively), then for trained model f and its optimal solution f*, there exists multimodal competition error e, such that:
<2 R(f*) - (fmp(f") and c S R(f) Remp(f)
(14)
where R(-) (E.Luu(YY.)) and Remp(.) denote the generalization and empirical model risks, and Lrask represents the corresponding task loss, which could vary depending on the objective, such as cross-entropy for classification or mean-squared error for regression. A higher indicates stronger effect of multimodal competition, implying that the dominance of one modality significantly impacts the model's generalization. These incqualities are empirically observed through improvements in
21

---

generalization achieved by adjusting the training objectives to address multimodal competition. without altering the model or the data. Numerically estimating e would require knowledge of the optimal solution f", which is typically unavailable. Lastly, transitioning from the late-fusion models
of Huang et al. (20 we need to extend this definition by replacing correlation with MI to consider also non-linear statistical dependencies.
A.3 Datasets
is an emotion recognition dataset with audio and video modalities. It features a diverse group of 91 actors, covering a wide range of ages, ethnicities, and genders. To ensure consistency, each actor is positioned at an equal distance from the camera, expressing six distinct emotions: Happy, Sad, Anger. Fear, Disgust, and Neutral. In alignment with methodologies from prior video frames are sampled at 1 fps, selecting 3 consecutive frames, while audio segments are sampled 22 kHz, capturing 3 seconds that correspond with the chosen video frames. Audio analysis utilizes window size of 512 and a step size of 353 samples for Short-Time Fourier Transform (STFT), creating log-Mel spectrograms. For advanced models, our methodology aligns with Goncalves et al. [12), incorporating audio signals sampled at 16 kHz and utilizing pre-calculated facial features. Unlike previous approaches [42 29 6), our dataset division follows Goncalves et al. 12]. excluding actor overlap between training. validation, and test sets. We report standard deviation (std) across folds for consistency.
CREMA-D [4]
studies 42 29
AVE [51]: contains 4143 videos across 28 event categories with a wide range such as frying food or playing guitar, each with temporally labeled audio-visual events of least 2 seconds. Following [6] video segments where the event occurs are sampled at fps for 4 frames, with audio resampled at 16 kHz using CREMA-D's STFT settings. AVE provides predefined training, validation, and test splits. Our std is derived from three random seeds on the same test set.
features real-life action videos from YouTube in 101 action categories, expanding on UCF50. We select samples that include both video and audio modalities, narrowing our focus to 51 action categories. Data preparation mirrors that of the AVE dataset. For model evaluation, we utilize the 3-fold split offered [47), reporting the std across these folds.
UCF101 447:
Something-Sometthnng (V2) [14]: presents 220,847 video clips where individuals execute 174 distinct, object-agnostic hand actions, spanning a wide range of simple hand movements without reliance specific objects. This extensive collection markedly exceeds the data volume of prior datasets. In alignment with Radevski et al. [45]. we integrate Optical Flow (OF) as the additional modality for capturing dynamic motions. We sample videos at fps. retaining 16 frames per sample. Our data preparation adheres to the pipeline outlined in Radevski et al. 45 Results are reported based on three random seeds on the same validation set.
CMU-MOSI [65] and -MOSEI [67] datasets serve as benchmarks for multimodal sentiment and emotion analysis. MOSI comprises 2, video clips with video, audio and text modalities. MOSEI expands on this with around 10x more YouTube movie review clips from 1000 different speakers. Both datasets annotated with sentiment scores (-3 to 3) and following previous works [52] we employ metrics such as 7-class accuracy, binary accuracy, F1 score, mean absolute error, and correlation with human annotations for model evaluation. We use for both datasets the aligned versions following (31). The reported std is derived from three random seeds on the same test set.
A.4 Models: Backbone Unimodal Encoders
In line with previous research 29 42 6 63], our initial experiments adopt ResNet-18 [17] as the unimodal encoder for handling both video and audio modalities in the CREMA-D, AVE, and UCF datasets. These models are randomly initialized and incorporate adaptive pooling to accommodate diverse input dimensions. For the CMU-MOSEI dataset, we exploit 5-layer Transformer (53)
similar to previous works 32 31).
We extend our investigation to include a larger, pre-trained set of unimodal encoders that are optimally suited for each specific modality. This selection process designed to rigorously assess whether state-of-the-ane models exhibit susceptibility to the same phenomena under investigation. We exploit these encoders on the CREMA-D dataset. Following [33 on CREMA-D, we deploy the first 12 layers of the Way2Vec2 3 60 model with self-supervised pretrained weights for speech recognition,
22

---

allowing the Wav2Vec2 model to be finetuned. For the video modality, we extract the facing bounding boxes with the multi-task cascaded convolutional neural network (MTCNN) face detection algorithm [68] and afterward the facial features of every available frame exploiting EfficientNet-B2 (50) as frozen feature descriptor. The extracted audio features and pre-calculated facial features are further refined using 5-layer Conformer [15], initialized from scratch. The total model size is 183M parameters. Although the model includes additional components, we refer to it as "Conformer" for simplicity.
For the AVE dataset, we use a similar architecture to CREMA-D, where each branch utilizes an advanced, pretrained model aligned with AVE data, followed by 5-layer Conformer. For the video branch, we use the ViVIT model [2] pretrained on Kinetics [24]]. and for the audio branch, the HuBERT [18] model pretrained on Audioset [T]] resulting in model with 232M parameters. Ensuring the audio pretraining included non-speech data was important for the pretraining to be beneficial. We also refer to this model as "Conformer," although it incorporates different large pretrained models in this instance.
In the case of the Samnethin-SSmeethingg dataset, our methodology builds upon the insights presented by Radevski et al. 45], which highlight the importance of modality-specifie processing in multimodal tasks. For both video and optical flow data, we adopt the Swin Transformer (35] as the backbone encoder to each modality. This state-of-thh-an architecture excels in capturing hierarchical and spatiotemporal features through its shifted window attention mechanism.
A.5 Experimental Details
In this section, we outline the necessary details to reproduce our experiments. Across all datasets, we follow consistent procedure: we first determine an appropriate learning rate (Ir) for the unimodal models by testing several candidates until finding one that works across both modalities. While this step could be avoided by exploiting parameter-specifie learning rates, we expected stability implications which we aimed to avoid. For all the experiments of the same dataset/model pair, we use the same hyperparameters, except when fine-tuning pretrained encoders, where we apply learning rate scaled one magnitude lower.
All models are optimized using Adam 26 with a cosine learning rate scheduler and a steady warm-up phase, except for the Samething-SSmettinne
dataset, where we use Adaw [36]. Early stopping is applied for all models, with maximum epochs set to 100 for ResNet and Transformer models, 50 for Conformer models, and 30 epochs in total for Swin Transformers without early stopping. Batch sizes are adjusted based on computational resources, with ResNets and Transformers both using a batch size of 32, Conformers using 8, and Swin-TF using 16. These settings ensure balanced performance and efficient training across all experiments. In several instances, initializing the encoders with pre-trained weights from unimodal training proved beneficial. This was particularly effective for datasets without precomputed features and models without access to larger-scale SSL pretraining. We use different learning rates (Ir) and weight decay (wd) values across experiments, tailored to each dataset and model. For ResNet models, we use Ir 1e-3 and wd 1e-4 for CREMA-D, while both AVE and UCF use Ir = 1e-4 4 and wd = le 1e-4. MOSEI on and three modalities, the hyperparameters Similarly, for Conformer models, we set Ir = 5e-5 and wd = 5e-6 for CREMA-D, while AVE uses Ir 1e-4 4 and wd le 4. Finally, for Swin-TF models trained the Homething-Somethinn dataset, we configure Ir- = 1e-4 4 and wd = 0.02.
4. For Transformer models, including MOSI and consistent Ir 4 and wd 1e-4.
Each of the previous methods includes its own set of hyperparameters, typically just one, with some exceptions such as MSLR or D&R, which requires additional parameters. For each dataset/model combination, conduct brief hyperparameter search, ensuring an equitable number of trials across methods. Due to the extensive list of hyperaarameters, we will provide detailed configurations for each experiment in our GitHub repository. The repository link will be included here following the double-blind review process.
Lastly, all of our experiments run on single GPU with different nodes being used for different experiments. For the largest ones we utilized H100 with 80Gb vram to run the experiments of Sth-Sth which required the longest of all up to 48 hours per run. For the rest, we would have from some minutes on the smallest experiment up to 4-5 hours for the datasets CREMA-D, AVE and UCF depending on the GPU and the available RAM.
23

---

A.6 Bounding task-irrelevant information via CEB
Our objective to maximize the information shared between modalities that is irrelevant to the supervised task. Directly estimating the conditional mutual information [(x1; Y) is challenging in high- high-dimensional settings, so we relax the objective by penalizing both the individual and shared irrelevant information, leading to the decomposition:
X2 Y) H( Y) (X2
Y) ( Y)
(15)
- 1(x;;X2 Y) + H(1 Y) + H Y) = (X1, Y)
(16)
We lower-bound the conditional joint entropy using CEB 7 as follows,
H( Y) - -p(e). [logp(21, y)]
(17)
Bp(n) [log gog y) (L 18) (1 Iv))
(18)
=2(e) [log (211 y)]
(19)
Assuming a conditional Gaussian model g( 22 y) = -N ((21, ((x1, p(y) a²11, we define μ(y) as a deterministic function h Y:0h 0n Then, the conditional entropy term is upper-bounded as:
(Z1, Z2) that predicts a target joint representation from Y.
H X2 Y) < -Bp(z1zaa)) [log g011,02 y)] x Ep(a lli(()))
h(y; (xy; 1I (20)
Thus, we approximate the entropy term with an MSE loss, encouraging (x1, to deviate from any deterministic function when it is not informative for the task. Maximizing H(X, Y) and (X2 Y) incentivizes each modality to retain information that task-independent, aligning with the goal of minimizing X2 Y) and suppressing lask-irrelevant alignment between modalities.
A.7 MCR on M modalities
In this section provide the analysis of MCR for M number of modalities. In that case, the total mutual information I(X1,..., YM;Y) can be decomposed into contributions from individual modalities and their subsets
T(X1, XM;YY XC(X ).S00
Σ T(SYY (X,.... T(X,,) I(X XM Y), (21)
where S S [X,,.,, } represents a subset of all modalities, excluding the empty set (S * 0). The term (X1,.., XM. S denotes the complement of S, capturing the set of modalities not included in S. The mutual information I(S; Y {X,, Xw) } \ S) quantifies the information shared between the subset and the target variable Y. conditioned on the remaining modalities. This formulation ensures that all modalities, along with their combinations, are accounted for in the summation. It comprehensively captures interactions at every granularity, from individual modalities ([S) 1) to the full set of modalities -M)
While experimented with reducing the number of terms by considering only the cases where |S] = 1, we observed a slight improvement in performance when including all terms for three modalities. However, as the number of modalities increases, might be beneficial to sub-select and exclude certain terms to mitigate the computational burden and prevent an overflow of terms.
A.8 Ablation Study Game Strategies
We perform an ablation study to test our hypothesis that framing multimodal competition regulariza- tion game benefits the model by avoiding destructive loss interactions each backbone encoder. Table compares among the three strategies: Collaborative, Independent, and Greedy. The results show that, across the models allowing backbone encoders to maximize their own CMT term and concurrently minimizing the others (Greedy strategy) consistently yields the best performance. This result demonstrates that framing multimodal models as competing modalities using game-theoretic principles in the loss terms can be beneficial in balancing these loss terms.
24

---

Table 2: Ablation Study: Game Strategies
Comparison of model accuracy on CREMA-D dataset for different game strategies: Collaborative, Independent, and Greedy, using both ResNet and Conformer backbones.
Collaborative min LMIPD
Independent min LMIPDX,
Greedy
Dataset/Model Setting
min max LMPPDX, 8-1
8
8:
CREMA-D ResNet + MCR
73.4+30
76.0120
76.2
CREMA-D Conformer MCR
82.9
82.6#26
85.7
AVE ResNet + MCR
67.9+25
72.5+10
73.4
AVE Conformer MCR
88.9
88.611111
88.8
UCF ResNet MCR
55.1
14.8
55.2
MOSI V-T TF MCR
73.7
73.6+17
75.2
MOSI V-A-T TF MCR
75.7
74.4+1.1
76.5
MOSEI V-T TF + MCR
80.41005
80.4
80.8104
MOSEI V-A-T TF MCR
80.7
80.8
81.1
Sth-Sth Swin MCR
64.9
64.9
65.0
A.9 Ablation Study Loss Components
Table 3 presents the model's performance comparison when different loss components are applied. The models utilize pretrained initialization: ResNet with unimodal pretraining and Conformer with SSL. Two key observations can be made:
1. The concurrent exploitation of both LMPD and LCon is yielding consistent improvement. Ex- ploiting them separately leads to smaller improvement for Loon and even a decline for LMIPD- suggests that alignment in the latent space between the modalities is necessary for the permutations be effective.
2. The LCEB term, which penalizes task-irrelevant information, improves the Conformer model's performance, likely due to its pretraining on large, unlabelled datasets that introduce such irrelevant information. In contrast, for the ResNet model, where pretraining already focuses on task-related information, the LeER term does provide additional benefits.
Table 3: Ablation Study: Regularization Components Accuracy (%) ResNet and Conformer models across datasets with different combinations of MCR components: LMIDD, LCous and LCEB- Results indicate that combining CMID and LCoa is crucial for improvement, while CCEB does not benefit all models.
MCR Components
ResNet AVE
Conformer
SwinTF Sth-Sth
LMPP
Lcom
LOER
CREMA-D
UCF
CREMA-D
AVE
73.4+25
71.1+14
50.0+20
84.1
87.9 87.9-111
64.74a.
74.1:29
72.1105
49.4
83,9
8..
64.8102
73.4121
72,6106
54.8
84,5103
88.7-14
64.7.0.1
76.2
73.3
55.1
84.5:00
88.7
64.8
75.6219
72.1
54.7211
85.7
88.8-1.0
65.0
A.10 Ablation Study Perturbation methods
To estimate the importance of modality, we perturb one modality while keeping the other fixed and observe the change in the model's output. Several prior methods have proposed ways to do this, but each comes with trade-offs. Additive noise has been used to maximize output variance as a proxy for functional entropy [10], though this increases sensitivity to noise, conflicting with goals such as smoothness and robustness 49 1 Task-specific augmentations (21 32 rely on
handcrafted strategies that may not generalize across domains or modalities. Zero-masking strategies used for approximating Shapley values [29] are theoretically grounded but often unreliable in high-
25

---

dimensional settings 37 and require multiple forward passes, increasing computational and memory demands.
Based on these previous works we explore three types of perturbation methods to analyze their impact on the performance of MCR: noisy perturbations, zero-masking, and permutations. Each method was applied in different spaces (input space and latent space) or within the batch structure to determine how effectively MCR can leverage these perturbations to enhance multimodal learning. In Table we summarize the different approaches we examine and in Table 5
we present the results of an ablation study comparing the performance of MCR under these perturbation techniques across multiple datasets: CREMA-D, AVE, UCF, MOSEI, and MOSI.
Table 4: Overview of the approaches examined for the permutation methods.
Noise in the Input Space
Adking noise directly to the input features of each modality, simulating realistic data corruption. For its implementation follow [10]
Shapley Input-Space Perturbations
Following the approach of [29]. Shapley -nnduced values are used to determine the importance of inpul modalities. Applying noise to the latent representations and encouraging robustness at the feature extraction level.
Noise in the Lutent Space
Zeros in the Latent Space
Zero-masking latent representations to disrupt one modality. Permuting data points within the batch to disrupt alignment.
Within- Batch Permurations in the Latent Space
Table 5: Ablation study comparing different perturbation methods for MCR across multiple datasets.
The table shows the performance of MCRwhen combined with various perturbation techniques, including input or latent space noise, Shapley values in the input space, and within-batch permutations.
Method
CREMA-D
AVE
UCF
MOSEI
MOSI
MCR with Noise Input-Space
75.3
12.1
54.6208
80.5
74.7
MCR with Shapley Input-Space
73.6225
72.6299
55.5
79.82us
74.31az
MCR with Noise Latent- Space
73.6211
72.6204
54.5
80.
72.3
MCR with Zero Latent-Space MCR with Permutations Latent-Space
73.6218
73.3
54.5204
79.6204
73.6124
76.1
73.3
55.2218
80.8
75.2217
We observe that Shapley-based input-space perturbations show competitive performance, particularly in datasets like UCF, MOSI, and MOSEI, while woise-based methods (both input and latent spaces) achieve reasonable performance, they consistently underperform other techniques. While input-space perturbations could be viable option, they significantly increase computational complexity, as they require an additional forward pass through the typically large unimodal encoders for each sample. This limitation, which we analyze in Appendix[A.14 [A.14] makes them less favorable as a practical solution. Finally, these findings support the choice of permutations as the preferred perturbation method, while suggesting that further exploration of alternative strategies could potentially Icad to even greater improvements.
The semantic meaning of this perturbation depends on whether the permuted sample shares the same label as the original. the labels match, the perturbation is semantically valid and can be seen as an implicit augmentation. In this case, large output change indicates that the model may be relying on spurious or unstable features within the modality. If the labels do not match, the resulting input is semantically inconsistent and can be interpreted as out-of-distribution. If the model's output is insensitive to such perturbation, this suggests that the modality is being ignored. Conversely, a sensitive reaction may indicate an overreliance on features not robust to semantic shifts. Therefore each category of permuted samples contributes differently to the final output. In practice, we apply both semantically consistent and inconsistent permutations during training. This choice introduces minimal computational overhcad and appears to slightly improve convergence stability.
We note that the gradients Ve, LMIPD, and Vo, LMIDD, can negatively impact model robustness depending on the type of perturbation applied. When perturbations yield -fitt--disttitttinn unimodal inputs, such as sero-masking or additive noise, the resulting gradients may encourage the model to learn spurious patterns. In contrast, our main experiments use within-batch permutations, which preserve the in-distribution structure of each modality. Under permutation as the perturbation method, the LMIPD formulation remains symmetric regardless of which modality perturbed, and the gradients contribute constructively to learning in all branches.
26

---

A.11 Analysis of Multimodal Error
We extend the error analysis from Section by comparing unimodal and multimodal predictions in Figure 6 The results echo the pattern seen in CREMA-D (Figure 4) MCR excels when at least one unimodal model predicts correctly, but still trails MLB and AGM when all unimodal models fail. An exception is the MOSI dataset, where MCR performs well in synergy, even with three modalities.
A.12 Statistical Importance
We assess the statistical significance of the performance differences between our method and each baseline using the Wilcoxon Signed-Rank Test, applied to per-dataset average results. To control for multiple comparisons, we apply the Holm correction to the resulting p-values. We consider results significant if the adjusted p-value is below = 0.05. Full results are shown in Table 6
Table 6: Wilcoxon Signed-Rank Test results (two-sided) comparing method each baseline across datasets. Statistically significant comparisons after Holm correction (a = 0.05) are bolded.
Comparison
Raw p-value Holm-adjusted p-value
MCR vs Ensemble
0.00195
0.02148
MCR vs Joint Training
0.00195
0.01074
MCR vs Multi-Loss
0.00195
0.00716
MCR vs Uni-Pre Frozen
0.00195
0.00537
MCR vs Uni-Pre Finetuned
0.00195
0.00430
MCR vs OGM
0.00781
0.00955
MCR vs AGM
0.00195
0.00358
MCR vs MLB
0.00195
0.00307
MCR vs ReconBoost
0.03125
0.03438
MCR vs MMPareto
0.00195
0.00269
MCR vs D&R
0.15625
0.15625
Greedy Collaborative Greedy vs Independent
0.00977
0.00977
0.00195
0.00391
LMIPD + Leon + LEER vs No Reg
0.03125
0. 0.12500
LMIPD LCon. + LCEB vs LMIPD
0.04311
0.08623
LMIPD + LCon + LCEB vs Loon
0,06789
0,09052
LMIPD LCon + LCEB vs LMIPD Loon.
0.84375
0.84375
Dynamics of MIPD Components
To illustrate the learning dynamics of the MCR regularizer, loss components during training. Figure 7 plots the two MIPD terms, corresponding to the video and text modalities, from a training run on the MOSI V-T dataset.
analyze the evolution of its
These MIPD terms, which serve as proxies for the unique contribution of each modality, exhibit a dynamic, alternating behavior. The fluctuations reflect shifts in which modality is more influential on the fused output at different stages of training. This plot visualizes MCR's mechanism for actively balancing modality importance, preventing from consistently dominating the other. It is important to note that while this alternating pattern the desired behavior, its specific form and prominence vary across different training illustrative example of the dynamic interplay MCR encourages. This prevents static dominance by single modality, which key to mitigating modality competition.
and datasets. Therefore, this figure is presented as clear,
27

---

ResNet
ANttmodaa Mode!
mone
WWog
Iewr 20.
Woog
Werr
1033
Comez
511
Cornec
23.5
1133
Comees
Addo
1.4
203
Adda
Vole
Nalo 13t 200
Ai
Wele
Oeven
Adlo
5,30
000
Conformer
Traising
Nutmooa Model
Wrong
(Wrr
Woog
41
Werr
Comeet
commet
25
12.9
572
Comeect
11.8
12.
Commet
23.4
Adko Vile A0D
Aale
Nib Veler
Adn
Aall
Oeneet
Auloo
Green
ResNet
Ensembte
Training
Nutmmsta Mozel
ioon
224
WWr
3.7
2-5
12:5
1.0
Whong
11.5
46
Werr
9.4
Connet
Conwe
Comee
6.7
14.0
10.0
Comwe
13.7
37.1
Auki
Zall
14b
Comen
MOSI VTT Transformer
MCR
Enserbtle
joint Teiting
AGM
M.B
(Mo Norttrraaa Nodtl
smong
Whogg
Whor
Whang
Whor
20
357
342
261
251
7.0
15.0
269
+.80
vita
Dwnt
Mi Video-Thet
vide
Deriet Grien
MOSI VTT Transformer
MCR
Enserttle
jont Teiling
AGM
M.B
Wulttmoddh Modeel
mmve
4.5
32
32
42
26
WWWV
27
455
18.1
43
72
51
34
40
10.1
WWWng
53
311
10
4.3
3.0
10.2
Caree
MOSEI V-T Transformer
MCR
Ensembtee
Joint Trsiing
ASM
MLB
No Buotmooa Hote
imong
Weogg
Wtorr
7.8
Whong
Weorg
1.5
06
187
2.2
71.
5.0
555
255
+200
350h
Cares
Vide
MOSEI V-AAA Transformer
MCR
Ensembtee
Joint Talking
AGM
MLB
mong
53
15
31
34
67
WW0
33
2.0
74
12
33
54
1.5
14
71
W0003
5.
3.4
3.4
7.2
Werr
6.0
3.6
311
17
250g Pith 7ol 700.
Come
Video-Tot.Aaala
28

---

Sth-Sth V-OF Swin-TF
NGR
Brsemble
Jont Traning
AGM
NLB
74
Careect
143
1.
14.3
41.3
3.0
123
17.3
21.0
36.1
39.0
Audio Video Unimodal Models
Cancee
Figure 6: Error comparison matrices across pairs of datasets and models, comparing unimodal predic- tions with multimodal models trained using various methods, including Ensemble, Joint Training, AGM, MLB, and MCR for the datasets AVE, UCF, MOSI, MOSEI and Something -Something (Sth-Sth). Each column of the confusion matrix represents cases where both unimodal predictions are incorrect, where only one correct, and where both are correct. The results highlight that MCR consistently performs well in cases where at least one unimodal prediction is correct. Additionally, MLB and AGM in many instances outperform MCR in discovering synergetic information, which refers to the "Both/All Wrong" column, highlighting current limitation of MCR.
With MCR
Without MCR
MIPD; (Video Modalty) (Text Modality
MIPD, (Mideo Modialty)
MIDD; (Text Mocalty)
Salding iigertan (NAVO) 64
Compettion cauass Functusiore
0.-
2000
Stepe
Figure 7: The plots show the two MIPD losses (MIPD, and MIPD2) for each modality, with (left) and without (right) regularization. With regularization, the losses exhibit an alternating pattern, demonstrating MCR's s ability to dynamically explore multiple modality contribution combinations and prevent collapse into single modality. Without regularization, the text modality initially contributes more strongly, but the video modality eventually outperforms and dominates.
A.13 Reproduction Challenges for MIA
using the authors' officially released code. Despite carefully replicating the training procedure, were unable reach close to the reported performance. Specifically, our reproduced accuracy results on our five different settings as follows: CREMA-D (ResNet) 61.7-41, CREMA-D (Conformer) 1 AVE (ResNet) 44,5 AVE (Conformer) 72.4 and UCF (ResNet) 47.9-1.4 These results are substantially lower than those reported in Table 1 of the original paper. We suspect the discrepancy may be due to missing implementation details. We include these findings in the interest of transparency and to support ongoing efforts toward reproducibility in multimodal learning.
We attempted to reproduce the results reported for the MLA method [69
A.14 Computational Speed and Memory Analysis of Perturbations
The computational load imposed by any sample that requires an additional pass can be divided into the encoders f1, f2 and the fusion network fa. The encoders have a computational cost of cost per sample, and the fusion network has a cost of cost per sample. Thus, the total computational complexity ((M + 1) * N (costene + cost.)), where M is the times we draw noisy samples to which we add the non-perturbed batch and N is the batch size. If we now use permutation samples that can be directly drawn from the latent space, the additional computational complexity reduced to O(N cosl (M + 1)+ N +cosl.). Compared to the necessity of supervised forward pass this cost.) computational burden. most state-of -hete--rrttt models, each modality encoder is significantly larger than the fusion network, resulting in costenc >> coste-
translates to an addition of O(M
29

---

In such networks, permutations can have almost negligible additional computations. The memory footprint follows similar pattern.
B Proof of Supervised Contrastive Loss as Lower Bound
We consider the supervised contrastive loss with ei being the critic function and rewrite it as follows:
E log (111, Σjel (11, (111,,,3 v2,)
Ccon (Cn, (21,
Σ EED
(22)
REP.
where P: {p D 11 = ) In supervised contrastive learning, the presence of multiple positive samples turns this into a multi-label problem, unlike traditional noise contrastive estimation (NCE) methods [41] which typically assume only positive sample. By taking the version of supervised contrastive learning with the expectation over the positives outside of the log, we can interpret each classification as average of classifiers, with each classifier focusing identifying positive samples.
of the
For each positive sample Ps, aim to derive the optimal probability of correctly identifying that point, denoted = p. This is done by sampling the point from the conditional distribution p(ay, F W/) while sampling the remaining points from the proposal distribution p(æ2,). This approach mirrors the technique used in InfoNCE [41] and leads to the following derivation:
p 11. Mi)
p(12)
1ED,
p(d = pl T1, v1)
(23)
Σ EEED p(2 I1,, Mi) LED,Y)
II p(22,)
p (21,066)
p(#2p)
(24)
p(azz, (2) p(waa)
ΣjED
24 is proportional to ai x plaz, pla2,1 us) plaup)
The optimal value for the critic function & in Equation MI between the variables can be estimated follows:
The
p(zz, .1.) p(22g))
Oppe-
E p-P:
log
(25)
D
p(azg (m M1 p ((2 )
Σ
p(22, 21) p(mag)
JED
p(12,)
p(1z 11. M4)
E
log
(26)
p P~P:
p(12 11, Vi)
p(12,)
JED
p(12,)
p(12, 11 Mi)
E iwD
E pr/P
log
+ p p(
(N - (N-1)
1)
(27)
x1,)
JED
p(1r2,
p(222))
E inrD
E log
1 +
(N (N 11
(28)
p~PP
pl222
Ii, Mi)
p(xz,))
E (wD
E log pr-P.
N
(29)
11 _2(2
vi)
log N [[x;; X1, Y), using MI properties 38 Chapter 6.3.4]
(30)
log N - I(X;; Y|X)) X1)
(31)
Therefore, by taking both sides of the contrastive loss to predict X₂ from X, and X, from X₂ we derive to T(X2; YJX,) [Xx,; Y|X2) + I(X2; X1) 2 log I. This trivially also holds for other s that obtain a (orse(higher) Lcon- Simarly to InfoNCE, the bound becomes more accurate as N increases, while due to the the term TP. is not affected by the number of positive pairs.
30


Reviews:
[{"review_id": "1m8hb3jsL2", "reviewer": "Reviewer_dDxc", "summary": "The manuscript proposes the Multimodal Competition Regularizer. It is a training strategy designed to address the problem of modality competition in multimodal learning, where certain data modalities dominate training and suppress others. MCR introduces a game-theoretic regularization framework that adaptively balances the contribution of each modality by decomposing mutual information into unique and shared task-relevant components. The method is evaluated across a variety of synthetic and real-world multimodal datasets. It outperforms existing baselines and prior approaches in both classification and regression settings.", "questions": "The game-theoretic framework currently uses discrete strategies with fixed values (-1, 0, 1) to balance modality contributions. Have the authors considered using continuous or learnable weights for this optimization, which could enable a softer and potentially more adaptive balance between modalities during training?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. It is a novel and interesting game-theoretic regularization method for balancing contributions across modalities in multimodal training. It addresses a well-known challenge of modality competition that is often overlooked in the literature.\n2. The experimental evaluation is thorough and convincing, as the authors validate across six diverse, widely used datasets.\n\nWeaknesses:\n1. No statistical testing between the best model and baselines in Tables 1, 3, and 5. One can use, for example, Wilcoxon and correct for multiple comparisons with the Holm correction.\n2. In Table 1, some models are without std.", "was_revised": true, "final_justification": "I am maintaining my score as Accept.", "rebuttal": "We thank the reviewer for pointing this out the absence of standard deviation in Table 1 for MOSI, MOSEI, and Something-Something was due to those results being reported from a single run. We have now addressed this by rerunning our training pipeline on each method three times, using the same dataset splits (as provided by each benchmark). The corresponding dataset descriptions have been updated to reflect this protocol and ensure full reproducibility. The revised results, including standard deviations, are shown below:\n\n|                 | MOSI                  | MOSI                  | MOSEI                 | MOSEI                 | Sth-Sth               |\n| :-------------- | :-------------------- | :-------------------- | :-------------------- | :-------------------- | :-------------------- | \n|                 | TF    | TF | TF    | TF | Swin-TF   |\n| Method          | V-T       | V-A-T     | V-T       | V-A-T     | V-OF      |\n| Unimodals       | V: 54.1$\\pm$3.7 A: 53.7$\\pm$0.6 T: 72.1$\\pm$3.3 | | V: 64.8$\\pm$0.2 A: 64.4$\\pm$0.2 T: 78.9$\\pm$1.7 | | V: 61.4$\\pm$0.2 OF: 50.8$\\pm$0.1 |            \n| Ensemble        | 70.5$\\pm$2.1          | 67.2$\\pm$1.5          | 78.4$\\pm$0.7          | 77.2$\\pm$0.6          | 64.6$\\pm$0.2          |\n| Joint Training  | 73.0$\\pm$1.3          | 73.6$\\pm$1.3          | 80.5$\\pm$0.2          | 80.8$\\pm$0.3          | 57.5$\\pm$0.1          |\n| Multi-Loss      | 72.1$\\pm$0.4          | 73.6$\\pm$2.9          | 80.0$\\pm$0.7          | 80.2$\\pm$0.5          | 61.5$\\pm$0.1          |\n| Uni-Pre Frozen  | 73.3$\\pm$1.8          | 72.7$\\pm$1.6          | 79.9$\\pm$0.5          | 79.8$\\pm$0.3          | 64.0$\\pm$0.2          |\n| Uni-Pre Finetuned | 73.1$\\pm$2.3          | 73.7$\\pm$0.7          | 80.3$\\pm$0.4          | 80.3$\\pm$0.2          | 62.1$\\pm$0.2          |\n| OGM             | 73.9$\\pm$1.1          | $\\otimes$             | 79.7$\\pm$0.6          | $\\otimes$             | 57.8$\\pm$0.5          |\n| AGM             | 74.5$\\pm$1.5          | 73.9$\\pm$1.9          | 79.3$\\pm$0.4          | 80.2$\\pm$0.3          | 56.6$\\pm$0.4          |\n| MLB             | 72.4$\\pm$1.7          | 74.2$\\pm$1.7          | 80.1$\\pm$0.5          | 80.5$\\pm$0.4          | 61.6$\\pm$0.2          |\n| ReconBoost      | $\\otimes$             | $\\otimes$             | $\\otimes$             | $\\otimes$             | 56.1$\\pm$0.3          |\n| MMPareto        | 73.4$\\pm$1.0          | 73.7$\\pm$0.6          | 79.3$\\pm$0.6          | 79.5$\\pm$0.8          | 54.6$\\pm$2.8          |\n| D&R             | $\\times$              | $\\times$              | $\\times$              | $\\times$              | 61.2$\\pm$0.7          |\n| MCR             | **75.2$\\pm$1.7** | **76.5$\\pm$1.4** | **80.8$\\pm$0.4** | **81.1$\\pm$0.4** | **65.0$\\pm$0.1** |\n\n\nThank you for the suggestion on the statistical testing, we have now run Wilcoxon and Holm correction on the paired comparison of our method with each one based on the different model/dataset averaged performance. We conclude that one D&R that achieves better on AVE-Conformer produces non-significant p values. The comparisons are included in a new appendix section with table as shown below:\n\nWe have now performed Wilcoxon signed-rank tests and Holm corrections to compare MCR against each baseline across the averaged results for each combination of dataset/model. Compared to the rest of the multimodal training methods, we see that MCR is statistically significant, except for D&R, which outperforms MCR on the AVE-Conformer setup. We have also conducted \nseparate statistical tests on Tables 3 and 5, but we did not find the variations to be significant under the given tests. The full results are presented in the table below and included in a new appendix section. \n\nWe have performed Wilcoxon signed-rank tests with Holm correction to compare MCR against each baseline across the averaged results for each dataset/model combination. MCR shows statistically significant improvements over all multimodal training methods, except for D&R, which outperforms MCR on one of the setups, namely the AVE-Conformer. We note, however, that the number of evaluation points (9 in total) is relatively limited, which constrains the strength of such statistical claims. Despite that, our results are shown in the table below and included in a new appendix section.\n\n| **Comparison** | **Raw $p$-value** | **Holm-adjusted $p$-value** |\n| :------------------------------------------- | ----------------: | --------------------------: |\n| **--- MCR vs Baselines ---** |                   |                             |\n| MCR vs Ensemble                              | **0.00195** | **0.02148** |\n| MCR vs Joint Training                        | **0.00195** | **0.01074** |\n| MCR vs Multi-Loss                            | **0.00195** | **0.00716** |\n| MCR vs Uni-Pre Frozen                        | **0.00195** | **0.00537** |\n| MCR vs Uni-Pre Finetuned                     | **0.00195** | **0.00430** |\n| MCR vs OGM                                   | **0.00781** | **0.00955** |\n| MCR vs AGM                                   | **0.00195** | **0.00358** |\n| MCR vs MLB                                   | **0.00195** | **0.00307** |\n| MCR vs ReconBoost                            | **0.03125** | **0.03438** |\n| MCR vs MMPareto                              | **0.00195** | **0.00269** |\n| MCR vs D&R                                   | 0.15625           | 0.15625                     |\n\nFor Tables 3 and 5, the ablation studies were conducted on a subset of the dataset/model combinations, which does not provide enough samples to support reliable statistical testing. Once the ablations are completed across all 9 datasets, we plan to include statistical significance testing in the camera-ready version to ensure consistency with our main results."}, {"review_id": "1LFQ5wCPsp", "reviewer": "Reviewer_QAwA", "summary": "The paper aims to addresses the common “modality competition” problem in multimodal training. The authors propose the Multimodal Competition Regularizer (MCR), a framework that first uses mutual information analysis to divide each representation into three parts: task-relevant information that is unique to one modality, task-relevant information that is shared by several modalities, and shared noise that is irrelevant to the task. Afterward, the method applies three complementary loss functions(LMIPD, LCon and LCEB) together with a gradient-scheduling strategy inspired by game theory, ensuring that every modality makes an appropriate contribution. The authors also present empirical results on a controlled synthetic dataset and several real-world benchmarks.", "questions": "N/A", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\n1. I am not a specialist in game theory, the way to merge mutual-information decomposition with a game-theoretic gradient scheduler is new to me and appears conceptually innovative.\n2. Estimating conditional mutual information through latent-space permutation is a good practical solution as it is computationally efficient, offering a feasible alternative to more expensive input-space perturbations. \n3. The method shows strong results, and the experimental discussion is detailed and systematic, covering both synthetic and multiple real-world datasets. \n\n**Weakness:**\n\n1. As the authors themselves note, when every unimodal branch fails, the proposed MCR offers little advantage, suggesting that the framework has not yet unlocked full cross-modal complementarity in such scenarios.\n2. The experiments focus mainly on classification and regression tasks; additional evidence on generative problems (text-to-image / image-to-text) would strengthen claims about the method’s general applicability.", "was_revised": true, "final_justification": "Merging mutual-information decomposition with a game-theoretic gradient scheduler is innovative. I maintain my rating as ‘Accept’.", "rebuttal": "We appreciate the reviewer’s observation regarding cross-modal complementarity and we agree that this is a valid and important limitation. As noted in our discussion (Section 5) and detailed in the error analysis (Section 4.3, Appendix A.11), MCR excels at routing information when at least one modality provides a reliable signal. However, its ability to capture synergetic gains, particularly when all unimodal branches fail, is currently limited.\n\nOur initial hypothesis was that encouraging each modality to influence the fused prediction would inherently promote synergetic behaviour. However, the results suggest that emergent synergy does not arise automatically from this dynamic. We view this as an open challenge and an exciting direction for future work. Specifically, we are exploring extensions that encourage the joint representation to encode complementary information beyond what each unimodal pathway contributes individually, e.g., via terms like $I(f(x_1, x_2); Y \\mid f(x_1), f(x_2))$. Unlocking full cross-modal complementarity, especially in difficult cases where all modalities are individually weak, remains an open frontier. We thank the reviewer for highlighting this and agree that addressing this challenge is essential for further advancing multimodal learning.\n\n\nRegarding the reviewer’s suggestion for additional evidence on generative problems. We agree that evaluating multimodal training methods on generative tasks would broaden their empirical scope. It is important to clarify that our method is designed for settings where multiple modalities are available as inputs. In contrast, tasks like text-to-image or image-to-text generation typically involve only a single input modality, trying to generate the second one, and thus do not align with the central premise of MCR, which is to regulate the interaction and competition between input modalities during training. Applying MCR to such generative tasks would require a reformulation of the framework. We agree that extending MCR to generative tasks involving multiple conditioning modalities (e.g. audio + image → caption) would be a compelling next step. In such setups, managing how each modality influences the generation process could benefit from the same principles that underlie MCR. However, evaluating training strategies in generative settings remains challenging, particularly given the lack of established benchmarks for comparing multimodal training dynamics in these tasks. Additionally, many of the baseline methods we compare against are not directly applicable in those settings without major adaptation. For these reasons, we focused on classification and regression tasks where (1) multiple input modalities are jointly present, and (2) fair, well-established evaluation benchmarks for multimodal balancing exist. We appreciate the reviewer highlighting this direction, and we believe it represents an exciting opportunity for extending our framework in future work."}, {"review_id": "jrFrOIno9I", "reviewer": "Reviewer_33BC", "summary": "- The authors propose an information theory-based formulation of loss terms of multimodal training, which aims to balance the information from different modalities.\n- The proposed method, i.e., Multimodal Competition Regularizer (MCR), encompasses various loss terms derived from the intuitions of information theory. Specifically, the authors have crafted the MCR loss by summing \"Task-Relavant Unique Information from each modality\" to capture the unique information from each modality, \"Shared Information across modalities\", and \"Task-Irrelevant Shared Information\". \n- MCR shows the meaningful gains beyond the existing multimodal learning methods in various benchmarks.", "questions": "My main questions are based on Weaknesses 1 and 2.\n- Loss term-wise analysis is required to understand the behavior of multimodal training (Weakness 1). It would be glad to see the additional analysis from the authors during the rebuttal.\n- In-depth discussions on the two vague conclusions (Weakness 2). I hope to hear from the authors about the further discussions on these issues.\n\nAdditional suggestion.\n- To my understanding, this work mainly focuses on how different modalities interact with each other (synergy vs. competition). Would you compare with the following recent work, which tries to understand how different modalities make synergy.\n- \"Can One Modality Model Synergize Training of Other Modality Models?\" ICLR 2025.", "limitations": "Yes, but not sufficient to address Weaknesses 1 and 2. Also, I believe that the work has the potential to formalize the behavior of multimodal learning, but the precise analysis does not seem to be finalized.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strength 1: Providing the information-theoretic perspective to understand multimodal training**\n- The most impressive feature of this work is that it provides an information-theoretic perspective to understand the behavior of multimodal training. In prior works, researchers have already captured that the competition or synergy between modalities emphasizes the development of the delicate fusion of multimodal features. However, they are limited to focusing on technical improvements instead of providing a principled way to combine multimodalities.\n- From this perspective, the authors have formalized the related mutual information terms to facilitate the multimodal training; thereby suggesting a practical loss term to show the performance gains.\n- In the multimodal research society, it is quite vague to formally understand the synergy and competition between modalities. \n- I conjecture that the formulations of this work can be a tool for understanding how multimodal learning behaves. For example, a certain algorithm shows some limited performance in a certain case, then we can borrow the loss term in this work to quantify: unique information from each modality, and task-relevant/irrelevant information.\n\n**Strength 2: Extensive simulations across various scenarios**\n- The proposed method has been tested on various scenarios, including 6 datasets with a variety of [V],[A],[T], and [OF] modalities, and four architectures of ResNet, Conformer, Transformer, and Swin-TF.\n- I believe that the wide and extensive simulations truly validate the efficacy of the suggested loss terms.\n\n**Weakness 1: Ablation studies on the loss terms are required.**\n-  The MCR loss in Eq. 5 includes three loss terms, where each one contains an essential meaning to understand the behavior of multimodal training. Therefore, I think that the term-wise analysis is crucial to find out whether each modality synergizes or competes. For example, when we see the learning trend of each loss term (loss behaviors) during the training, we can figure out how the training is done by aggregating the information across modalities.  However, I cannot find any related analysis from this perspective.\n\n**Weakness 2: Some important parts remain undiscussed.**\n- First of all, in Section 3.2, the authors claim that the \"Greedy\" stance of each modality is effective compared to collaborative and independent behaviors. I wonder why the greedy behavior is optimal. Does it depend on each case, or a general trend across cases? In-depth discussions about the findings should be addressed in the paper.\n- Second, MCR underperforms when all modalities are not sufficient to solve the task (both fail). What is the reason? I conjecture that the choice of \"greedy\" in Section 3.2 likely hinders collaboration between modalities, thereby harming performance in the \"both fail\" case. Additional discussions are needed.", "was_revised": true, "final_justification": "The paper offers a novel perspective to understand the interaction between modalities by using a rigorous approach with information theory. I am fairly certain that the paper meets the NeurIPS standard, as it focuses on impactful research rather than incremental improvements.", "rebuttal": "We thank the reviewer for the suggestion on exploring the different loss terms during training. Indeed, analyzing the evolution of each component of the MCR loss (i.e., the two MIPD terms, the contrastive loss, and the CEB term) could provide valuable insights into the learning dynamics and modality interactions throughout training. In our experiments, we do track these components individually. For instance, we observe that the contrastive loss typically decreases smoothly, while the two MIPD terms alternate in which modality dominates at different points during training, often reflecting shifts in the modality that contributes more to the fused output. However, these trends do not necessarily remain smooth during training, and interpreting them in a systematic and generalizable way has proven non-trivial. Furthermore, leveraging this information to explicitly infer synergy or competition among the modalities would require extending our ways of evaluating the multimodal models. We recognize that such loss-term dynamics analysis is a meaningful step toward deeper understanding and calibration of multimodal learning behavior. In response to the reviewer’s suggestion, we have now included an additional paragraph in the discussion section acknowledging the value of this direction and outlining our intent to pursue it in future work. We appreciate the reviewer highlighting this important perspective.\n\n\nFurthermore, the thoughtful conjecture regarding the greedy strategy in Section 3.2 is well taken. Our current implementation promotes competition between modalities (k = −1), which indeed prioritizes maximizing each modality’s contribution, even at the cost of the other. While this improves routing (as shown in “one correct” cases), it may hinder synergetic behaviour, especially when no single modality suffices on its own. To examine this, we conducted a further error analysis of the different gaming strategies on the \"both wrong\", and we observed that there was no difference among the three game strategies.\n\n| Model           | Greedy    |           | Independent |           | Collaborative |           |\n| :-------------- | :-------- | :-------- | :---------- | :-------- | :------------ | :-------- |\n|                 | MM False  | MM True   | MM False    | MM True   | MM False      | MM True   |\n| CREMA_D ResNet  | 13.5      | 4.6       | 13.6        | 4.4       | 13.6          | 4.5       |\n| AVE ResNet      | 20.3      | 4.8       | 20.5        | 4.6       | 21.4          | 3.7       |\n| UCF ResNet      | 33.5      | 9.7       | 33.4        | 9.7       | 33.5          | 9.7       |\n\nOur initial hypothesis was that promoting each modality to influence the output would naturally encourage collaboration between them, thereby enhancing synergetic behavior. However, our empirical results did not support this assumption. We acknowledge in the discussion of the paper that there is no term to promote such synergy, and we are currently working on improving this aspect of the framework. We agree that this limitation highlights an important avenue for further discussion and future research. \n\n\nLastly, thank you for bringing to our attention the recent work \"Can One Modality Model Synergize Training of Other Modality Models?\". We have carefully examined it and find that it tackles a complementary but distinct problem. Their focus is on knowledge distillation across modalities, particularly how a modality can aid the training of another, and especially in unpaired data settings. This is indeed an impressive contribution to cross-modal representation learning, but it differs fundamentally from our goal. In contrast, our work addresses how to regulate modality interactions within joint multimodal training, and how these dynamics affect the performance of the fused model, rather than improving solely the unimodal branches. Nevertheless, there is a growing body of work that aims to quantify modality interactions, and we have drawn inspiration from several of these efforts, e.g., Liang et al. “Quantifying & Modeling Multimodal Interactions”. To evaluate their estimation of multimodal interactions, they rely on human-annotated ground truth regarding the informativeness of each modality, which is typically not available. In contrast, our objective was to develop a method that is modality- and task-agnostic, and thus we deliberately avoided reliance on such supervision. That said, we recognize the promise of this direction, and in future work, we aim to incorporate calibrated measures of uniqueness, redundancy, and synergy. These tools could offer deeper insight into the behavior of multimodal systems."}, {"review_id": "t1sJE63GQe", "reviewer": "Reviewer_UPy6", "summary": "This paper addresses the problem of \"modality competition\" in multimodal learning, a phenomenon that leads to suboptimal model performance. The authors introduce the Multimodal Competition Regularizer, a novel method derived from a mutual information decomposition. The approach combines refined bounds on mutual information terms—to enhance both information extraction and estimation—with a game-theoretic framework that balances modality contributions. Experimental results on synthetic and real-world datasets demonstrate that MCR achieves significant performance gains over existing methods.", "questions": "1. In Section 3.3, for estimating conditional mutual information, why did you choose to use permutation instead of simply masking the modality (e.g., zeroing it out)? Masking would also guarantee the removal of information from the corresponding modality and seems easier to implement.\n   \n2. The loss function in Equation 10 already focuses on maximizing conditional information, specifically $I(X_1;Y|X_2)$ and $I(X_2;Y|X_1)$. Could you elaborate on the conceptual difference and additional benefit provided by the MIPD term?\n   \n3. There appears to be a notational inconsistency in Equation 12. The gradient on the left-hand side is with respect to the full parameter set $\\theta$, while the gradients on the right-hand side are with respect to the parameters of the individual encoders ($\\theta_1$ and $\\theta_2$).", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper is well-written and easy to follow.\n   \n2. The proposed method leverages information theory to measure and analyze the different components of multimodal information, providing a principled approach to the problem.\n   \n3. The effectiveness of the method is validated through extensive experiments on both synthetic and real-world datasets.\n\n**Weaknesses:**\n\n1. The presentation of tables and figures could be enhanced for better clarity. Specifically, Table 1 lacks clear integration with the introduction section and fails to motivate the core problem addressed in the paper effectively. Additionally, the figure and table captions are excessively verbose, which may distract readers from grasping the key insights they are meant to convey.\n   \n2. The visualization in Figure 5 is confusing. It is not immediately clear what constitutes a \"better\" result, making the figure unintuitive. Furthermore, it is difficult to see how the figure illustrates the concept of \"synergistic information\" mentioned on line 244.\n   \n3. The motivation for certain aspects of the proposed method is unclear. In Equation 12, since modality 1 is perturbed in the $L_{MIPD_2}$ term, the gradient $\\nabla_{\\theta_1} L_{MIPD_2}$ can be considered a noise term. The paper does not sufficiently explain the purpose of applying a weighting factor to this term.", "was_revised": true, "final_justification": "The authors have addressed my concerns, and I raise my rating to 5.", "rebuttal": "We would like to thank the reviewer for the thoughtful comments.\n\nWe agree that several of our figure and table captions were overly verbose, as we initially aimed to make them fully self-contained. Following your suggestion, we have now revised the captions throughout the paper to be more concise and focused, emphasizing the key takeaways without overwhelming the reader. Regarding the integration of Table 1 with the introduction, we understand the reviewer’s concern to be that the table did not clearly reinforce the core motivation of the paper, namely, the presence of multimodal competition in standard training settings. We have now clarified this in both the introduction and the caption of Table 1, explicitly highlighting how Joint Training underperforms relative to the Ensemble baseline, which provides early empirical evidence of competition between modalities. These changes aim to better guide the reader in connecting the table to the central problem addressed by MCR.\n\nRegarding the error matrices figure, we agree that the original figure lacked clarity in conveying what constitutes a “better” result and how it connects to synergy and routing. To address this, we have revised the caption to explicitly state that a good result corresponds to low percentages in the top row (multimodal failure) and high percentages in the bottom row (multimodal success). We have also visually annotated with boxes the first CREMA-D ResNet matrix to highlight the \"Both Wrong\" column as evidence of synergy and the remaining columns as routing. These additions make the connection to our discussion more explicit and convey some of the key messages more clearly.\n\nRegarding the interesting question about Eq. 12, we agree that our explanation in the manuscript was insufficiently detailed. Whether the gradient of the $L_{MIPD}$ with respect to the perturbed modality provides useful or detrimental signal depends critically on the type of perturbation applied. When the perturbation corresponds to predefined or out-of-distribution inputs, such as zero-masking or adding noise, the resulting gradients may encourage the model to fit spurious patterns, potentially harming robustness. In contrast, when the perturbation is implemented via within-batch permutations (as in our main experiments), the perturbed modality remains in-distribution, and the loss $L_{\\text{MIPD}\\_1}$ can have a constructive influence on $\\theta_1$. The formulation is symmetric with respect to $L_{\\text{MIPD}\\_1}$ and $L\\_{\\text{MIPD}\\_2}$ under permutation-based perturbations under the JSD, but this symmetry breaks under arbitrary or task-information removal augmentations (e.g., those that remove class-relevant information entirely). In Appendix A.10, we explore such perturbation types and clarify that applying both $L\\_{\\text{MIPD}}$ under these conditions may degrade model robustness by reinforcing spurious or misleading input patterns. It is worth noting that removing the $L_{\\text{MIPD}}$ terms for these alternative perturbation strategies did not lead to improved performance. We have now added this clarification to the manuscript to better contextualize the behavior of $L\\_{\\text{MIPD}}$ under different perturbation regimes.\n\nWe have also incorporated the reviewer’s suggestion to evaluate zero-masking as a perturbation method for estimating conditional mutual information. While we had previously considered zero-masking in the input space, particularly in the context of computing Shapley values, we had not extended this approach to the latent space. In response, we have now expanded our ablation study (Appendix A.10) to include zero-masking in the latent space, which we find provides similar computational advantages to the permutation-based method. The corresponding results are shown in the table below.\n\n| Method                               | CREMA-D      | AVE          | UCF          | \n| :----------------------------------- | :-----------: | :-----------: | :-----------: |\n| MCR with Noise Input-Space           | 75.3$\\pm$2.9 | 72.1$\\pm$1.1 | 54.6$\\pm$0.8 | \n| MCR with Shapley Input-Space         | 73.6$\\pm$1.5 | 72.6$\\pm$0.9 | **55.5$\\pm$0.6** |\n| MCR with Noise Latent-Space          | 73.6$\\pm$1.1 | 72.6$\\pm$0.4 | 54.5$\\pm$0.7 | \n| MCR with Zeros Latent-Space           | 73.6$\\pm$1.9 | **73.3$\\pm$0.5** | 54.5$\\pm$0.4 | \n| MCR with Permutations Latent-Space   | **76.1$\\pm$1.1** | **73.3$\\pm$0.5** | 55.2$\\pm$1.8 | \n\n\nRegarding Equation 10, while the proof indicates a lower bound that encompasses $I(X_1; Y \\mid X_2)$ and $I(X_2; Y \\mid X_1)$, it does not provide control over these two terms. The MIPD components introduced in our formulation make these conditional dependencies explicit and separable, enabling us to modulate their relative influence during training. This separation allows for flexible strategy design in how modality interactions are encouraged or discouraged. Adjusting this balance can lead to performance gains beyond simply maximizing each individual term, as shown in the corresponding ablation study (Appendix A.8). The MIPD formulation thus introduces control over inter-modal behavior that would otherwise be entangled only as a joint lower bound in the objective of Equation 10.\n\nFinally, we thank the reviewer for pointing out the notational inconsistency. We intended to describe the parameter updates for each encoder separately, rather than for the joint parameter set. To clarify this, we have rewritten the gradient expression as:\n\n$$ \\nabla\\_{\\theta\\_1} \\mathcal{L}\\_{MIPD} = \\lambda\\_M ( \\nabla\\_{\\theta\\_1}\\mathcal{L}\\_{MIPD\\_1} + k \\nabla\\_{\\theta\\_1}\\mathcal{L}\\_{MIPD\\_2} ) $$\n\n$$ \\nabla\\_{\\theta\\_2} \\mathcal{L}\\_{MIPD} = \\lambda\\_M ( \\nabla\\_{\\theta\\_2}\\mathcal{L}\\_{MIPD\\_2} + k \\nabla\\_{\\theta\\_2}\\mathcal{L}\\_{MIPD\\_1} ) $$\n\nThis updated notation should resolve the ambiguity and align the gradients with the individual parameter sets."}]

Author rebuttal:

Meta-review: **Paper Summary:**\
This paper proposes a novel game-theoretic regularization framework to address the modality competition issue in multimodal learning. The paper presents an interesting methodology with clear results and has received generally positive reviews. The authors engaged constructively with the reviewers during the discussion phase, leading some reviewers to improve their scores following the rebuttal. The paper’s strengths lie primarily in its innovative approach, its valuable contributions to multimodal learning, and the experimental validation of its effectiveness. The main challenges concern the in-depth analysis of the model’s dynamics, its applicability to specific scenarios, and its scalability, all of which have already been addressed during the rebuttal phase.

**Justification:**\
This paper introduces a novel game-theoretic regularization framework to addressing modal competition, which is both innovative and insightful. Following thorough discussions between the authors and the reviewers, the paper received unanimous Accept recommendations. Provided that the authors carefully address the issues raised during the rebuttal in the final version, this work has the potential to reach high quality."

**Summary of Rebuttal Period:**\
Reviewers dDxc, 33BC, and UPy6 highlighted issues with the experimental presentation, and Reviewer QAwA further raised concerns regarding the paper’s scalability and applicability. Reviewers 33BC and UPy6 questioned certain terminology and details. The authors engaged in thorough discussions with the reviewers, and after the rebuttal, most of these issues were satisfactorily resolved, with some reviewers subsequently improving their scores for the manuscript.
Decision: Accept (spotlight)