Title: Balancing Multimodal Training Through Game-Theoretic Regularization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Deep Learning

Multimodal Competition
Abstract: Multimodal learning holds the promise for richer information extraction by capturing dependencies across data sources. Yet, current training methods often underperform due to modality competition, a phenomenon where modalities contend for training resources, leaving some underoptimized. This raises a pivotal question: how can we address training imbalances, ensure adequate optimization across all modalities, and achieve consistent performance improvements as we transition from unimodal to multimodal data? This paper proposes the Multimodal Competition Regularizer (MCR), inspired by a mutual information (MI) decomposition designed to prevent the adverse effects of competition in multimodal training. Our key contributions are: 1) A game-theoretic framework that adaptively balances modality contributions by encouraging each to maximize its informative role in the final prediction. 2) Refining lower and upper bounds for each MI term to enhance the extraction of both task-relevant unique and shared information across modalities. 3) Proposing latent space permutations for conditional MI estimation, significantly improving computational efficiency. MCR outperforms all previously suggested training strategies and simple baselines, demonstrating that training modalities jointly lead to important performance gains on synthetic and large real-world datasets. We release our code and models at https://github.com/kkontras/MCR.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of \"modality competition\" in multimodal learning, a phenomenon that leads to suboptimal model performance. The authors introduce the Multimodal Competition Regularizer, a novel method derived from a mutual information decomposition. The approach combines refined bounds on mutual information terms—to enhance both information extraction and estimation—with a game-theoretic framework that balances modality contributions. Experimental results on synthetic and real-world datasets demonstrate that MCR achieves significant performance gains over existing methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper is well-written and easy to follow.\n   \n2. The proposed method leverages information theory to measure and analyze the different components of multimodal information, providing a principled approach to the problem.\n   \n3. The effectiveness of the method is validated through extensive experiments on both synthetic and real-world datasets.\n\n**Weaknesses:**\n\n1. The presentation of tables and figures could be enhanced for better clarity. Specifically, Table 1 lacks clear integration with the introduction section and fails to motivate the core problem addressed in the paper effectively. Additionally, the figure and table captions are excessively verbose, which may distract readers from grasping the key insights they are meant to convey.\n   \n2. The visualization in Figure 5 is confusing. It is not immediately clear what constitutes a \"better\" result, making the figure unintuitive. Furthermore, it is difficult to see how the figure illustrates the concept of \"synergistic information\" mentioned on line 244.\n   \n3. The motivation for certain aspects of the proposed method is unclear. In Equation 12, since modality 1 is perturbed in the $L_{MIPD_2}$ term, the gradient $\\nabla_{\\theta_1} L_{MIPD_2}$ can be considered a noise term. The paper does not sufficiently explain the purpose of applying a weighting factor to this term.\n\nQUESTIONS:\n1. In Section 3.3, for estimating conditional mutual information, why did you choose to use permutation instead of simply masking the modality (e.g., zeroing it out)? Masking would also guarantee the removal of information from the corresponding modality and seems easier to implement.\n   \n2. The loss function in Equation 10 already focuses on maximizing conditional information, specifically $I(X_1;Y|X_2)$ and $I(X_2;Y|X_1)$. Could you elaborate on the conceptual difference and additional benefit provided by the MIPD term?\n   \n3. There appears to be a notational inconsistency in Equation 12. The gradient on the left-hand side is with respect to the full parameter set $\\theta$, while the gradients on the right-hand side are with respect to the parameters of the individual encoders ($\\theta_1$ and $\\theta_2$).\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The authors propose an information theory-based formulation of loss terms of multimodal training, which aims to balance the information from different modalities.\n- The proposed method, i.e., Multimodal Competition Regularizer (MCR), encompasses various loss terms derived from the intuitions of information theory. Specifically, the authors have crafted the MCR loss by summing \"Task-Relavant Unique Information from each modality\" to capture the unique information from each modality, \"Shared Information across modalities\", and \"Task-Irrelevant Shared Information\". \n- MCR shows the meaningful gains beyond the existing multimodal learning methods in various benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength 1: Providing the information-theoretic perspective to understand multimodal training**\n- The most impressive feature of this work is that it provides an information-theoretic perspective to understand the behavior of multimodal training. In prior works, researchers have already captured that the competition or synergy between modalities emphasizes the development of the delicate fusion of multimodal features. However, they are limited to focusing on technical improvements instead of providing a principled way to combine multimodalities.\n- From this perspective, the authors have formalized the related mutual information terms to facilitate the multimodal training; thereby suggesting a practical loss term to show the performance gains.\n- In the multimodal research society, it is quite vague to formally understand the synergy and competition between modalities. \n- I conjecture that the formulations of this work can be a tool for understanding how multimodal learning behaves. For example, a certain algorithm shows some limited performance in a certain case, then we can borrow the loss term in this work to quantify: unique information from each modality, and task-relevant/irrelevant information.\n\n**Strength 2: Extensive simulations across various scenarios**\n- The proposed method has been tested on various scenarios, including 6 datasets with a variety of [V],[A],[T], and [OF] modalities, and four architectures of ResNet, Conformer, Transformer, and Swin-TF.\n- I believe that the wide and extensive simulations truly validate the efficacy of the suggested loss terms.\n\n**Weakness 1: Ablation studies on the loss terms are required.**\n-  The MCR loss in Eq. 5 includes three loss terms, where each one contains an essential meaning to understand the behavior of multimodal training. Therefore, I think that the term-wise analysis is crucial to find out whether each modality synergizes or competes. For example, when we see the learning trend of each loss term (loss behaviors) during the training, we can figure out how the training is done by aggregating the information across modalities.  However, I cannot find any related analysis from this perspective.\n\n**Weakness 2: Some important parts remain undiscussed.**\n- First of all, in Section 3.2, the authors claim that the \"Greedy\" stance of each modality is effective compared to collaborative and independent behaviors. I wonder why the greedy behavior is optimal. Does it depend on each case, or a general trend across cases? In-depth discussions about the findings should be addressed in the paper.\n- Second, MCR underperforms when all modalities are not sufficient to solve the task (both fail). What is the reason? I conjecture that the choice of \"greedy\" in Section 3.2 likely hinders collaboration between modalities, thereby harming performance in the \"both fail\" case. Additional discussions are needed.\n\nQUESTIONS:\nMy main questions are based on Weaknesses 1 and 2.\n- Loss term-wise analysis is required to understand the behavior of multimodal training (Weakness 1). It would be glad to see the additional analysis from the authors during the rebuttal.\n- In-depth discussions on the two vague conclusions (Weakness 2). I hope to hear from the authors about the further discussions on these issues.\n\nAdditional suggestion.\n- To my understanding, this work mainly focuses on how different modalities interact with each other (synergy vs. competition). Would you compare with the following recent work, which tries to understand how different modalities make synergy.\n- \"Can One Modality Model Synergize Training of Other Modality Models?\" ICLR 2025.\n\nLIMITATIONS:\nYes, but not sufficient to address Weaknesses 1 and 2. Also, I believe that the work has the potential to formalize the behavior of multimodal learning, but the precise analysis does not seem to be finalized.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to addresses the common “modality competition” problem in multimodal training. The authors propose the Multimodal Competition Regularizer (MCR), a framework that first uses mutual information analysis to divide each representation into three parts: task-relevant information that is unique to one modality, task-relevant information that is shared by several modalities, and shared noise that is irrelevant to the task. Afterward, the method applies three complementary loss functions(LMIPD, LCon and LCEB) together with a gradient-scheduling strategy inspired by game theory, ensuring that every modality makes an appropriate contribution. The authors also present empirical results on a controlled synthetic dataset and several real-world benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\n1. I am not a specialist in game theory, the way to merge mutual-information decomposition with a game-theoretic gradient scheduler is new to me and appears conceptually innovative.\n2. Estimating conditional mutual information through latent-space permutation is a good practical solution as it is computationally efficient, offering a feasible alternative to more expensive input-space perturbations. \n3. The method shows strong results, and the experimental discussion is detailed and systematic, covering both synthetic and multiple real-world datasets. \n\n**Weakness:**\n\n1. As the authors themselves note, when every unimodal branch fails, the proposed MCR offers little advantage, suggesting that the framework has not yet unlocked full cross-modal complementarity in such scenarios.\n2. The experiments focus mainly on classification and regression tasks; additional evidence on generative problems (text-to-image / image-to-text) would strengthen claims about the method’s general applicability.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The manuscript proposes the Multimodal Competition Regularizer. It is a training strategy designed to address the problem of modality competition in multimodal learning, where certain data modalities dominate training and suppress others. MCR introduces a game-theoretic regularization framework that adaptively balances the contribution of each modality by decomposing mutual information into unique and shared task-relevant components. The method is evaluated across a variety of synthetic and real-world multimodal datasets. It outperforms existing baselines and prior approaches in both classification and regression settings.

STRENGTHS AND WEAKNESSES:
Strengths:
1. It is a novel and interesting game-theoretic regularization method for balancing contributions across modalities in multimodal training. It addresses a well-known challenge of modality competition that is often overlooked in the literature.
2. The experimental evaluation is thorough and convincing, as the authors validate across six diverse, widely used datasets.

Weaknesses:
1. No statistical testing between the best model and baselines in Tables 1, 3, and 5. One can use, for example, Wilcoxon and correct for multiple comparisons with the Holm correction.
2. In Table 1, some models are without std.

QUESTIONS:
The game-theoretic framework currently uses discrete strategies with fixed values (-1, 0, 1) to balance modality contributions. Have the authors considered using continuous or learnable weights for this optimization, which could enable a softer and potentially more adaptive balance between modalities during training?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-03