Title: Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportion
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Leanring; Modality Imbalance.
Abstract: Multimodal learning (MML) is significantly constrained by modality imbalance, leading to suboptimal performance in practice. While existing approaches primarily focus on balancing the learning of different modalities to address this issue, they fundamentally overlook the inherent disproportion in model classification ability, which serves as the primary cause of this phenomenon. In this paper, we propose a novel multimodal learning approach to dynamically balance the classification ability of weak and strong modalities by incorporating the principle of boosting. Concretely, we first propose a sustained boosting algorithm in multimodal learning by simultaneously optimizing the classification and residual errors. Subsequently, we introduce an adaptive classifier assignment strategy to dynamically facilitate the classification performance of the weak modality. Furthermore, we theoretically analyze the convergence property of the cross-modal gap function, ensuring the effectiveness of the proposed boosting scheme. To this end, the classification ability of strong and weak modalities is expected to be balanced, thereby mitigating the imbalance issue. Empirical experiments on widely used datasets reveal the superiority of our method through comparison with various state-of-the-art (SOTA) multimodal learning baselines. The source code is available at https://github.com/njustkmg/NeurIPS25-AUG.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the persistent challenge of modality imbalance in multimodal learning, proposing that the core issue stems from the inherent classification ability gap between modalities. The authors introduce a sustained boosting approach that incrementally strengthens weaker modality classifiers via residual error minimization. Additionally, an adaptive strategy dynamically assigns new classifiers based on modality-specific confidence scores. The theoretical convergence analysis is solid and bolstered by empirical results showing superior performance across diverse benchmarks. The results of visualizations, t-SNE plots, and convergence curves provides transparency into model behavior. However, despite the impressive results, the paper could benefit from further analysis on efficiency and more diverse ablation studies. Overall, this paper contributes a fresh viewpoint and a robust solution to the community.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper proposes a novel method to address the multimodal imbalance problem, with the following key advantages:\n(1)  the paper addresses a real and understudied issue in multimodal learning, i.e., disproportionate classification ability across modalities. It further identifies this problem as a key contributor to modality imbalance. Furthermore, it introduces a novel sustained boosting algorithm that explicitly enhances weaker modality classifiers;\n(2) theoretical analysis (Theorem 1) supports convergence guarantees of the proposed gap minimization framework; \n(3) consistent gains across metrics (Acc/MAP/F1) and modalities (video, audio, text) in Table 1 are impressive. Code release and exhaustive implementation details (Sec 4.1) align with open-science best practices; \n(4) the paper is clearly written, logically coherent, and well organized, making it easy to follow and understand.\n\nFurthermore, several weaknesses in the paper should be noted: (1). Lemma 1 only provides an abbreviated title, and it appears that the full statement or introduction may be missing; (2). Details on computational cost vs. performance tradeoff are limited and deserve deeper discussion. Training overhead (Fig 5) grows linearly with classifiers. This is problematic for resource-limited settings, yet mitigation strategies are absent; (3). The ablation study is primarily conducted on one dataset (CREMAD), which may not generalize well.\n\nQUESTIONS:\nMy key concerns about the paper are outlined below: \n\n(1). The relationship between training cost and the number of classifiers, as well as the corresponding performance, warrants further discussion.  \n\n(2) The authors are encouraged to report the results of ablation studies on additional datasets to validate the generality of the proposed method.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a promising and adaptable framework for improving multimodal learning in scenarios where certain modalities dominate others in classification performance. The proposed sustained boosting and adaptive classifier assignment modules offer an elegant way to dynamically rebalance classification capacity without heavily redesigning the architecture. Results show strong empirical gains across six datasets. However, a more thorough comparison and discussion of the similarities and differences with existing boosting-based method, i.e., ReconBoost, are necessary to better highlight the novelty of the proposed approach. Nevertheless, this is a meaningful and actionable contribution to the field of robust multimodal systems.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n1. Novelty of the Proposed Method: The proposed multimodal learning approach to address the modality imbalance issue is applicable to real-world multimodal systems where sensor strengths vary. And the adaptive classifier assignment strategy reduces the need for manual model tuning during training. \n\n2. Flexibility of the Proposed Method: Successful extension to 3 modalities (NVGesture) proves versatility beyond audio-visual/image-text tasks. Performance gains are consistent across tasks with more than two modalities, showing versatility.\n\n3. Effectiveness of the Experiments: Extensive experimental results demonstrate the effectiveness of the proposed method. Experiments include meaningful evaluation metrics (accuracy, MAP, MacroF1) suitable for practical deployment.\n\n4. Theoretical Proof: The convergence guarantees presented in Section 3.4 reinforce the credibility of the empirical results.\n\n## Weaknesses:\n1. Comparison with ReconBoost [1]: The authors propose a boosting algorithm to address the problem of modality imbalance. A detailed comparison with the ReconBoost method, including both similarities and differences, is necessary to clarify the contributions of the proposed approach.\n\n2. Selection of Scoring Function: The author employs the existing scoring method OGM [2]; however, it would be valuable to discuss whether more effective alternatives are available.\n\nReference:\n[1]. Hua, Cong, et al. Reconboost: Boosting can achieve modality reconcilement. ICML, 2024.\n[2]. Peng, Xiaokang, et al. Balanced multimodal learning via on-the-fly gradient modulation. CVPR, 2022.\n\nQUESTIONS:\n1. In what ways is the proposed method innovative compared to ReconBoost?\n\n2. No ablation on the confidence scoring function—would other metrics (entropy, loss) work better?\n\n3. Although the authors demonstrate the effectiveness of their method on a trimodal dataset, relevant experimental details are lacking.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript introduces a theoretically sound and empirically validated approach to address modality imbalance in multimodal learning. By crafting a specialized boosting algorithm, it enhances the classification ability of weaker modality-specific classifiers. Through an analysis of multimodal model capacity, the work sheds fresh light on the roots of modality imbalance. Comprehensive evaluations against state-of-the-art methods demonstrate the proposed approach’s clear advantages, and supplementary experiments further attest to its overall efficacy. However, the paper would benefit from deeper discussion and additional experiments regarding the selection of the boosting strategy and the broader applicability of the technique.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The authors reconceptualize modality imbalance as a disparity in classification capacities, an angle largely overlooked in previous studies, which prompts a reconsideration of the problem through the lens of model capacity.\n2. The boosting framework is effectively tailored for the multimodal context, marking a creative application of boosting techniques.\n3. Experiments conducted on six diverse datasets consistently show superior performance compared to leading baselines such as ReconBoost and DI‑MML.\n4. The work includes extensive analyses—hyperparameter sweeps, ablation trials, and visualizations—to underscore the proposed method’s strengths.\n5. Figure 4 clearly shows improved classification outcomes for weaker modalities, bolstering confidence in the approach.\n\n**Weaknesses:**\n\n1. Since the boosting algorithm is rooted in gradient boosting, it remains uncertain whether other boosting variants could match its performance.\n2. The current method is validated only for late fusion; its effectiveness with early fusion architectures is not explored.\n3. The annotation “Full Prediction vs. t−1 CLS” in Figure 3 is unclear—more precise labeling or explanation is needed.\n4. The design and parameters of the toy experiments are insufficiently described; supplying full experimental details would enhance reproducibility.\n\nQUESTIONS:\n1. Could other boosting paradigms yield comparable improvements, and what underlying mechanisms would support their success?\n2. In what ways could this boosting-based approach be adapted for or integrated into early fusion multimodal frameworks?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the often-overlooked issue of modality imbalance in multimodal learning (MML), particularly the disproportion in classification ability between strong and weak modalities. Instead of aligning the learning rates or optimizing joint features, the authors focus on directly enhancing the classification performance of the weaker modality. To achieve this, they propose a Sustained Boosting algorithm, which jointly minimizes classification and residual errors to progressively improve weak classifiers. Additionally, an Adaptive Classifier Assignment (ACA) strategy is introduced to dynamically assign more classifiers to underperforming modalities, guided by confidence scores. Theoretical analysis is provided to guarantee the convergence of the proposed method in terms of closing the loss gap between modalities. Empirical results across six multimodal datasets—including audio-video and image-text pairs—demonstrate that the proposed approach consistently outperforms a wide range of state-of-the-art baselines, including ReconBoost, LFM, and others. Ablation and sensitivity analyses further validate the contribution of each component.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper exhibits strong technical rigor and proposes a clearly motivated shift in focus—from aligning learning dynamics to balancing classification abilities across modalities. This perspective is refreshing and highlights a previously under-discussed source of performance degradation in MML. The use of a boosting-based framework is well-justified, and the idea of sustaining the optimization over both classification and residual errors is thoughtfully executed. Importantly, the inclusion of the Adaptive Classifier Assignment strategy provides a dynamic mechanism to address imbalance throughout training, rather than relying on fixed configurations.\n\nMoreover, the authors support their claims with extensive experiments across six datasets that cover diverse modalities and tasks. Theoretical convergence analysis is presented with full proofs, ensuring the robustness of the proposed approach. The empirical results are generally strong, often yielding the best performance on each benchmark.\n\nHowever, the work is not without its limitations. While the authors do touch on model overhead, the computational and memory cost introduced by using multiple classifiers—especially during inference—is not analyzed in detail. This raises concerns about the scalability and deployability of the approach in real-world applications, particularly those requiring low-latency or resource-constrained inference. Additionally, although the performance gain is consistent, the margin over strong baselines like ReconBoost or LFM is sometimes modest. Another limitation is that the approach is designed for cases where all modalities are fully present, and the framework’s behavior under partial or missing modality scenarios is not explored. Finally, while the adaptive strategy is a novel idea, there is a lack of discussion on its stability—e.g., how it reacts to noisy modality-specific confidence scores or abrupt changes in modality quality during training.\n\nQUESTIONS:\nAdaptive Classifier Assignment: If the confidence score for a modality fluctuates due to noise or temporary degradation in input quality, does the algorithm risk overcompensating by adding unnecessary classifiers, potentially leading to overfitting? It would also be helpful to see results from a stress test on this adaptive strategy under noisy modality conditions.\n\nHow does your method behave when a modality is missing or corrupted? Given that the method is trained on full-modal input, would sustained boosting and classifier assignment degrade gracefully if a modality is unavailable during inference?\n\nA natural baseline to compare against would be modality-specific knowledge distillation, where weak modalities learn from stronger ones via teacher-student dynamics. Has this been explored or considered?\n\nLIMITATIONS:\nSee questions", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}