Title: Adaptive Re-calibration Learning for Balanced Multimodal Intention Recognition
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal data

Modality imbalance

modality valuation

Sample Calibration

Encoder Calibration
Abstract: Multimodal Intention Recognition (MIR) plays a critical role in applications such as intelligent assistants, service robots, and autonomous systems. However, in real-world settings, different modalities often vary significantly in informativeness, reliability, and noise levels. This leads to modality imbalance, where models tend to over-rely on dominant modalities, thereby limiting generalization and robustness. While existing methods attempt to alleviate this issue at either the sample or model level, most overlook its multi-level nature. To address this, we propose Adaptive Re-calibration Learning (ARL), a novel dual-path framework that models modality importance from both sample-wise and structural perspectives. ARL incorporates two key mechanisms: Contribution-Inverse Sample Calibration (CISC), which dynamically masks overly dominant modalities at the sample level to encourage attention to underutilized ones; and Weighted Encoder Calibration (WEC), which adjusts encoder weights based on global modality contributions to prevent overfitting. Experimental results on multiple MIR benchmarks demonstrate that ARL significantly outperforms existing methods in both accuracy and robustness, particularly under noisy or modality-degraded conditions.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAdaptive Re-calibration Learning (ARL) is a plug-and-play framework that remedies modality imbalance in multimodal intention-recognition (MIR) and sentiment-analysis pipelines. The method first quantifies per-sample modality importance through a Leave-One-Out (LOO) valuation, whose complexity grows only linearly with the number of modalities . These scores drive two complementary calibrations: Contribution-Inverse Sample Calibration (CISC), which temporarily masks dominant modalities at the feature level, and Weighted Encoder Calibration (WEC), which periodically re-weights modality-specific encoders using a blend of stability and contribution signals . When attached to three strong baselines (MAG-BERT, MulT, TCL-MAP), ARL boosts twenty-class MIntRec accuracy by up to +1.73 (pp) and MOSI sentiment accuracy by up to +1.60 (pp) while retaining robustness under missing-modality conditions .\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n1. Well-Motivated and Effective Solution to Modality Imbalance\nThe paper addresses a critical challenge in multimodal intention recognition—modality imbalance—by proposing a novel and practical solution. The Adaptive Re-calibration Learning (ARL) framework introduces two complementary mechanisms: Contribution-Inverse Sample Calibration (CISC) and Weighted Encoder Calibration (WEC), which jointly optimize modality usage at both the sample and model levels. This dual-path strategy outperforms existing methods by dynamically balancing modality importance, especially under noisy or missing modality conditions.\n2. Technical Novelty and Computational Efficiency\nARL leverages a Leave-One-Out (LOO) method for modality valuation, offering a computationally efficient alternative to Shapley values with linear complexity. This makes the approach scalable and suitable for real-world applications. The synergy between CISC and WEC provides a more holistic and adaptive calibration mechanism compared to prior techniques that focus only on either gradient modulation or dropout.\n3. Strong Empirical Validation and Practical Applicability\nExtensive experiments across multiple datasets and architectures (e.g., MAG-BERT, MulT, TCL-MAP) demonstrate consistent performance improvements in both accuracy and robustness. ARL shows particular strength in handling degraded or missing modalities, making it highly valuable for deployment in real-world systems such as robotics, intelligent assistants, and autonomous systems.\n\nWeaknesses\nDespite its strengths, the paper has several areas that could benefit from improvement:\n1. Lack of Formal Definition of Modality Imbalance: The central problem of the paper is not formally defined, nor is there a quantitative metric provided to measure modality imbalance. This limits the theoretical grounding of the approach.\n2. Limited Training Procedure Description: The paper lacks a clear step-by-step explanation of how ARL integrates with the training process of base models. For example, the frequency of WEC updates and interaction between CISC and WEC during optimization are not clearly outlined.\n\nQUESTIONS:\nBelow are key questions and suggestions for the authors, focusing on actionable points that could clarify or strengthen the paper:\nQ1: Can you provide a formal definition and measurement of “modality imbalance”?\n- Why it matters: This is the core problem addressed by the paper. A formal definition would help readers understand the scope and assumptions of the method.\n- Suggestion: Introduce a mathematical formulation or statistical measure (e.g., KL divergence, entropy) to quantify modality imbalance at both the sample and dataset levels. You may consider using the LOO contribution scores to define an imbalance index.\nQ2: Could you provide a pseudocode or flowchart of the full ARL training process?\n- Why it matters: The integration of CISC and WEC into the training loop is not fully transparent, limiting understanding of how the method interacts with standard training procedures.\n- Suggestion: Include a pseudocode or visual flowchart showing the order and frequency of operations (e.g., when WEC is updated, how often CISC is applied per batch/epoch), and how gradients are computed and combined.\n\nLIMITATIONS:\nYes, the authors have adequately discussed the limitations of their work in Section 5 and Appendix A.2. They acknowledge that:\n- ARL's performance depends on the expressiveness of the base model.\n- There is increased computational overhead due to the LOO evaluation and dual calibration mechanisms.\n- Privacy concerns and potential misuse are considered in broader impacts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an adaptive recalibration learning (ARL) framework to solve the modality imbalance problem in multimodal intent recognition through a dual-path calibration mechanism. The main innovations include:  1) proposed a contribution inverse sample calibration (CISC) mechanism to dynamically mask the dominant mode at the sample level.   2) designed a weighted encoder calibration (WEC) mechanism to adjust the encoder weights at the architecture level,  3)developed an efficient leave-one-out (LOO) modality evaluation method to replace the calculation of complex Shapley values\n\nSTRENGTHS AND WEAKNESSES:\nStrength.\n 1. The paper introduces a novel Adaptive Re-calibration Learning (ARL) framework that uniquely addresses modality imbalance through complementary sample-level (CISC) and architectural-level (WEC) calibration mechanisms.\n\n\n2. Experimental results across multiple benchmarks show consistent improvements over strong baselines,  particularly under challenging noisy/missing-modality conditions.\n\nWeakness.\n\n1. As listed in the related work, the imbalanced multimodal learning is a widely studied problem. However, the experiments do not take the existing method into comparison.\n\n2. The time cost increases linearly with the number of modes. It would be better to design a more efficient method to measure the modality contribution at the sample level.\n\n3. The model performance depends on several key hyperparameters, such as the mask threshold τ in CISC, λ (stability influence weight), and α (contribution influence weight) in WEC. Experiments show that the value of α has a significant impact on the results (Tab. 3), but the optimal range of other hyperparameters under different data sets or modality configurations is not systematically analyzed.\n\n4. This paper claims that ARL can address the imbalanced learning at the sample level. However, no experimental evidence is provided to demonstrate this.\n\nQUESTIONS:\nsee the weakness\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the task of Multimodal Intention Recognition (MIR). The authors solve the modality imbalance issue in MIR, where models tend to over-rely on dominant modalities, thereby limiting generalization and robustness. Specifically, the authors propose Adaptive Re-calibration Learning (ARL), a novel dual-path framework that models modality importance from both sample-wise and structural perspectives. There are two key mechanisms in ARL: Contribution-Inverse Sample Calibration (CISC) and Weighted Encoder Calibration (WEC), which dynamically masks overly dominant modalities at the sample level and adjusts encoder weights based on global modality contributions to prevent overfitting, respectively. Experimental results on multiple MIR benchmarks demonstrate the effectiveness of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well written and organized, which is easy to follow;\n- The proposed approach is technically sound, especially for the Contribution-Inverse Sample Calibration (CISC), which is interesting;\n- Experiments are conducted on several mainstream benchmarks for verification;\n\nWeakness:\n- Missing overall running efficiency analysis. While applying ARL to off-the-shelf approaches leads to performance improvements as shown in Tables 4 and 5, it remains unclear whether the proposed approach may cause significant training efficiency burden. In Fig. 3, the authors made Computational Cost comparison between the proposed LOO and Shapley-based solution, this is helpful, but it does not clearly show the overall computational cost introduced by the proposed ARL when applied to off-the-shelf approaches., e.g., MAG_BERT and MulT. The authors are suggested to add more detailed training efficiency analysis to clarify whether their approach is a lightweight solution, which could be easily integrated into existing approaches w/o causing significant training cost. \n- The state-of-the-art comparisons (e.g., CMU-MOSI) is somewhat weak. I acknowledged that the authors aim to show their approach can generally improve existing approaches, it is still encouraged to include more SOTA approaches for comparison (just for comparison, w/o apply ARL on them), e.g., [1, 2] on CMU-MOSI, or maybe recently published one [3].\n- Missing references: [4, 5]. In [4], the authors proposed to drop spatial tokens in order to make the model more focus on the temporal tokens, which is similar to the proposed LOO here; [5] is also working on balancing multiple modalities. Please add these references in the related work for discussion;\n- Minor issues: Some typos, e.g., L37 and L39, missing empty spaces after the citation; \n\n[1] Contextual Augmented Global Contrast for Multimodal Intent Recognition. CVPR24.\n\n[2] Decoupled multimodal distilling for emotion recognition. CVPR23.\n\n[3] EMOE: Modality-Specific Enhanced Dynamic Emotion Experts. CVPR25.\n\n[4] DropMAE: Masked Autoencoders with Spatial-Attention Dropout for Tracking Tasks. CVPR23.\n\n[5] Wasserstein Modality Alignment Makes Your Multimodal Transformer More Robust. TMLR25.\n\nQUESTIONS:\nOverall, the paper has clear motivation and technically sound solution. Thus, my initial rating is boardline accept. I would like to see more author responses in terms of the overall running efficiency and SOTA comparisons. If well addressed, I would like to raise my rating.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper addresses the robustness in performance of multimodal intention recognition under modality-compromised conditions. It proposes Adaptive recalibration learning to model modality importance. Experiments are conducted on two benchmark intention detection datasets.

STRENGTHS AND WEAKNESSES:
The paper is easy to follow and written well. The motivation is clear and the idea is explained well. The idea is novel, to the best of my knowledge.


The idea is dependent on the base model and may not be able to eradicate the bias involved in the base model.

QUESTIONS:
1) The authors are doing hard masking (Lines 185). Why is it necessary? If the weaker modalities are too weak, which is quite often the case, this might actually hurt the performance. Did the authors consider trying soft masking?


2) Leave one out method kind of assumes that modalities are independent of each other, which is not usually the case. How does inter-modality dependence effect the proposed method?

3) The threshold in Eq 10, is it tuned for each modality separately, one for each dataset, or one across all experiments? 

4) Same question as 3 for the η sensitivity parameter in Eq 9.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03