Title: Hyper-Modality Enhancement for Multimodal Sentiment Analysis with Missing Modalities
Venue: NeurIPS 2025
Accepted: yes
Keywords: multimodal sentiment analysis; incomplete learning;
Abstract: Multimodal Sentiment Analysis (MSA) aims to infer human emotions by integrating complementary signals from diverse modalities. However, in real-world scenarios, missing modalities are common due to data corruption, sensor failure, or privacy concerns, which can significantly degrade model performance. To tackle this challenge, we propose Hyper-Modality Enhancement (HME), a novel framework that avoids explicit modality reconstruction by enriching each observed modality with semantically relevant cues retrieved from other samples. This cross-sample enhancement reduces reliance on fully observed data during training, making the method better suited to scenarios with inherently incomplete inputs. In addition, we introduce an uncertainty-aware fusion mechanism that adaptively balances original and enriched representations to improve robustness. Extensive experiments on three public benchmarks show that HME consistently outperforms state-of-the-art methods under various missing modality conditions, demonstrating its practicality in real-world MSA applications.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework, Hyper-Modality Enhancement (HME), for Multimodal Sentiment Analysis (MSA) in the presence of missing modalities. The method begins by extracting modality-specific representations, then enhances them using batch-level cross-sample information via a modality enhancement module. These enhanced features are integrated through a hyper-modality generation module that fuses cues across all modalities. To address redundancy and noise, the framework incorporates a Variational Information Bottleneck (VIB). Experimental results on multiple benchmark datasets demonstrate that HME outperforms prior methods, especially under high missing rates, showing strong robustness and generalization.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper tackles the underexplored challenge of missing modalities in MSA using a cross-sample enhancement mechanism, which introduces complementary information from semantically similar samples in the same batch—without relying on full modality reconstruction.\n\n2. The uncertainty-aware fusion strategy is thoughtfully designed, using modality-wise variance to weight the contribution of each representation. The accompanying qualitative analysis of uncertainty weights provides insightful understanding of how different modalities contribute under varying conditions.\n\nWeakness:\n\n1. Batch size sensitivity: Since enhanced features are derived from samples within the same batch, it would be valuable to analyze how performance varies with batch size. This could help understand the dependency of HME on batch-level diversity. Qualitative examples would be helpful to understand why certain samples were chosen for enrichment.\n\n2. Interpretation of prompts: The paper could benefit from further analysis of the learnable prompts used in the Perceiver and hyper-modality interaction modules. What information is being captured or attended to by these prompts?\n\n3. More detailed ablation on VIB: While VIB is shown to reduce redundancy, a more granular study of the influence of the VIB loss weight $\\alpha$ would be useful. Additionally, a comparison of performance with and without VIB (especially under different MR values) would clarify its specific contribution.\n\nQUESTIONS:\n1. Experimental setting clarification: It is unclear whether the missing modality setting is consistent across training and testing. For example, when evaluating under “L missing”, is the language modality also missing during training, or only during inference? This clarification is important for interpreting the robustness of the model.\n\n5. Clarification of Figure 3b (Section 4.4): The authors state that the enhanced hyper-modality representations capture more sentiment-relevant information, forming distinct clusters in the t-SNE plot. Could you elaborate more on this point?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, authors introduce Hyper-Modality Enhancement (HME), a novel framework for robust multimodal sentiment analysis under missing modality conditions. By leveraging cross-sample semantic cues to enrich observed modalities without explicit reconstruction, HME addresses limitations of prior methods that rely on pseudo-missing simulations or teacher-student frameworks. The approach comprises three key components: (1) a hyper-modality generation module that distills sentiment-relevant information from other samples via learnable prompts and variational information bottleneck (VIB) for noise filtering, (2) an uncertainty-aware fusion mechanism that adaptively balances original and enhanced representations based on representation reliability, and (3) extensive experiments demonstrating consistent performance gains over SOTA methods on CMU-MOSI, CMU-MOSEI, and IEMOCAP datasets under various missing modalities.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths :\nThe content of this paper presents a technically sound and well-motivated framework for robust multimodal sentiment analysis under missing modalities:\n(1) HME addresses key limitations of prior work by avoiding explicit modality reconstruction and instead leveraging cross-sample semantic cues through learnable prompts and a variational information bottleneck (VIB).\n(2) Three-module design, modality enhancement, hyper-modality generation, and uncertainty-aware fusion, is conceptually clear and addresses both noise reduction and representation enrichment. \n(3) The experimental evaluation is thorough, demonstrating consistent improvements over strong baselines (e.g., IMDer, LNLN) across three datasets (CMU-MOSI, CMU-MOSEI, IEMOCAP) and varying missing-modality scenarios. \n\nWeakness:\nTechnical contributions are solid, but the paper’s evaluation has limitations that temper its impact. \n(1) Experiments focus primarily on within-dataset generalization under controlled missingness patterns (fixed/random missing protocols), but do not explore domain shifts or out-of-distribution robustness. \n(2) Theoretical underpinnings of certain design choices, such as the similarity threshold (tₛ) for cross-sample selection and the interplay between VIB compression and uncertainty weighting, are not fully justified. \n(3) Finally, while the visualizations (e.g., t-SNE plots) aid interpretability, the paper could better articulate how hyper-modality representations qualitatively differ from raw features and why this distinction matters for sentiment prediction.\n\nQUESTIONS:\n1. Authors claim VIB reduces noise and redundancy, but it’s unclear how VIB interacts with the uncertainty-aware fusion mechanism. For example:\n(1)Does VIB’s compression degrade important sentiment signals?\n(2)How does the balance between compression (I(Z,X)) and informativeness (I(Z,Y)) affect performance?\n\n2. Experiments focus on in-dataset missingness (fixed/random masks), but real-world missingness often arises from domain shifts (e.g., sensor failures in deployed systems). How would HME perform on datasets with:\n(1)Natural missingness (e.g., Twitter posts with missing images)?\n(2)Cross-dataset evaluation (e.g., training on CMU-MOSI and testing on IEMOCAP)?\n\n3. Why use cosine similarity for cross-sample selection? Are there alternatives (e.g., contrastive learning)?\n\n4. How does the uncertainty weighting (Eq. 5) relate to uncertainty quantification in Bayesian deep learning?\n\n5.What are the ethical risks of inferring emotions from incomplete data (e.g., misrepresentation of marginalized groups)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is focused on the task of multimodal sentiment analysis with missing modalities. To address this task, the authors propose an approach, called Hyper-Modality Enhancement, which leverages cross-sample information to enrich each observed modality with semantic information from other samples and thus prevent performance degradation when some data modalities are missing. Furthermore, the approach uses an uncertainty-aware fusion mechanism that balances the original and enriched representations to improve model robustness. Experimental results on several benchmark datasets show that the proposed approach performs better than the baselines considered under several missing modality settings.\n\nSTRENGTHS AND WEAKNESSES:\nThe approach proposed is intuitive and at the same time original. The two modules are well motivated and the authors do a good job at both providing the big picture and explaining the details. \n\nThe authors compare the performance of the proposed approach with multiple existing baselines on multiple datasets. Experimental results show that the proposed approach performs better than the baselines under various missing modality settings. \n\nAblation studies show that all components of the proposed approach contribute to the performance of the model. \n\nThere are a few components of the model that rely on prior works and approaches. The authors could provide some additional details about those approaches in the Appendix, if not in the main manuscript, to make it clearer what is novel to their approach. Otherwise, the proposed approach may be seen as rather incremental. \n\nThe authors evaluate the baseline models by comparison with the proposed approach under two missing modality protocols (fixed missing and random missing). At least one of the protocols (random missing) has been used in prior works. The authors used existing implementations to get results for the baselines. It is not clear if the exact data used in prior works (with missing modalities) was not available or why the authors were not able to compare directly with results from prior works. It may be that the results of the baselines are not fully optimized, although the authors followed the specific guidelines from prior works. Also, the authors did not justify well the fixed missing protocol, which may not have been used in prior works.\n\nQUESTIONS:\nAre there any existing multimodal datasets that have missing modalities already, so that one does not have to simulate that using the fixed missing and random missing protocols?\n\nIs the fixed missing protocol used here for the first time? What is the motivation for it?\n\nLIMITATIONS:\nThe authors discuss several limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Hyper-Modality Enhancement (HME)  that addresses the challenge of missing modalities in real-world scenarios. Instead of reconstructing missing modalities for MM sentiment analysis. The main ideas is that HME enriches missing modalities with semantically relevant information from other samples in the dataset. The main components are a Hyper-Modality Representation Generation Module that uses cross-sample enhancement and bottleneck for noise reduction, and an uncertainty-based fusion module that weights modality contribution based on variance estimates.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1.  Addresses an interesting and practical problem in a novel way: the cross-sample enhancement strategy is innovative leveraging information from semantically similar samples across the dataset.\n2. Good evaluation section: Extensive experiments on three benchmark datasets (CMU-MOSI, CMU-MOSEI, IEMOCAP) with both fixed and random missing protocols. Thorough analysis.\n3. Good result: HME achieves notable performance gains  - albeit there is quite a bit of tuning going on \n4. Uncertainty-aware fusion (while not very novel) seem to fit well in this framework\n5. The details in the appendix are much appreciated.\n\n\nWeaknesses\n\n1. Unclear to me how the baseline system compares to SOTA. \n2. Not clear how the proposed method could be plug n play to any MSA architecture.\n3. Scalability/performance concerns to a new dataset with very limited data and unknown distribution\n4. Hyperparameter Sensitivity: The method introduces several hyperparameters (similarity threshold, uncertainty bounds etc.) that that appear to require careful tuning for different datasets (see appendix they seem very spiky)\n\nQUESTIONS:\n1. How do you compare with SOTA for your baseline system?\n2. Is your method applicable to other / any MSA architecture?\n3. Variance based fusion is very flaky. Can you quantify your claims that for missing modalities your variance estimates are more robust?\n4. It would be interesting to note how your hyper-parameters transfer to a new database?\n5. How much data to you really need to identify relevant samples and to tune your hyperparameters?\n6. How would your approach work with more tightly integrated MM MSA frameworks, e.g., https://arxiv.org/pdf/2504.11082 or https://lab-msp.com/MSP-Podcast_Competition/IS2025/\n\n[I will revisit my review after some of these questions and issues are clarified by the authors]\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Hyper-Modality Enhancement (HME) framework for Multimodal Sentiment Analysis which is robust against missing modalities. The main contribution is a two-stage pipeline. First, the \"Hyper-Modality Representation Generation\" module enriches the representation of each available modality by borrowing semantically relevant information from other samples within the same training batch. This cross-sample enhancement reduces the model's reliance on having complete data for every sample. Second, an \"Uncertainty-Aware Fusion\" mechanism adaptively combines the original modality representations with these newly enhanced ones. It uses Variational Information Bottleneck (VIB), allowing the model to dynamically trust the enhanced features more or less depending on their estimated reliability.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The proposed method looks technically sound and well-motivated. The architecture thoughtfully combines several concepts to enhance robustness to missing modalities.\n- The empirical evaluation is thorough and convincing.\n- The paper is well-written, structured, and easy to follow. \n\n**Weaknesses:**\n- The core enhancement mechanism relies on finding similar samples within a mini-batch. This makes the model's performance potentially sensitive to batch composition and size. \n- The pairwise similarity calculation within a batch has a computational complexity of $O(B^2)$, where $B$ is the batch size. While this is acceptable for the batch sizes used in the paper, it could become a bottleneck for training with very large batches, limiting scalability.\n- The methodology section is presented logically but seemed a little bit overloaded with symbols and equations inside paragraphs. Careful rewriting will enhance readability.\n\nQUESTIONS:\n- The cross-sample enhancement is a good idea but is dependent on the quality of neighbors in the batch. Have you considered strategies to mitigate the risk of \"negative enhancement\" from misleadingly similar samples?\n- How does the selection of batch size impact the performance?\n- As mentioned in line 135-137, \"If no representation satisfies the threshold $t_s$, or if the modality is missing, we instead average all available representations of the current modality within the batch to capture the modality-specific information\" - What is the reason behind this assumption? If I understood it correctly, the samples are not very similar and thus they can be from different classes. How does averaging them provide an \"Enhanced\" representation?\n- How does the total number of parameters and number of learnable parameters compare with other methods?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-14"}