Hyper-Modality Enhancement for Multimodal Sentiment Analysis with Missing Modalities
Authors: Yan Zhuang
Minhao LIU
Wei Bai
Yanru Zhang
Wei Li
Jiawen Deng
Fuji Ren
Venue: NeurIPS 2025 poster ()
Keywords: multimodal sentiment analysis; incomplete learning;
Primary area: applications
TL;DR: HME tackles the challenge of missing modalities in multimodal sentiment analysis by generating hyper-modality representations, enhancing information integration and reducing reliance on complete datasets.

Abstract: Multimodal Sentiment Analysis (MSA) aims to infer human emotions by integrating complementary signals from diverse modalities. However, in real-world scenarios, missing modalities are common due to data corruption, sensor failure, or privacy concerns, which can significantly degrade model performance. To tackle this challenge, we propose Hyper-Modality Enhancement (HME), a novel framework that avoids explicit modality reconstruction by enriching each observed modality with semantically relevant cues retrieved from other samples. This cross-sample enhancement reduces reliance on fully observed data during training, making the method better suited to scenarios with inherently incomplete inputs. In addition, we introduce an uncertainty-aware fusion mechanism that adaptively balances original and enriched representations to improve robustness. Extensive experiments on three public benchmarks show that HME consistently outperforms state-of-the-art methods under various missing modality conditions, demonstrating its practicality in real-world MSA applications.

Hyper-Modality Enhancement for Multimodal Sentiment Analysis with Missing Modalities
Yan Zhuang lit, Minhao Liu 12.5 Wei Bai², Yanru Zhang!? Wei Li' Jiawen Deng" Fuji Ren
Universiiyy of Electronic Science and Technology of China
Shenzzen Institute for Advanced Study, UESTC, Tsinghua University
(202211081370, 202331102001111111111111111 101111001139111 westc. edu. cn, ba23@maile, tsinghua. weigghaa.eeu..... [minhaoliu, yarruzangg dengjj, renfrrrreeejjjuuutc....m en Equal contribution "Corresponding authors
Abstract
Multimodal Sentiment Analysis (MSA) aims infer human emotions by inte- grating complementary signals from diverse modalities. However, in real-world scenarios, missing modalities common due data corruption, sensor fail- ure, or privacy concerns, which can significantly degrade model performance. To tackle this challenge, we propose Hyper-Modality Enhancement (HME), a novel framework that avoids explicit modality reconstruction by enriching each observed modality with semantically relevant cues retrieved from other samples. This cross-sample enhancement reduces reliance on fully observed data during training, making the method better suited to scenarios with inherently incomplete inputs. In addition, we introduce an uncertainty-aware fusion mechanism that adaptively balances original and enriched representations to improve robustness. Extensive experiments on three public benchmarks show that HME consistently outperforms state-of-the-ar methods under various missing modality conditions, demonstrating its practicality in real-world MSA applications.
1 Introduction
Multimodal Sentiment Analysis (MSA) aims to identify an individual's emotional state toward given topic, person, entity by leveraging complementary signals from modalities such language, audio, and video [T3]. By integrating verbal, vocal, and visual cues, MSA offers a more nuanced understanding of human emotions and has been widely applied in emotion- human-computer interaction and social media analysis. Considerable progress has been made in designing sophisticated fusion networks to integrate multimodal information 4 However, in real-world scenarios, missing modalities frequently occur due to factors such as data corruption, sensor failures (10), privacy concerns 11 12). which pose significant challenges to the robustness of MSA systems.
systems like
Recent efforts have explored the challenge of MSA under missing modality conditions [3318 For example, IMDer [15] adopts
model to reconstruct missing modalities, while HRLF [18] employs teacher-student framework, where the teacher model is trained with fully observed data and the student model learns to handle missing modalities by knowledge transfer. Although these methods achieve promising results, they share two key limitations. (i) They typically adopt pseudo-missing setting, where missing modalities are artificially simulated by masking full-modality training data, while the ground-truth representations of the masked modalities used for supervision. However, such fully observed samples are often scarce in practice, limiting the scalability and real-world applicability of these methods. (ii) These approaches mainly rely on
still
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

the modalities available in the current sample, overlooking the potential of leveraging cross-sampe information that could enrich the representations and enhance the robustness of sentiment predictions.
To address these challenges, we propose Hyper-Modality Enhancement (HME), a framework de- signed to address missing modalities without relying on full-modality reconstruction. Specifically, HME consists of two key components: (i) Hyper-Modality Representation Generation: This com- ponent learns shared representation for the current sample using small set of learnable prompts, conditioned on its available modalities. Simultaneously, it extracts modality-specifie information from other samples to enrich this shared representation with sentiment-relevan signals. To ensure robustness, we apply Variational Information Bottleneck (VIB) [9), which filters out irrelevant or noisy information. (ii) Uncertainty-aware Fusion: Given that representations from different modalities and samples may vary in reliability, HME incorporates an uncertainty- mechanism that estimates the confidence of each input. This mechanism adaptively integrates these representations, mitigating the influence of noisy or unreliable signals and enhancing robustness under missing modality conditions. Together, these components mitigate the limitations of pseudo-missing training strategies and enhance the model's robustness in coping with realistic missing modality patterns. Our main contributions are as follows.
fusion
A novel hyper-modality enhancement mechanism is introduced leverage cross-sample semantic cues for strengthening observed modalities, without relying on explicit modality reconstruction.
An uncertainty-awwe aware fusion mechanism is developed to adaptively combine original and en- riched features, enabling robust sentiment prediction under incomplete modality conditions.
Our extensive experiments on three benchmark datasets demonstrate that HME achieves superior performance and greater robustness compared to existing methods.
2 Related Work
2.1 Multimodal Sentiment Analysis with Missing Modalities
Multimodal Sentiment Analysis (MSA) aims to predict sentiment by integrating information from language, video, and audio modalities. However, many established approaches (5] 7 20 assume the consistent availability of all modalities across training. validation, and testing. Their performance often degrades significantly when any modality missing 14 15 18] a scenario frequently encoun- tered in real-world applications. To tackle this challenge, some approaches attempt to reconstruct missing modalities from the observed ones within the same sample like DiCMoR (24), IMDer [15), GCNet 13. MCTN (25) MMIN 26 DCCA 227. and DCCAE (28), often relying on ground-truth representations of the masked modalities for supervision. Others adopt a teacher-student framework, where teacher model trained on full-modalit data guides student model operating on incomplete inputs 16 [1]. Although effective when trained on fully observed datasets, these methods commonly assume access to complete data during training supervise either the reconstruction or the knowledge transfer process. This assumption limits their applicability in scenarios where full-modality samples are scarce or unavailable. Besides, existing methods often treat each sample independently, overlooking valuable contextual cues that could be leveraged from semantically related instances. In contrast, our proposed HME framework operates directly on incomplete data, without relying on modality reconstruction or teacher supervision. It enhances representations by leveraging contextual information from related instances, enabling more robust sentiment prediction.
2.2 Hyper-Modality Representation
Recent advancements in multimodal learning have explored "hyper-modality representations" 21 to efficiently integrate information from diverse sources. These approaches often employ compact prompts distilled representations of individual modalities to capture resk-relevant features. For instance, MPLMM (29) pre- pre-trains complete multimodal datasets, distilling this comprehensive knowledge into prompts that can later supplement or stand in for missing modalities. Similarly, methods like ALMT 21 and LNLN [14] construct hyper-modality representations by distilling information from other modalities into prompts, but crucially, they presume the constant availability of the language modality as primary anchor. While these techniques demonstrate strong performance,

---

Pre-Processing
Hyper-Modality Representation Generation
Batck Dats
Langunge
Video
Audio
Hy
Enhaneement
Modality
Ey
Modulity
Hyper-
Ry
R
ComrDD
Data Flav
Es
Generation
RA
Pooling
Feature Extraction
Eyie o FL
Variational
Mindag Modaliiss
Information
+v..
Xy
Bottleneck
FAFFA
Modatinee
Modality Drop
XL Fratuse
Origissl
X
F1 C1
Hy Fy dv
HA FA
X1
Preconcd
Fraturs
Xx
H1
Modally
Entrefflag
Uncertainty Attention
Uncertainty Attention
Uncertainty Attention
Uncertainty Attention
EL Fabooliigs
Eshunced
Modality Processing
Hype:
R1 Rejresmuti
Medally
X1
4
Tramformer
Eacader
S
Sp
Trassfrrmer
Hy
Fi
Dowisel
Inhalding
Transformer Encoder
dG
Tramnfernee
OHHH
MLP
LyASk
5t.
Tised
Entheddings
Multimodal Hyper-Modality Fusion
9 Predictions
Figure 1: Visualization of HME framework. It consists of three modules: Pree-rocessing Module,
Hyper-Modality Representation Generation Module and Multimodal Hyper-Modality Fusion Module.
their practical applicability is often limited by their core assumptions: either the necessity of pre- training on complete datasets or the continuous presence of a specific modality. Such requirements frequently unrealistic in real-world scenarios. In contrast, our proposed HME model is designed to robustly generate hyper-modality representations directly from any available modality, without these restrictive prerequisites.
Methodology
The overall structure of the proposed HME is shown in Figure 1 consists of three modules, the Pre-Processing module processes the multimodal input to extract the modality rep- resentations (introduced Section (.2). the Hyper-Modality Representation Generation Module provides enhanced hyper-modality information (introduced in Section 3.3). and Multimodal Hyper- Modality Fusion module fuses the enhanced representations with original representations based on the uncertainty of the enhanced representations (introduced in Section 3.4
3.1 Problem Definition
Given the multimodal input D = xX. each representation X E R xd., is char- acterized by a sequence length Tm and feature dimensionality dm, where m E (T., V, A} denotes modality and N is the length of the dataset. For simplicity, we omit the superscript i and use X,, to denote the representation of modality m. In real-world scenarios, some modalities may be missing for individual samples. To model this, missing modality representations are replaced with zero vectors [6 13 [4]. For simplicity, we use X to uniformly represent the modality m regardless of whether it is present or not. The goal of MSA with missing modalities is to predict the sentiment using incomplete modalities (X), X1, )
3.2 Pre-Processing Module
Representations of each sample are firstly extracted to form the initial modality representation [XL.Xyy- XA). To simulate real-world scenarios with incomplete data, random modality dropout applied to generate inputs with missing modalities, denoted as {X), X', X'A- For each input modality X",. m E {L, A). following prior research [16 22 we then use ID temporal convolu- tional layer with kernel size of 3 x standardize all modalities to the same dimension (d) and sequence length (I)) Position embeddings [30) are added to encode temporal information, resulting
3

---

Ex
Atlention
Perceiver
E,
Attention
R2
Ey
Attention
Modality Enhancement (Language Modality)
Hyper-Modality Ceneration (Language Modality)
(a) Modality Enhancement modulc.
(b) Hyper-Modality Generation module.
The pipeline of Modality Enhancement (a) and Hyper-Modality Generation (b) module (e.g. using language modality). Here 'Q, V denote the query, key and value attention, respectively.
Figure 2:
in: Xm = Wax3(X,,) m + PE(T, d), where PE d) denotes position embeddings. These represen- tations passed through transformer encoder [30, F2 (-), capture dynamic modality -ppecicc information. Finally, the Global Average Pooling (GAP) layer is applied to produce compact modality representations: Hm GAP(F pm (Xm)), where H,, R om is the network parameter.
3.3 Hyper-Modality Representation Generation Module
The Hyper-Modality Representation Generation Module generates the enriched modality represen- tation by injecting sentiment-relevann information from other samples. Given modality-speciffc representations Hm. the module sequentially applies three components: Modality Enhancement, Hyper-Modality Generation and Variational Information Bottleneck, respectively. First, Modality Enhancement strengthens each modality's 's and each sample's representations via few learnable prompts, enabling prompts to dynamically gather informative signals from samples in current batch. Then, Modality Generation fuses the enhanced modality specific and shared information, se- lectively integrating cross-sampp sentiment cues into a richer, hyper-modalic, representation. Finally, Variational Information Bottleneck filters noise and redundancy, distilling compact, high-quality representations. We next detail each sub-module.
Modality Enhancement. For each modality a similarity matrix is first computed using samples in the current batch, denoted as Sm E BBx where B is the batch size. Taking modality as an example in Figure 2 (a), the similarity between representations is measured using cosine
ATh, sim,h,, hy) = ||A.II-Ilb,|1 where h; and h; denote different samples within the same modality m in current batch. To better handle missing modalities and to reduce the reliance on individual representations- wwhic can introduce ohile- ensuring that the selected representations remain meaningful, we simultaneously apply a similarity threshold and averaging strategy. Specifically, a similarity threshold to set: representations with similarity scores exceeding to are selected and then averaged form an enhanced representation. If representation satisfies the threshold l., or modality is missing, we instead average all available representations of the current modality within the batch to capture the modality-specii specific information. This process is formulated as:
similarity 16 31:
H - (ug((m((dd)) ifldr + Ø (ugg(Hm. * 0), otherwise
where Idx (S.m > t.] denotes the indexes of the representations, of which the similarities with i-th representation are higher that t.. And Hm.j + 0 denotes the non-missing representations of modality m. Aug(-) denotes averaging the representations.
After obtaining the enhanced representations H'm Perceiver model 32 refine these representations. Specifically, the Perceiver model uses learnable prompts to learn from the representations by using cross modal transformer blocks and transformer blocks. In the cross- modal transformer, the imput representations H',, serve keys and values, while learnable prompts E'm E xd act as queries. Here lp is the length of the prompt. And the attention (30] is defined as:
employed to
E'. WQ. WT (H",) )H Wy.
Em = A dm... (E', H' ) Softmax(
Vd
where WQ., WT, Wr, E dx are trainable weights. Subsequent layers apply standard transformer structure with identical input for the query, key, and value. The final enhanced modality representation Em E ddd is obtained by averaging the output of the last transformer layer.

---

To capture common information across modalities and facilitate hyper-modality interaction, we introduce another set of learnable prompts E's E R4xd These prompts are designed aggregate features from all available modalities for each sample, even when some modalities are missing. The computation of Eig follows a similar procedure to Eq. 2 with three key modifications: (i) replacing F'm with F''; (ii) replacing Hy with the concatenated features [ Hv, Ha); and (iii) adjusting the dimensions of the associated trainable weights accordingly. The final representation Eg E IR' is also obtained by averaging the output of the last transformer layer.
Hyper-Modality Generation. This module generates the representation of each modality of the current sample by integrating sentiment-relevant information from other samples, thus obtaining the hyper-modality representation, which can be seen in Figure 2 (b). Taking modality L as an example, we compute the enhanced representations of modality L with modalities V and A using Eq.2 setting queries to Et, keys and values to EA and Ev. These are defined as EL, ATTNG,,. (EL, Ey) and E ATTN (EL, EA). The sentiment-relevant information are then distilled into the overall representation of each sample through setting Es as query: Rg A s. (Es, ELL,, EL,A)). Through performing on all three modalities, we obtain Rm ATT mm.m (Eg, mm Em,mal); where m, m,,, m2 E (L, V, A), m # mi * m2.
Variational Information Bottleneck. Selecting representations based solely on similarity can introduce errors, samples with similar features may have different labels. Furthermore, the uncertain availability of modalities can also lead to redundancy when integrating information. To address these challenges, we employ the Variational Information Bottleneck (VIB) [19] reduce noise and redundancy in the representations. VIB approximates the information bottleneck (33 by learning encoding Z that is maximally expressive about target Y while being maximally compressive about input X through optimizing the following function:
LVIB I(Z,Y)
X).
Here I(.) is the mutual information, ,3 is the Lagrange multiplier. Specifically, we follow the previous ways that use the CTASK (introduced in Eq. 9 and Kullback Leibler divergence [19 336 to measure the I ((, Y) and 1(Z through:
N
LVIB = N
1
Σ TASK (1)h v)) + 3KTTp((//>>))) I)),
where: p(z](a))) ~ N(i; Ii,, and pr = So (zi), o; = So (a::). Here two MLP layers Sa, and soz are used to encode the mean and standard deviation of z;. and another MLP layer Sog is used to map the 2; into the logits 91th using 8.th = fo, (2.). And the ce-parameterization trick [37] is also adapted as: 2; = 14 + eas, E EV I). Here 2; is the compressed denoised representation of 24, and the whole representation of each modality can be denoted Fm E Rd. Through assigning X Rtm and Z = Fm. in Eq. 3 we obtain the compressed hyper-modality enhanced representations F,, and their corresponding standard deviation o. for each modality m.
3.4 Multimodal Hyper-Modality Fusion Module
The effectiveness of multimodal fusion depends on the reliability of the representations from each modality. As the representation power of compressed representations can vary, it is essential to account for this variability during the fusion process. Drawing inspiration from the EAU framework [36), which uses variance measure of uncertainty in representations that higher variance indicates more scattered representations, we incorporate the standard deviation hyper-modality representations to guide their fusion with the original representations.
Specifically, we compute the reciprocal of the standard deviation, expressed as Gm = o.. teps where eps 100 prevents division by zero. Additionally, since the difference of the standard deviation may very high, we limit the influence of uncertainties using thresholds t and f,, constraining Sm to lie within [t,, t.]. Besides, the contribution of the enhanced hyper- hyper-modality modality should not exceed the
1
original modality representation. thus we define the uncertainty weights as: U2. [1.0, (m. to fuse the Hm and F... Furthermore, the interactions between the hyper-modality enhanced representations Fm may also provide sentiment-relevant information, thus we define the uncertain weights of these three representations as:
exm
U3
(5)
Σ Eme(L.V.A) E5m
5

---

Then the uncertainty weights are combined with attention weights [30] for further fusion through:
Hm Fm W a.m WT. h.m
A
(6)
Vd
and
4 Flav W W. o.
vd
where Wam,, Wh.m W..W a. are trainable parameters, and Flav = [FL. Fv. Fal-
The integrated representation of each modality is then fused with A through: Sm = So Softmar(U) in A [HH,, F,], and the fused representations of three hyper-modality enhanced rep- Suftma A Fiav. The fused representations are then
resentations is calculated as: Sμ concatenated across modalities to form Sall = [S Sy SH], where Sall E RR Saul is then fed into a transformer encoder, denoted (.) for further fusion. And the output, Himut is flattened into R and passed through two-layer MLP fe,. The MLP reduces the representation first to Id and then to Rt, producing the final prediction ÿ. Here, l represents the dimensionality of the task labels. The process can be formulated as: fo, (Fa (Sai)).
SA.
Overall Loss. The overall loss for the HME framework is defined as:
Laut = CTASK (ú.y) + aLyin-
8
Here the Lvin is averaged over the three modalities, and LTASK for regression and classification tasks is defined as:
[CrossEntropu(
CTASK-
(9)
100 - vl
Regression
4 Experiments
4.1 Datasets and Implementation Details
Datasets. Three widely-used datasets are adapted, named CMU-MOSI (38) CMU-MOSEI [39 and IEMOCAP E0.. CMU-MOSI and CMU-MOSEI consist of 2,199 and 22,856 video clips, respectively. Each clip has regression label ranging from -3 (strongly negative) to 3 (strongly positive). While IEMOCAP is a dialogue dataset with 4,453 samples, categorized into four emotional classes: neutral, happy, sad, and angry. Following prior research (29 [13, the 'positive/negative' accuracy and Fi-score is adapted in MOSI and MOSEI, and averaged accuracy and Fi-score of four classes is used to evaluate the IEMOCAP.
Feature Extraction. For text, the pre-trained BERT-base model embeddings. For the video modality, we perform face detection and feature extraction using the MTCNN 42 and OpenFace (43) toolkits. For audio, Mel-frequency cepstral coefficients and pitch features are extracted using the COVAREP toolkit 44
used obtain contextual
Implementation Details. The performance of each model is evaluated with two protocols, fixed missing protocol and random missing protocol. For the former protocol, we test the performance on the test set with the same missing modality. For example, 'L' in Table that only language modality is available. Under this protocol, the modality is only removed during validation and
inference- not during training, which aligns with following existing approaches [13 15] 24 29 and
is intended to simulate real-world scenarios where certain modalities may be missing only at test time. While for random missing protocol, following existing approaches [13 15 24), we randomly
drop some modalities for each sample. Here missing rate (MR) is adapted to evaluate the missingness
of the dataset. The MR is defined as MR = 1 - modalities in 8th sample, 'N' denotes the total number of the samples in each dataset. Since least one modality is present, so the a; 2 0 and MR < 3. Following GCNet [13], we also choose the MR from 0.1. .... 0.7], where 0.7 is the approximation of The MR is kept the same in both training, validation and testing phase. To mimic the real-world situations, we set the paradigm in training, validation and testing phase that the modalities are dropped and can not be used for guidance for reconstruction. Due to the space limits, more implementation details can found Appendix A.2.
N Σm
a: NxN- where a; is the number of the available

---

Table 1: Performance MOSI and IEMOCAP with missing modalities. 'ACC/F1' is reported. '+':
results under fixed missing protocol; results under random missing protocol.
Datasets
Avail
GCNet
MPLMM
DiCMoR
IMDer
LNLN
HME
L
83.7/83.6
83.8/83.8
84.5/84.4
84.8/84.7
84.9/85.1
85.7/85.6
V
56.1/55.7
54.7/42.1
62.2/60.2
61.3/60.8
52.2/58.9
63.6/63.4
A
56.1/54.5
54.6/42.0'
60.5/60.8
62.0/62.2
52.2/58.9
62.7/62.2
MOSI
L
84.3/84.2
83.8/83.71
85.5/85.4
85.5/85.4
84.9/85.1
85.8/85.8
L.A
84.5/84.4
83.7/83.6
85,5/85.5
85,4/85.3
84.9/85.2
85.8/85.7
A.V
62.0/61.9
57.3/42.6
64.0/63.5
63.6/63.4
$2.2/58.9
64.6/64.5
L,A.V
85.2/85.1
83,8/83.87
85.7/85.6
85.7/85.6
84.3/84.6
86.4/86.3
Avg.
73.1/72.8
71.7/65.9
75.4/75.1
75.5/75.3
70.8/73.8
76.4/76.2
L
-776.1
69.2/69.3
75.2/68.2
75.1/67.2'
76.8/74.4
79.1/78.1
V
-661.6
57.6/57.0
73.6/68.81
75.1/70.1*
74.5/69.9
75.7/70.8
A
163..5
59.8/59.7
76.0/68.7'
75.5/70.2
77.1/74.6'
78.9/76.8
IEMOCAP*
L.V
-77.4
74.7174.5
74.4/72.1
75.2/72.6
77.0/75.5
79.7/78.9
L,A
-179.1
76.0/75.4
78.7/76.51
76.6/71.7
77.9/75.4
80.7/80.0
A.V
-165.4
67.3167.4
76.3/75.21
76.3/72.3
77.3/74.9
78.1/76.3
L,A.V
-/82.7
+
78.7/76.5
77.5/75.4
78.4/77.4
81.1/80.6
Avg.
-77.3
67.4/67.2
76.1/72.3
75.9/71.4
77.0/74.6
79.0/77.4
0.0
85.2/85.1
83,8/83.8'
85.7/85.6
85.7/85.7
84.2/84.0'
86.4/86.3
0.1
82.3/82.3
80.6/80.71
83.2/83.21
84.6/84.4
81.4/8144
84.9/84.7
0.2
79.4/79.5
77.7/77.9
81.6/81.3
81.7/81.8
78.7/78.7
82.9/82.9
0.3
77.2/77.2
75.0/75.1
78.4/77.91
79.9/79.4
74.7/74.8'
81.1/81.0
MOSI
0.4
74.3/74.4
70.0/69.8
76.2/74.7
78.1/77.2!
70.3/69.9
79.9/80.0
0.5
70.0/69.8
67.4/66.6
72.6/72.71
73.6/73.7!
67.1/66.3
76.4/76.4
0.6
67.7/66.7
63.4/61.8*
71.3/71.4*
72.9/69.81
62.5/60.5
74.5/74.4
0.7
65.7/65.4
59.2/56.2*
69.1/69.2*
67.7167.1*
61.0/5.33
73.5/71.7
Avg.
75.2/75.1
72.1/71.5T
77.3/77.0
78.0/77.4
72.5/71.7
80.0/79.7
0.0
178.4
76.4/72.0
78.7/76.51
77.5/75.4
78.4/77.4
81.1/80.6
0.1
-177.5
76.1/70.8
77.1/73.71
77.2/72.8
78.0/76.3
80.9/80.3
0.2
-177.3
75.7167.9
76.4/72.5
75.9/71.6'
77.6/75.7
80.7/80.0
0.3
-/76.2
75.6/67.4
75.8/71.6
75.7/70.8
77.5/75.4
79.0/77.8
IEMOCAP
0.4
-/75.1
75.5/66.6*
75.5/69.0
74.9/70.8
77.4/44.0
78.7/77.3
0.5
-/73.8
74.0/66.2
75.1/68.5t
74.7168.21
76.8/72.9
78.5/77.1
0.6
-171.9
73.2/66.0"
74.5/68.0*
73.5/67.0*
76.6/72.6
77.8/76.1
0.7
-771.4
72.4/65.7
74.3/67.5'
73.1/65.3/
76.5/70.21
77.6/76.0
Avg.
-175.2
74.9/67.8
75.9/70.9
75.3/70.2
77.4/74.3
79.3/78.2
4.2
Comparison with the State-of-the-arts
We compare HME with the state-of oftt--of-the-an the-art models specifically designed for MSA under missing modality scenarios, including GCNet [13,, IMDer [I5), DiCMoR [24), LNLN (14) and MPLMM 229. In some cases, we were unable to directly compare with results reported in prior work under identical settings-either because the specific missing modality protocol was not used, or the results for certain datasets were not reported for those conditions. For a fair comparison, we re-implement the models from their open-source codes according to our training paradigm, which are denoted with symbol f, More results on MOSEI dataset can be found in Appendix A.3.1.
Quantitative Results. Table I summarizes the performance of HME and baselines across MOSI and IEMOCAP datasets under the both missing modality protocols. Under fixed missing protocol, HME generally outperforms existing models. On MOSI dataset, HME surpasses all baselines across every metric, achieving an average performance increase 0.9. For the IEMOCAP dataset, HME performs comparably to GCNet when all modalities are available but demonstrates significant advantages over other models in scenarios with missing modalities. On average, across all missing modality scenarios, HME improves ACC by 2.0 and Fl score by 2.8 over the best-performing baseline. This strong performance extends to the random missing protocol. On both MOSI and IEMOCAP datasets, HME consistently outperforms all models across all metrics and at every MR. Specifically, HME yields

---

Table 2: Ablation results of HME components MOSI and IEMOCAP datasets.
Datasets
HME
w/o ÉL
w/o Ev
w/o EA
w/o Es
w/o Um
w/o HMG
Ablations on Fixed Missing Protocol
MOSI
76.4/76.2
75.2/74.7
75.2/75.1
75.3/74.6 75.2/74.7
75.4/75.1 79.1/76.3
75.0/74.5 78.5/76.8
JEMOCAP
79.0/77.4
78.4/76.3
78.1/76.2
78,5/76.5
78.2/76.1
Ablations on Random Missing Protocol
MOSI 79.0/78.7
77.3/76.8
77.376.7
76.7/76.2
77.2/76.9
77.6/77.2
76,6/6.22
IEMOCAP 79.0/77.8
78.4/76.8
78.4/76.7
78.3/76.5 78.4/76.9
78.2/76.4
78.7/77.2
average improvements in ACC and Fl-score of 2.0 and 2.3, respectively, on MOSI, and 1.9 and 3.9, respectively, on IEMOCAP. The consistent superiority of HME across both missing protocols underscores its effectiveness and robustness in handling various missing modalities.
4.3 Ablation Study
To better understand the contribution of each component in HME, we conduct an ablation study on the following parts across both missing modality protocols: (i) 'w/o Em," refers to the model without the hyper-modality enhanced representation of modality m; (ii) 'w/o Es' refers to the model without
the overall representation of the current sample; (iii) 'w/o Um" refers the model without uncertainty weights, using only vanilla attention weights for fusion; and (iv) 'w/o HMG refers to the model without the hyper-modality generation module, where the modality representations Em are directly passed into the VIB for further processing. The results of averaged performance of two missing protocols are presented in Table 2 with more detailed results in Appendix A.3.2.
Effects of the HME components. In both missing protocols, removing any component resulted in decline in performance across all datasets and missing protocols, highlighting the effectiveness of each part of HME. Moreover, the importance of specific components varies across datasets. For example, removing the HMG module causes the largest performance drop in MOSI datasets under both missing protocols, whereas on the IEMOCAP dataset, the removal of Um results in the largest performance decline. Furthermore, the removal of uncertainty weights Un also leads substantial drop in performance, particularly in F1 score, emphasizing the role of uncertainty in accurately guiding the model distinguish between sentiment categories. Overall, Um. and HMG contribute the most to performance improvement. For example, removing HMG leads to the largest performance drop on the MOSI and MOSEI datasets under the fixed missing protocol, while removing Um has the most significant impact under the random missing protocol in IEMOCAP and MOSEI.
4.4 Visualization of the Training Loss and Representations
To better analyze the convergence of the loss components during training, conduct experiments on the IEMOCAP dataset with an MR value of 0.7, which depicted in Figure (a). Note that the TVIB,1 and LVI the two components of LvIB- Specifically, training is conducted over 100 epochs, with early stopping applies at the 60th epoch. We observe that all loss components of HME gradually decrease loss components. This consistent reduction in both overall and local losses, with minimal fluctuations, further validates the effectiveness of HME's loss component structure.
time and stabilize after few epochs, indicating the rational design of HME's
Besides, to understand the differences between the enhanced hyper-modalin representations Fm and the original representations Hm. we visualize them using t-SNE. Figure (b) shows these visualizations on the MOSI test set with an MR of 0.7. We find that the enhanced hyper-modality representation Fm captures additional sentiment-relevant information, forming distinct clusters that do not overlap with the original representations Hm. More visualizations for the fused representations S.n and Sh be found in Appendix A.3.9.
4.5 Qualitative Analysis
To investigate the relationship between uncertainty weights and the input modalities, we compute the average uncertainty scores across three modalities under 14 missing-modality modality scenarios derived from two pistioo- setups on the validation set. Fach scenario is executed five times to ensure reliability, and the results, including 95% confidence intervals, presented in Table Further

---

1.0
0.8
Luil
H
Training Loss
Lrask
0.8
Ha
0.6
LvBB
1
Hv
Lypa
0.6
0.4
L
FL
0.4
FA
0.2
Fv
0.2
0.0
10
20
30 Epochs
40
50
60
0.0
0.0
0.2
0.4
0.6
0.8
1.0
(a)
(b)
Figure 3: Visualization training loss convergence on the IEMOCAP dataset (a) and learned representations on the MOSI dataset (b) with MR 0.7. (a) All losses show clear convergence during training. (b) The enhanced hyper-modality representations capture distinet information compared to the original ones, forming separate clusters.
Table 3: Average uncertainty weights of the three modalities IEMOCAP dataset.
MR
0.1
0.2
0.3
0.4
0.5
0.6
Fr
0.026--0003
0.424
+0.007
0.006
0.323+00.019
0.322+-0003
FA
0.24410.034
0.24810.000
0.2991100029
0.319:10.046
0.338:10.037
0.34811.003
0.330=00.034
0.32810.004
0.360:10.025
0.34510.002
0.339:10.031
0.33010.029
V
L.A
L.V
A.V
Fg
0.200--0.034
0.273
0.322+0.001
0.301
0.312+0.028
0.364
FA
0.350=0.004
0.467 134
0.389+00.00
0.305+:0.013
0.416:10.062
0.311=10.026
Fv
0.360=0.028
0.26010.104
0.289
0.344110011
0.272:10.041
0.325=100017
t
0.1
0.2
0.3
0.4
L,,V((t,.77
MR=0.7
Fr. 0.358-0.012
0.35310.016
0.432110003
0.461110.003
0.409:10.003
0.29310.002
FA 0.219-0.013
0.2381100117
0.24711.004
0.23010.000
0.257-10.009
0.36511.021
Fv 0.423-0.010
0.403+0.012
0.318+00004
0.310+0.003
0.335++0.005
0,342+00031
analyses of model stability (in A.3.3), error (in A.3.4), generalization (in A.3.5), VIB components (in A.3.6), enhanced representations (in A.3.7), hyper-parameters (in A.3.8), case visualization (A.3.9) detailed in the Appendix.
and computational efficiency (in A.3.10
Our findings reveal that the uncertainty weights of the three modalities are strongly influenced by the number of missing modalities (i.c., the MR value). Specifically: (i) When fewer modalities are missing (low MR, e.g., MR 0.0, 0.1, 0.2): The language modality exhibits the smallest variance in its enhanced representations, providing the most information, while the audio modality shows the largest variance. This may explain why prior studies often regard language as the primary modality in fusion process [4 21]. (ii) As the MR value increases: The audio modality's variance gradually decreases, and its uncertainty weights increase, reflecting its growing contribution to the fusion process. Conversely, the language modality's uncertainty weights decrease as its representation variance grows. (iii) Similar trends are observed under the fixed missing protocol. For example, when only one modality is present (M M = 0.7), the audio modality tends maintain smaller variance.
Since the hyper-modality representations are generated from representations that satisfy the similarity threshold t., we further analyze the relationship between uncertainty weights and f... Specifically, we conduct experiments under the full-modality scenario (MR 0.0, corresponding to L.A.V) to isolate the impact of f, uncertainty weights, which is shown in Table 3 Our findings reveal the following: (i) When l, is low (e.g., 0. 0.2): The video modality demonstrates the smallest variance in its representations, resulting in the highest uncertainty weights, while the audio modality exhibits the largest variance. (ii) As l, increases, The language modality's uncertainty weights rise, gradually aligning with its behavior in low-MR scenarios, where it dominates with higher uncertainty

---

Input
Selected Samples For Enhancement
AND CAMERON SUFROSED TO BE PLUCKY I GUESS BUT SHES JUST KINF OF USELESS
AND STILL BORING!
NOTHING THATS WORTH GOING TO THEATERS TO SEE
UM YEAH THE CHARACTERS DIDNIT CARE WEATHER THEY LIVED DIED
Language;
Audio
Figure 4: Case visualization of selected samples with t. = 0.9 on the MOSI dataset. Green boxes indicate representations with the same label, while red boxes denote samples with different labels.
Input CAMERNN IDIAZ WHOS SUPPOSED TO BE PLUCKY I GUESS BUT SHES JUST KINF OF USELESS
Selected Samples For Enhancement
AND HE WAS STILL BORING!
NOTHINS THATS WORTH GOING THEATERS
WHO SEEMED TO HAVE SENUINE AFFECTION FOR OTHER HUMAN BEINGS
Language;
Video
Audio
Figure 5: Case visualization of selected samples with t. = 0.5 on the MOSI dataset. Green boxes indicate representations with the same label, while red boxes denote samples with different labels.
weights. (iii) Across all t. values: The audio modality's 's uncertainty weights remain consistently low, mirroring its behavior in low-MR scenarios.
To better understand the the proposed modality enhancement, we visualize selected sample pairs from the MOSI dataset under different similarity thresholds. Specifically, we examine two configurations with a MR of 0.7 and batch size of 128: (i) similarity threshold of 0.9 (Figure 4 and (ii) similarity threshold of 0.5 (Figure 5 A higher similarity threshold tends to select samples that are more semantically consistent with the current label. Conversely, lowering the threshold increases sample diversity but also introduces a few mismatched labels. Nonetheless, the subsequent VIB and uncertainty-awerr fusion mechanism effectively reduce the impact of these mismatches. Further analysis of the effects batch size and selected samples is provided the Appendix A.3.9.
Conclusion and Discussion
In this paper, we present the Hyper-Modality Enhancement (HME) framework to address the chal- lenges posed by missing modalities in multimodal sentiment analysis. By integrating sentiment- relevant information from other samples, HME effectively enhances the representation of the sample. The incertainty-aaan fusion strategy further ensures that the enhanced representations are robust and reliable, when generated data varies in quality. Experimental evaluations on multiple benchmark datasets demonstrate that HME outperforms existing state-of-thh-arr methods, achieving superior performance in terms of both accuracy and robustness when handling missing modalities.
Discussion of Limitation. While the use of VIB helps reduce redundant information, generating separate representations for each modality can still result in overlapping content. Besides, HME only focuses on mitigating noise from other samples but does not account for potential noise within the original modality representations themselves. We plan address these challenges in future work.
Discussion of Societal Impacts. Recognizing emotions from incomplete multimodal data enhances the robustness and usability of MSA systems but also raises ethical concerns. Inference from partial signals may lead to inaccurate or biased predictions, potentially misrepresenting individuals or groups. Moreover, the use of MSA models can introduce privacy risks or enable surveillance without consent.
10

---

Acknowledgments
This work was supported by the National Natural Science Foundation of China (Grant No. U24A22200 and No.62 (0.62276048), the Natural Science Foundation of Sichuan Province (Grant No. No.2024NSFTD0042. No.2024 20.2024NSSS00006, and No. N0.222522NFSC114877 and the Fundamental Re- search Funds for the Central Universities (No.ZYG220240022 and No.ZYGX20242005),
References
[1] R. Das and T. D. Singh, "Multimodal sentiment analysis: survey of methods, trends, and challenges," ACM Computing Surveys, vol. 55, no. 13s, pp. 1-38, 2023.
[2] A. Gandhi, Adhvaryu, S. Poria, Cambria, and A. Hussain, "Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions," Information Fusion, vol. 91, pp. 424 444, 2023.
[3] P. P. Liang, A. Zadeh, and L..P. Morency, "Foundations and trends in multimodal machine learning: Principles, challenges, and open questions." arXiv preprint arXiv:2209.03444 2022.
[4] G. Hu, T.-E. Lin, Y. Zhao, G. Lu, Y. Wu, and Y. Li. "Unimse: Towards unified multimodal sen- timent analysis and emotion recognition," " in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 7837-7851.
[5] Y. Li, Y. Wang, and Z. Cui, "Decoupled multimodal distilling for emotion recognition," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023, pp. 6631-6640.
[6] R. Lin and H. Hu, "Multi-task momentum distillation for multimodal sentiment analysis." IEEE Transactions on Affective Computing, 2023.
[7] Y. Zhuang, Y. Zhang, Z. Hu, X. Zhang, J. Deng, and F. Ren, "Glomo: Global-local modal fusion for multimodal sentiment analysis," in Proceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 1800-1809.
[8] Y. Zhuang, Y. Zhang. J. Deng, and F. Ren, "R3dg: Retrieve, rank and reconstruction with different granularities for multimodal sentiment analysis." Research, 2025.
[9] Y. Zhuang, W. Bai, Y. Zhang, J. Deng, Z. Hu, X. Zhang, and F. Ren, "Multi-level contrastive learning for multimodal sentiment analysis," IEEE Transactions on Multimedia, 2025.
[10] A. Liu, Z. Tan, J. Wan, Y. Liang, Z. Lei, G. Guo, and S. Li, "Face anti-spoofing via adversarial cross-modality translation," " IEEE Transactions on Information Forensics and Security, vol. 16, pp. 1775922772 2021.
[11] S. Wei, C. Luo, and Y. Luo, "Mmanet: Margin-awand aware distillation and modality regular- ization for incomplete multimodal learning," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023, pp. 20 049-
[12] M. Jaiswal and E. M. Provost, "Privacy enhanced multimodal neural representations for emotion recognition," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 05, 2020, pp. 7985-7993.
[13] Z. Lian, L. Chen, L. Sun, B. Liu, and J. Tao, "Genet: Graph completion network for incomplete multimodal learning conversation." IEEE Transactions on pattern analysis and machine intelligence, vol. 45, no. 7, pp. 8419-8432, 2023.
[14] H. Zhang, Wang, and Yu, "Towards robust multimodal sentiment analysis with incomplete data," in The Thihth- Annual Conference on Neural Information Processing Systems, 2024.
[15] Y. Wang, Y. Li, and 7. Cui, "Incomplete nultimodality-difffuse emotion recognition," Advances in Neural Information Processing Systems, vol. 36, 2023.

---

[16] M. Li, D. Yang, X. Zhao. S. Wang, Y. Wang, K. Yang, M. Sun, D. Kou, Z. Qian, and L. Zhang, perrelation-dccouplee knowledge distillation for multimodal sentiment analysis with incom- plete modalities," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 12 12458-12468
[17] M. Li, D. Yang, Y. Lei, S. Wang. S. Wang, L. Su, K. Yang, Y. Wang. M. Sun, and L. Zhang, "A unified self-distillation framework for multimodal sentiment analysis with uncertain missing modalities," " in Proceedings of the AAAI Conference Artificial Intelligence, vol. 38, 2024, pp. 10074-10002.
[18] M. Li, D. Yang. Y. Liu, S. Wang, Chen, S. Wang. J. Wei, Y. Jiang, Q. Xu, X. Hou et al., "To- ward robust incomplete multimodal sentiment analysis via hierarchical representation learning." The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[19] A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy, "Deep variational information bottleneck." in International Conference on Learning Representations, 2022.
[20] Yang. Y. Yu, D. Niu, W. Guo, and Y. Xu, "Confede: Contrastive feature decomposition for multimodal sentiment analysis." " in Proceedings of the 6lst Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 7617-7630.
[21] H. Zhang, Y. Wang, G. Yin, K. Liu, Y. Liu, and T. Yu, "Learning tiguage--guided adaptive hyper-modality representation for multimodal sentiment analysis," in Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, 2023, pp. 756-767.
[22] Y.-H. H. Tsai, S. Bai, P. P. Liang, J. 7. Kolter, L..P. Morency, and R. Salakhutdinov, "Multimodal transformer for unaligned multimodal language sequences," in Proceedings of the conference. Association for computational linguistics. Meeting, vol. 2019. NIH Public Access, 2019,
6558.
[23] W. Rahman, M. K. Hasan, S. Lee, A. Zadeh, C. Mao, L.-P. Morency, and E. Hoque, "Integrating multimodal information in large pretrained transformers," in Proceedings of the conference. Association for Computational Linguistics. Meeting, vol. 2020. NIH Public Access, 2020, p. 2359.
[24] Y. Wang, Z. Cui, and Y. Li, "Distribution-coniisten modal recovering for incomplete multimodal learning." " in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023,
pp. 22 025-22 034.
[25] H. Pham, P. P. Liang, T. Manzini, L.-P. Morency, and B. Póczos, "Found in translation: Learning robust joint representations by cyclic translations between modalities,"" " in Proceedings of the AAAI conference on artificial intelligence, vol. 33, no. 01, 2019, pp. 6892-6899.
[26] J. Zhao, R. Li, and Q. Jin, "Missing modality imagination network for emotion recognition with uncertain missing modalities," in Proceedings the 59th Annual Meeting of the Association for Computational Linguistics and the I 1tth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, pp. 2608 2618.
[27] G. Andrew, R. Arora, J. Bilmes, and K. Livescu, "Deep canonical correlation analysis." in International conference on machine learning. PMLR, 2013, pp. 1247-1255.
[28] W. Wang, R. Arora, K. Livescu, and J. Bilmes, "On deep niet representation learning,"
in International conference on machine learning. PMLR, 2015, pp. 1083-1092.
[29] Z. Guo, T. Jin, and Z. Zhao, "Multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition," in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024. pp. 1726-1736.
[30] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, F.. Kaiser, and Polosukhin, "Attention all you need," Advances in neural information processing systems, vol. 30, 2017.
12

---

[3] W. Yu, H. Xu, Z. Yuan, and J. Wu, "Learning modality-speciff representations with self- supervised multi-task learning for multimodal sentiment analysis." " in Proceedings of the AAAI conference on artificial intelligence, vol. 35, no. 12, 2021, pp. 10790-10797.
[32] A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira, "Perceiver: General perception with iterative attention." in International conference on machine learning. PMLR, 2021, pp. 4651-4664
[33] N. Tishby, F. C. Pereira, and W. Bialek, "The information bottleneck method," arXiv preprint physics/0000057. 2000.
[34] X. Xiao, G. Liu, G. Gupta, D. Cao, S. Li, Y. Li, T. Fang, M. Cheng, and P. Bogdan, "Neuro- inspired information-theorrtie hierarchical perception for multimodal learning," in The Twelfth International Conference on Learning Representations, 2024.
[35] S. Mai, Y. Zeng, and H. Hu, "Multimodal information bottleneck: Learning minimal sufficient unimodal and multimodal representations"" IEEE Transactions on Multimedia, vol. 25, pp. 4121-4134, 2022.
[36] Z. Gao, X. Jiang. X. Xu. F. Shen, Y. Li, and H. I. Shen, "Embracing unimodal alcatoric uncertainty for robust multimodal fusion," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 876-268885
[37] D. P. Kingma, "Auto-encoding variational bayes." arXiv preprint arXiv: 2013.
[38] A. Zadeh, R. Zellers, Pincus, and L.-P. Morency, "Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos," arXiv preprint arXiv: 06259, 2016.
[39] A. B. Zadeh, P. P. Liang. S. Poria, E. Cambria, and L.-P. Morency, "Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph." in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), 2018, pp.
[40] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, "lemocap: Interactive emotional dyadic motion capture database," Language resources and evaluation, vol. 42, pp. 335-359, 2008
[41] D. M.-W. C. Kenton and L. K. Toutanova, "Bert: Pre-training of deep bidirectional trans- formers for language understanding," in Proceedings of naacL-HLL, vol. 1. Minncapolis, Minnesota, 2019, p.
[42] K. Zhang, 7. Zhang, Li, and Y. Qiao, "Joint face detection and alignment using multitask cascaded convolutional networks," IEEE signal processing letters, vol. 23, no. 10, pp. 1499- 1503, 2016.
[43] T. Baltrušaitis, P. Robinson, and L.-P. Morency, "Openface: an open source facial behavior analysis toolkit," in 2016 IEEE winter conference applications of computer vision (WACV). IEEE, 2016, pp. 1-10.
[44] G. Degottex, J. Kane, T. Drugman, T. Raitio, and S. Scherer, "Covarep-a collaborative voice analysis repository for speech technologies," in 2014 ieee international conference on acoustics, speech and signal processing (icassp). IEEE, 2014, pp. 960-964.
[45] M. K. Hasan, W. Rahman, A. B. Zadeh, J. Zhong, M. I. Tanveer, L.-P. Morency, and M. E. Hoque "Ur-funny: A multimodal language dataset for understanding humor," in Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPP-JCNiPP) 2019, pp- 2056-2056.
[46] S. Castro, D. Hazarika, V. Pésaz--Rsas, R. Zimmermann, R. Mihalcea, and S. Poria, "Towards multimodal sarcasm detection (an _ovviously__..... perfect paper)," in Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, 2019, pp. 4619-4629,
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer the "Abstract" and "1 Introduction" sections for our paper's contributions and scopes.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to the "5 Conclusion and Discussion" section for the limitations of our work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiaatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer:
[NA]
Justification: No theory assumptions and proofs are provided the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The "4.1 Datasets and Implementation Details" and "A.2 Additional Imple- mentation Details" sections of the paper describes all the information needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
The paper does not provide open access to the data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ mripss.cc/ubbliccguu.eee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The "4.1 Datasets and Implementation Details" and " "A. Additional Imple- mentation Details" sections of the paper specify all the training and testing details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Tables 3, 8 and 9 of the paper, we conducted significance tests on the experimental results to demonstrate the superior performance of the proposed framework.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еттоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The "A.2 Additional Implementation Details" section of the paper explains that all experiments are conducted on an NVIDIA GTX 3090 GPU. Besides, the "A.3.10 Computation Overhead and Trade-off Analysis" section of the paper explains that execution time and peak GPU memory usage of the proposed method.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurips. http://///eerppppcpcpppbttttttit itc
Answer: [Yes]
Justification: Our research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdictio)..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to the "5 Conclusion and Discussion" section for the societal impacts of work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The MOSI, MOSEL, IEMOCAP, UR-FUNNY and MUStARD datasets in this paper are existing assets and we cite the references.
Guidelines:
The answer NA means that the paper does
existing assets. The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
20

---

A Technical Appendices and Supplementary Material
A.1 Additional HME Details
In this section, we provide further details of the HME framework. After the sample selection step in [q[] the extracted features Hº are passed into a Perceiver network [32] to generate enhanced representations. The Perceiver refines the features through combination of cross-modal and self- attention mechanisms 30), enabling richer information exchange across modalities. The Perceiver operates by introducing a set of learnable latent vectors, denoted as E E I which interact with the input features H Each Perceiver layer alternates between two key components: cross-attention block and self-attention block [32.. In the cross-attention block, the latent vectors E,, act as the queries, while the input features H serve as the keys and values. This process allows the latent vectors to selectively attend to informative aspects of H formulated as:
Em Wo. WT (HM) )HH Wy.:
Hea ATTN amm (E H Softmax
(10)
Vd
where Wo., hh Wv, E Rddd are trainable weights, and Hee E Rlaxd The resulting features are then normalized and refined through a residual connection:
Hea LN Hea) + E
(11)
followed by feed-forward update:
Hイf (W + Hen,
(12)
where W is trainable. The subsequent self-attention block applies the same mechanism but sets the queries, keys, and values all to the same input (e.g., H,y). By stacking multiple layers of these alternating blocks, the Perceiver gradually produces a enhanced representation E, Finally, the enhanced modality representation Em IRd is obtained by averaging the outputs the last transformer layer.
A.2 Additional Implementation Details
In this section, we describe the datasets, and implementation details used in experiments.
MUStARD and UR-FUNNY datasets. We conduct additional experiments on the UR-FUNNY 45 and MUStARD (46) datasets, both designed for humor detection from multimodal data. The UR-FUNNY dataset contains 1,866 videos from 1,741 speakers, comprising 9,588 utterances. The data are divided into 7,614 training, 980 validation, and 994 test instances. The MUStARD dataset includes 690 videos, split into 539 training, 68 validation, and 68 test utterances.
Implementation Details. To ensure reliable training and prevent overfitting underfitting, we train 100 epochs and apply early stopping with a patience of 10 epochs across all reproduced baseline models and HME. Specifically, training is terminated when the validation loss fails to improve for 10 consecutive epochs. Besides, for both missing modality protocols, the reported results for HME and re- re-implemented baselines (denoted with symbol 1) are the averages from five separate runs. All models are evaluated on an NVIDIA GTX 3090 GPU.
For HME under the two missing protocols across three datasets, several hyper-parameters remain consistent, such as the Perceiver layers the modality enhancement module, which are set to layers, the prompt length 4p = 5. the similarity threshold = 0.6, and the bounds of uncertainty weights t/ = 0.2, tu = 1.0. However, some hyper-parameterr vary depending on the specific dataset and configuration. For the MOSI and MOSEI datasets, the default hyper-parametert under both settings are as follows: learning rate of 4c-5, batch size of 256, and VIB loss weight a = 1.0. The hidden dimension d for MOSI and MOSEI also differ across protocols. For MOSI, the fixed missing protocol uses a hidden dimension of 96, while the random missing protocol uses 192. In the case of MOSEI, the hidden dimension is set to 192 for both protocols. In contrast, for the IEMOCAP dataset, the default hyper-paramette are consistent across both missing protocols, with a learning rate of le-4, batch size of 24, VIB loss weight
0.5, and hidden dimension d 30.
To identify the optimal hyper-parameters of HME, we performed a search over a limited set of values. For the MOSI and MOSEI datasets, the learning rate was selected from 1c-5, 2c-5, 3c-5, 4c-5), the
21

---

Table 4: Performance comparison under fixed missing protocol MOSEI.
Datasets
Avail
GCNet
MPLMM
DiCMoR
IMDer
LNLN
HME
L.
83.0/83.2
84.6/84.6
84.2/84.3
84.5/84.5
84.1/84.4
85.5/85,4
V
61.9/61.6
62.2/53.2
63.6/63.6
63.9/63.6
62.9/77.2
64.6/62.9
A
60.2/60.3
61.8/49.4
62.9/60.4
63.8/60.6
62.9/77.2
64.1/59.7
V
84.3/84.4
85.2/85.1
84.9/84.9
85.0/85.0
84.4/84.7
85.7/85.7
MOSEI
L,A
84.3/84.4
85.1/85.1
85.0/84.9
85.1/85.1
84.1/84.4
85.6/85.4
A,
64.1/57.2
63.2/51.0
65.2/64.4
64.9/63.5
62.9/77.2
65.4/63.7
L.A.V V
85.2/85.1
85.3/85.3
85.1/85.1
85.1/85.1
84.1/84.5
86.2/86.2
Avg.
74.7/73.7
75.3/70.5
75.8/75.4
76.0/75.3
75.1/81.4
76.7/75.6
Table 5: Performance comparison under random missing protocol on MOSEI.
Datasets
MR
GCNet
MPLMM
DiCMoR
IMDer
LNLN
HME
0.0
85.2/85.1
85.3/85.3
85.1/85.1
85.1/85.1
85.1/85.1
86.2/86.2
0.1
82.3/82.1
83.7/83.5
83.2/83.1
83.5/83.3
83.1/83.0
84.0/83.8
0.2
80.3/79.9
81.4/80.8
81.2/80.8
81.3/80.9
81.7/81.0
82.3/81.9
0.3
77.5/76.8
79.2/78.3
79.0/77.9
79.4/78.8
79.1/78.1
80.0/79.6
MOSEI
0.4
76.0/74.9
77.1/6..1
76.9/75.2
76.1/74.3
76.9/75.6
77.6/76.4
0.5
74.9/73.2
75.6/73.8
73.7/71.7
75.3/72.4
75.1/72.4
75.7/74.7
0.6
74.1/72.1
72.7/70.6
71.1/70.4
71.2/66.5
72.5/68.6
73.0/72.5
0.7
73.2/70.4
70.5/69.2
70.6/68.9
71.1/65.9
70.6/65.2
72.4/69.2
Avg.
77.9/76.8
78.2/77.2
77.6/76.6
77.9/75.9
78.0/76.1
78.9/78.0
VIB loss weight a from (0.5. 0.6, 0.7, 0.8, 0.9, 1.0), and the hidden dimension d from (92, 128, 160, 192, 256). For the IEMOCAP dataset, the learning rate was chosen from (le-4, 5e-4, le-3, 2e-3). VIB loss weight a from (0.5, 0.6, 0.7, 0.8, 0.9, 1.0), and the hidden dimension from (30, 35, 40, 45, 50, 55).
To identify the hyper-parameter of the re-implemented baselines, we adhered to the hyper-parameter guidelines provided in the official implementations of these models. For models like IMDer (15) and DiCMoR (24). which included pre-trained weights, used the publicly available checkpoints and configurations. For baselines without available syper-parameterr for specific datasets, we aligned their pareeepprr with HME's settings where possible. When unique parameters were required, we referred to their recommended settings from other publicly available implementations.
Since MOSI and MOSEI regression tasks, consistent with previous work [29]. convert the ground truth and predictions into two categories: greater than and less than 0. This enables the calculation of binary classification metrics such accuracy and F1-score.
A.3 Additional Results
Here we present additional experimental results in this section. These include the performance of different models MOSEI dataset (in A.3.1), the ablation studies for each component under various missing modality scenarios across three datasets (in A.3.2), model stability analysis (in A.3.3), eтror analysis (in A.3.4), generalization and plug-and-play applicability analysis (in A.3.5), VIB components analysis (in A.3.6), enhanced representations analysis (in A.3.7), hyper-parameter analysis(in A.3.8), case visualization and representation visualization (in A.3.9), and computation overhead and trade-off analysis (in A.3.10).
A.3.1 Results MOSEI Dataset
In this section, we compare the performance HME with other baselines on the MOSEI dataset under two missing protocols. The results are shown in Tables and 5 Under the fixed missing protocol, HME performs slightly worse than LNLN the missing language modality scenario in terms of F1 score, but it delivers strong results across other scenarios for both metrics. In the random missing protocol, HME performs slightly worse than GCNet at MR values of 0.6 and 0.7, but it still outperforms the current ofate-of-the-art models all other missing rates. On average, HME improves
22

---

Table 6: Ablation results of fixed missing protocol three datasets.
Methods
L
A
L,V
L.A
A.V
L.A.V
Ablations on MOSI
HME
85.7/85.6
63.6/63.4
62.7/62.2
85.8/85.8
85.8/85.7
64.6/64.5
86.4/86.3
Er.
85.4/85.0
61.7/60.3
61.1/60.2
85.7/85.4
84.6/84.2
62.5/62.6
85.4/85.2
w/o Ev
85.1/84.8
61.3/61.2
62.0/62.1
84.9/84.6
85.7/85.4
62.5/62.3
85.2/85.2
w/o EA
84.9/84.9
60.5/58.1
62.3/61.0
85.2/85.1
85.5/85.3
63.0/61.8
85.8/85.7
w/o Es 85,4/85.0
61.7/60.3
61.1/60.2
85.7/85.4
84.6/84.2
62.5/62.6
85.5/85.4
w/o Un
85.2/85.0
61.0/60.8
61.1/60.7
85.7/85.6
85.7/85.4
62.7/61.9
86.1/86.0
w/o HMG
85.1/84.9
60.5/60.4
60.2/59.1
85.5/85.4
85.4/85.3
63.3/61.5
85.2/85.1
w/o VIB
84.8/84.6
61.1/61.3
61.9/616
85.7/85.6
85.2/85.0
62.7/62.5
85.4/85.2
Ablations on MOSEI
HME
85,5/85.4
64.6/62.9
64.1/59.7
85.7/85.7
85.6/85.4
65.4/63.7
86.2/86.2
w/o Ez 85.2/85.0
63.8/60.3
59.4/57.7
85.5/85.4
85.2/85.1
64.2/59.8
86. 1/86.0
w/o Ey 85.1/85.1
63.4/58.6
61.6/52,1
85.6/85.3
85.2/85.1
64.2/59.8
86. 1/86.0
w/o EA 85.2/85.0
63.8/60.3
60.2/53.1
85.4/85.4
85.2/85.1
64.2/59.8
85.9/85.8
w/o Es 85,2/85.1
63.1/59.9
60.6/56.8
85,5/85.3
85.3/85.2
64.0/59.6
85.8/85.7
w/o Um
85.1/85.0
63.6/55.5
61.5/55.5
85.4/85.3
85.1/85.1
64.1/57.2
16.6/86.0
w/o HMG
85.3/85.2
62.8/55.5
61.0/51.9
85.5/85.3
85.1/85.0
63.5/53.1
85.6/85.6
w/o VIB
85.4/85.3
64.5/60.3
61.8/66.8
85.6/85.5
85.2/85.1
65.3/62.4
85.9/85.8
Ablations on IEMOCAP
HME
79.1/78.1
75.7/70.8
78.9/76.8
79.7/78.9
80.7/80.0
78.1/76.3
81.1/80.6
w/o E,. 78.8/77.6
74.3/68.0
78.0/75.5
79.5/78.3
80.4/79.4
77.2/75.0
80.8/80.0
Ey 78.2/77.4
74.1/70.0
78.1/75.7
79.1/78.0
80.5/79.4
75.8/72.5
80.9/80.2
w/o EA 78.7/77.9
74.5/69.2
78.6/75.9
79.2/78.3
80.4/79.3
77.5/74.5
80.9/80.4
w/o Es 78.2/77.4
74.0/69.0
78.8/75.6
78.9/78.1
80.2/79.2
77.1/74.3
80.0/79.4
w/o U.
78.2/77.5
73.9/69.0
78.3/75.4
78.4/77.9
80.3/79.5
76.7/74.4
81.0/80.2
w/o HMG
78.8/77.7
74.9/70.0
78.4/76.3
78.7/77.8
80.1/79.4
77.8/75.9
80.8/80.2
w/o VIB 78.9/77.7
73.9/69.9
78.4/75.9
79.1/78.1
80.3/79.6
76.7/74.1
80.6/79.7
the metrics by 0.7 and 0.8 across all eight MR values. These findings indicate that HME offers superior overall performance and greater robustness compared to existing state-of- sate-of-the-ass aff-the-rrt models.
A.3.2 Detailed Ablation Results
In this section, we present detailed ablation results for both missing modality protocols, shown in Tables[[aandl Due to space constraints in the main text, we add ablation experiments for the VIB component here, labeled as 'w/o VIB'. To maintain the integrity of the HMG component, the VIB ablation was performed by removing the LYIB term in Lall during training.
Removing any component from HME consistently results in a performance drop across all missing modality scenarios. Specifically, removing the VIB leads to redundancy and interference from noisy information, reducing the model's effectiveness. Eliminating the enhanced hyper-modality representation Em, results in a decline, likely because it removes sentiment-relevant information that could be provided by other samples for the current sample. Removing the Un. component can result in lower-quality representations that are given more weight during the fusion process, especially in modalities with weaker representation capabilities, leading decrease in performance during fusion. Finally, removing the entire HMG module limits the model to using only individual modality representations from other samples, disregarding the interactions between these modalities, which are crucial for capturing sentiment-relevant information. These results highlight the critical importance and effectiveness of each component within the HME framework.
A.3.3 Model Stability Analysis
Since modality missing occurs randomly, we conduct five separate runs to comprehensively evaluate the performance HME and report the averaged results. To further assess the statistical significance of HME, we present the 95% confidence intervals for ACC and Fl scores under the random missing modality protocol on all three datasets, as shown Table Even the worst-case scenario, HME consistently shows competitive performance, demonstrating its strong performance.
23

---

Table 7: Ablation results of random missing protocol three datasets.
Methods
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Ablations on MOSI
HME
84.9/84.7
82.9/82.9
81.1/81.0
79.9/80.0
76.4/76.4
74.5/74.4
73.5/71.7
Er.
83.4/83.2
81.6/81.5
80.2/79.9
77.6/76.7
74.1/73.8
73.9/72.5
70.0/70.0
w/o Ev 84.2/83.9
81.9/81.5
79.7/79.0
77.3/77.0
74.4/73.9
73.2/72.3
70.7/69.3
w/o EA 83.2/82.9
81.6/81.3
79.3/78.8
77.1/7.6
74.4/74.0
71.2/71.2
70.4/68.7
w/o Es 84.2/83.8
81.1/80.9
80.0/79.8
76.7/76.6
75.2/74.9
72.3/72.4
70.6/69.6
w/o Um
83.7/83.4
82.2/82.2
79.9/79.6
78.5/78.4
75.2/74.8
72.4/71.5
71.3/70.7
w/o HMG
83.7/83.7
80.8/80.8
80.0/79.3
75.3/75.2
74.1/74.1
72.4/71.2
69.8/69.2
w/o VIB
83.1/82.8
81.1/81.1
78.8/78.8
78.2/77.6
75.2/74.8
72,6/72.1
71.0/69.5
Ablations on MOSEI
HME
84.0/83.8
82.3/81.9
80.0/79.6
77.6/76.4
75.7/74.7
73.0/72.5
72.4/69.2
w/o Ez 83.8/83.6
81.8/81.1
79.7/78.8
77.175.99
75.2/73.9
72.5/71.2
71.0/66.4
w/o Ey 83.7/83.4
81.6/81.0
79.0/78.6
76.8/75.6
74.8/73.5
71.8/70.9
70.9/65.8
w/o EA 83,5/83.2
81.7/81.1
79.3/78.6
77.175.2
74.8/73.1
72.6/69.5
70.8/65.8
w/o Es 83.7/83.4
81.7/81.2
79.2/78.0
77.1/7.33
75.3/73.6
72.3/71.0
71.5/68.7
w/o Um
83.2/83.0
81.5/80.7
79.5/78.0
76.6/75.0
74.5/71.6
72.5/70.1
70.8/65.6
w/o HMG
83.5/83.1
81.6/80.8
79.3/78.0
76.7/75.6
74.1/73.2
72.1/69.7
70.6/67.8
w/o VIB
83.7/83.3
81.7/81.2
79.1/78.6
76.7/76.0
75.2/73.9
72.7/710
71.0/67.7
Ablations on IEMOCAP
HME
80.9/80.3
80.7/80.0
79.0/77.8
78,7/77.3
78.5/77.1
77.8/76.1
77.6/76.0
w/o E, 80.3/79.4
79.8/78.9
78.8/77.5
78.5/76.8
78.0/76.4
76.7/74.6
76.5/74.1
w/o Ey 79.7/78.9
79.6/78.2
78.8/77.6
78.5/77.0
78.2/76.2
77.4/5.22
76.8/74.0
w/o EA 80.0/79.0
79.7/78.5
78.9/76.9
78.6/76.5
77.6/75.8
76.8/74.4
76.7/74.2
w/o Es 79.7/79.0
79.6/78.4
78.9/77.6
78.3/76.7
78.0/76.5
77.6/75.3
76.9/74.6
w/o U. 80.0/79.1
79.6/78.1
78.6/77.3
78.2/76.5
77.3/75.3
77.2/74.5
76.7/73.8
w/o HMG
80.3/79.6
80.0/79.0
78.9/77.3
78.5/77.0
78.2/76.4
77.715.77
77.3/75.4
VIB
80.0/79.3
79.9/79.1
78.8/77.5
78.3/76.7
78.3/76.1
77.1/74.8
77.0/74.6
Table 8: Performance under random missing protocol on MOSI (95% confidence intervals).
Datasets
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Statistical significance of ACC
MOSI
84,910.2
82.910.2
81.110.2
79.9
76.4110.
74.5110.6
73.51009
MOSEI
84.0.0.2
82.3.1..3
80.0.0.3
77.6110.2
75.7100.
73.0
72.410.5
JEMOCAP
80.9=0.2
80.7++0.3
79.0+10.4
78.7++0.3
78.5+0.3
77.8+0.2
77.6+0.1
Statistical significance of
MOSI 84,7=0.2
82.910.4
81.010.3
80.0
76.4100.4
74.410.6
71.710.5
MOSEI 83.8
81.911..
79.611..6
76.4=10.1
74.710.3
72.510.6
69.210.4
IEMOCAP 80.310.2
80.0.1..3
77.810.4
77.310.1
77.110.3
76.110.4
76.010.3
To better evaluate HME in real-world scenarios, we conduct additional experiments. (i) Since real-world modality missing varies, we test HME using 10 different random seeds to simulate diverse missing modality conditions. (ii) Recognizing that the number of available samples in practical applications can vary, which affects the generation of myder-modality representations, we test HME with 10 different batch sizes. The results of these experiments on the MOSI dataset, presented in Table 9 show that HME maintains strong stability and consistently delivers robust performance.
A.3.4 Error Analysis
There may be failure cases where samples within a batch are semantically similar but express different sentiments. When these samples are used to enhance each other, they can introduce conflicting signals, problem we call negative enhancement. To better understand the effect of negative enhancement, we conducted experiments on the MOST dataset, focusing on cases where the cosine similarity between selected representations was greater than 0.9, to examine whether these representations aligned more closely with the ground-truth labels with the model's predictions, and to identify the factors contributing to missclassification.

---

Table 9: Testing stability under random missing protocol on MOSI (95% confidence intervals).
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Statistical significance with 10 different random seeds
ACC
83.9+0.4
82.3+00.4
80.9+0.4
78.8+0.8
76.4
73.6+11.2
1.2
F1
83.6:1..4
82.2:10.4
80.710.4
78.910.8
76.1
73.1
71.5+0.4
Statistical significance with 10 different batch sizes
ACC
84,3+0.4
82.
81.0+0.4
78.6#0.7
76.4
74.0=0.9 73,6=1.4
70.1
F1
84.04:0.4
82.2:+0.4
80.7
78.7+0.7 76.1+0..4
71.7+0.4
Table 10: Alignment analysis high-similarity samples on MOSI,
Align with Ground Truth
Otherwise 12.92% (838/6485) 73.74% (716/971)
Correct Predictions
87.08% (5647/6485) 26.26% (255/971)
Wrong Predictions
The results, summarized in Table TO, reveal two key findings. (i) The selected representations align more with the model's predictions than with the ground truth. Specifically, there are 6,363 cases where they match the predictions, compared to 5,902 cases where they match the ground truth. This suggests that semantically similar tones may bias the model toward incorrect sentiment predictions. (ii) In misclassified instances, the selected representations tend reinforce the model's s incorrect predictions rather than the true labels (716 vs. 255 cases). This highlights the influence of negative enhancement in amplifying model bias during classification.
To address this, HME applies two strategies. (i) During sample selection, similarity threshold 4, is introduced: only samples with feature similarity above this threshold are considered valid neighbors. If the similarity falls below the threshold, the model either uses the average batch representation or discards the neighbor entirely. This reduces the risk of injecting misleading information. (ii) The VIB module helps suppress residual noise. By enforcing compressed and task-relevant representations, VIB filters out inconsistencies and ensures that the fused features remain robust. Together, these strategies lead to consistent improvements across datasets and missing- moddiityy scenarios.
A.3.5 Generalization and Plug-and-Play Applicability Analysis
To evaluate the plug-and-play applicability of the proposed HME framework, we integrated the Hyper- Modality Representation Generation (HMG) module into the MPLMM architecture and tested it under random missing-modality conditions on the MOSI dataset. As shown in Table incorporating HMG consistently improved performance, yielding average gains of +1.2 in accuracy and +1.8 in F1 score across different missing rates. These improvements highlight the plug-and-play compatibility of HME with existing models.
We further explored the generalization capability of HME from two perspectives: Syper-paramedd sensitivity and cross-dataset transferability. First, we assessed the generalization performance by directly applying the ayper-parameters trained and optimized on MOSI dataset, two external datasets, UR-FUNNY [45 and MUStARD [46. As reported in Table 12 HME again outperformed the MPLMM baseline across multiple missing-modality configurations. This consistent improvement suggests that the param-eas optimized on MOSI transfer well to other datasets, confirming the robustness and scalability of the HME.
Next, to test cross-dataset generalization ability, we trained and validated HME on one dataset and evaluated it on another using identical configurations. The results, summarized in Table 13 show a noticeable performance drop when transferring from MUStARD to UR-FUNNY, with average accuracy decreasing from 68.7 to 51.3 and Fl score from 69.9 to 55.5. In contrast, the reverse transfer showed a smaller degradation, with accuracy decreasing from 64.6 to 61.3 and F1 from 65.8 to 64.0. This can be attributed to inherent dataset differences and size disparities. MUStARD and UR- FUNNY differ substantially in modality composition and data distribution. For example, accuracy performance with all modalities ('L,A, V') present drops notably between the two datasets (75.0 vs. 49.7 and 72.6 v.s. 63.2). The presence of missing modalities further amplifies this domain gap. Moreover, the smaller size of MUStARD (539/68/68 for train/validationnness) provides limited variability compared to the much larger UR-FUNNY dataset (76 4/980/9944 making transfer from
25

---

Table 11: Ablation experiments of MPLMM on MOSI with random missing protocol.
Methods
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Avg.
Accuracy Performance
MPLMM
81.4
78.7
74.7
70.3
67.1
62.5
61.0
70.8
HMG
81.9
78.5
75.0
72.1
69.2
65.2
61.9
72.0
A
+0.5
-0.2
+0.3
+1.8
+2.1
+2.7
+0.9
+1.2
F1 Performance
MPLMM
81.4
78.7
74.8
69.9
66.3
60.5
58.3
70.0
HMG
81.9
78.6
75.1
72.3
69.3
64.5
61.1
71.8
A
+0.5
-0.1
+0.3
+2.4
+3.0
+4.0
+2.8
+1.8
Table 12: Performance UR-FUNNY and MUStARD datasets with fixed missing protocol.
Methods
A
L,A
I.V A.V
L,A.V
Avg.
Accuracy performance
UR-FUNNY
MPLMM
67.6
61.8
61.8
67.6
67.6
61.8
67.6
65.1
HME
70.6
63.2
61.7
75.0
72.0
63.2
75.0
68.7
Fi performance on UR-FUNNY
MPLMM
68.1
61.8
61.8
68.1
68.1
61.8
68.1
65.4
HME
70.6
63.9
67.2
75.0
73.1
64.6
75.0
69.9
Accuracy performance
MUStARD
MPLMM HME
71.7
52.3
53.0
72.4
71.7
54.8
72.4
64.0
72.9
53.3
52.5
72.8
72.9
55.5
72.6
64.6
F1 performance on MUStARD)
MPLMM
71.8
54.3
54.9
72.5
71.8
55.2
72.5
64.7
HME
73.0
56.6
56.8
73.0
73.0
55.8
72.7
65.8
MUStARD UR-FUNNY more challenging. Conversely, training UR-FUNNY benefits from greater data diversity, leading to a smaller performance decline when transferring to MUStARD.
A.3.6 VIB Components Analysis
To mitigate misleading semantic signals that may arise during cross-sample enhancement, HME includes a VIB module. As shown in Table TOJ the retrieved similar samples occasionally contain sentieent-related cues that contradict the sentiment label of the target instance, thereby introducing noise and degrading model performance. The VIB module addresses this issue by compressing latent representations and retaining only task-relevant information. The ablation results presented in Tables
Input
Selected Samples For Enhancement
WHO SEEMED TO HAVE GENUINE AFFECTION FOR OTHER HUMAN BEINGS
BECAUSE I TRULY LOVE AN ACTION FLICK ACTION COMEDY FLICK EVEN BETTER RIGHT
Languagel
Video
Audio
Figure 6: Case visualization of selected enhancement samples for each modality on the MOSI dataset. The similarity threshold is set 0.9 and the batch size Green boxes indicate representations with the same label, while red boxes denote samples with different labels.
26

---

Table 13: Generalization performance between MUStARD and UR-FUNNY.
Methods L
A
V
L,A L,V A.V LAA,V
Avg.
Accuracy Performance from MUSIARD to UR-FUNNY
HME
50.8
50.3
50.9
54.4
51.5 51.6 49.7
51.3
FI Performance from MUSIARD UR-FUNNY
HME
52.5
52.9
52.5
56.2
55.5 53.6
65.4
55.5
Accuracy Performance from UR-FUNNY to MUSIARD
HME
63.2 60.2
57.4
63.2 61.8 60.3 63.2
61.3
F Performance from UR-FUNNY MUSIARD
HME
66.3
62.9
60.2
66.3
64.7 60.4
67.1
64.0
Table 14: Performance comparison under different weights of VIB components.
I[2 X)
0.1
0.2
0.5
1.0
2.0
5.0
HME
78.0/75.9
77.4/75.3
76.8/74.9
76.5/73.7
76,0/72.8
75,5/71.3
I(Z,Y)
0.1
0.2
0.5
1.0
2.0
5.0
HME 77.3/74.9
77.5/75.4
77.6/76.0
77.5/75.1
77.3/74.9
77.1/74.5
VIB 0.1
0.2
0.5
1.0
2.0
5.0
HME
77.7/75.1
77.8/75.5
77.6/76.0
76.9/75.0
76.5/74.4
75.7/73.8
andd 7 inndicate that removing the VIB module consistently results in performance degradation across all datasets, underscoring its effectiveness in suppressing misleading or noisy cues.
The VIB objective comprises
components, X) and I(Z, Y). which quantify the amount of information retained from the input and the relevance of the latent representation to the target label, respectively. To assess the contribution of each component, we perform detailed analysis by varying their relative weights both independently and jointly. As reported in Table [4, increasing the weight of [(2, X). which enforces stronger compression, tends discard useful sentiment information and consequently reduces performance. In contrast, increasing the weight of L(Z, Y) initially improves accuracy by enhancing label alignment, but excessive emphasis on this term ultimately leads to performance deterioration, likely due to overfitting or the under-utilization of complementary multimodal cues.
A.3.7 Enhanced Representations Analysis
To examine the relationship between enhanced, raw, and salected-sample representations, we compute the mean squared error (MSE) between representations at different enhancement stages: modality enhancement (E.),, myper-modallit generation (Rm),, noise-reduced compression (F,.). and modality fusion (5,)). Each compared against the corresponding representation (H,,) and the average of selected samples (H²,,) Specifically, experiments are conducted samples containing all three modalities ('L,A.V') with similarity threshold of 0.9 on the MOSI training set. Table 15 reports the mean MSE across all modalities, and Table [6 focuses on the language modality.
The results show that enhanced representations (Em, Rm, F')) consistently closer H', than to Hm. with MSE differences within 0.2. This indicates that enhancement effectively incorporates information from related samples while preserving original modality representation. After fusion, Sn exhibits comparable MSE distances to both Hm and H' slightly favoring Hm This suggests that the incertaint--awat fusion mechanism prioritizes original representations while adaptively weighting enhanced features based on their variance.
Table 15: Mean MSE distance between different representations of three modalities on MOSI dataset.
Fm
R.
Fm.
S.
Hm
3.61
3.76
2.93
0.76
H'
3.47
3,63
2.79
0.77
27

---

Table 16: Mean MSE distance between language representations on MOSI dataset.
E1
Rg
S1
H
2.71
2.70
2.19
0.51
HL 2.53
2.52
1.99
0.53
76.0
76.0
76.00
75.8
75.75
75.6
F1-Score
75.6
75.4
75.2
75.4
75.00
75.0
75.2
74.75
74.8
74.50
74.6
0.1 0.2
0.5
1.0
12
15
20
25
Figure 7: Performance of different la. lu and Ip IEMOCAP dataset with MR 0.7.
Since the enhanced representations each stage obtained from the mean representation of the selected samples (H', E )R,), the attention weights of the learnable prompts cannot distinguish which selected sample contributes more. Therefore, omit the visualization and analysis of this part and leave it for future work.
A.3.8 Hyper-parameter Analysis
In this section, we present the analysis of srpeerpprameters, including the similarity threshold (t,), upper bound of uncertainty weights (fu), and the length of the prompts (2p). Figure shows the variation in F1 score for HME under the random missing modality protocol (MR R 0.7) on the IEMOCAP dataset, with different values of tg, and Ip.
Effects of different l.. As la increases, the score initially improves, then starts to decline. The worst performance occurs when i, 1.0, where HME relies solely on the average representation. At 1 0.1, the performance even better than at le 0.5. This suggests that when fewer modality representations available within the batch, lower values of t, allow for more diverse representations, even though they may have some error. The VIB module helps to extract useful information from these representations. As t, increases, the representations that meet the high similarity threshold become less frequent, reducing the amount of useful information, which leads to performance degradation.
Effects of different lu. As t, increases, the score shows similar trend of initially increasing, then decreasing. The highest performance is observed at L, 1.0. This improvement can be attributed to the fact that, at MR = 0.7 (where only one modality is present per sample), many of the original representations in the fusion process are likely to be missing and replaced with zero vectors. A Ly value 1.0 helps to compensate for these missing values, providing sufficient information.
Effects of different lp. As 4p increases, the Fl score again shows an initial improvement followed by decline. When ip is small, the model can only carry limited information, which restricts its performance. At lp = 5. the model contains the most complete set of modality information, yielding the best performance. However, lp increases further, the prompt may start include unnecessary or irrelevant information, which can negatively affect the model's 's performance.
A.3.9 Case Visualization and Representation Visualization
Here we visualize selected sample pairs from the MOSI dataset under different batch sizes. Specifi- cally, we examine an extra configuration with MR of 0.7, a similarity threshold of 0.9 and a batch size of (). From Figures 4 and [ we observe that larger batch sizes increase the likelihood that the selected samples share the same sentiment label as current input. When the batch size large, HME benefits from broader selection pool and tends to choose samples with labels consistent with the current input. contrast, when the batch size is small and some modalities are missing, HME relies the available representations for enhancement.
28

---

1.0
0.70
FL
Hu
0.65
FA
0.8
HA
0.60
Fv
Hv
0.6
0.55
SL
0.50
SA
0.4
SA
0.45
Sy
By
0.40
SH
0.2
SH
0.35
0.0
0.4 0.5 0.6 0.7 0.8 0.9 1.0
0.0
0.2
0.4
0.6
0.8 1.0
Figure 8: Visualization learned representations on the MOSI dataset with M R = 0.7.
Table 17: Comparison of parameter count and running time for different models on the MOSI dataset.
Models
Parameters
running time (1)
MPLMM
128,335,507
3,433s
DiCMoR
113,007,057
12,792s
IMDer
122,484.049
175,138s
HME 118,542,893
684s
Besides, to further understand the differences between the enhanced hyper-modality representations and the original representations, we visualize the additional representations using t-SNE: the original modality representation Hm, and the hyper-modality representation Fm and the fused representations Sn and Sh- Figure [8] shows these visualizations on the MOSI test set with an MR of 0.7. Our observations
follows: (i) After uncertainty-aware attention fusion, the information from Fm. is well integrated into the original representation. This results in non-overlapping clusters between Sm. Sn and Fm. (ii) The hyper-modality representation F., integrates effectively with the original modality representation Hm, as evidenced by overlapping information between the fused representations.
A.3.10 Computation Overhead and Trade-off Analysis
Computation overhead. The computational efficiency of HME with three state-of-the-art baselines compared the MOSI dataset. To ensure fairness, all models were trained for 100 epochs under identical settings on an NVIDIA GTX 3090 GPU. As shown in Table I7 HME completed training in just 684 seconds, which is only 20% of the runtime of MPLMM, 5% of DiCMoR, and less than 1% of IMDer. In terms of model size, HME has approximately 118M parameters, slightly more than DiCMoR but fewer than most other baselines. These results highlight the notable efficiency of HME in both training speed and parameter count.
Trade-off between performance and computational cost. Since HME's enhanced representations are highly sensitive to batch size, larger batches may improve performance but require greater computational resources. To examine this trade-off and explore the performance upper bound, we conducted additional experiments on MOSI under random missing protocol. Specifically, the average accuracy and Fl score with varying the batch size across seven missing rates is evaluated. As summarized Table 18] performance improves steadily as batch size increases, peaking at 256.
Table 18: Computational overhead of HME with different batch sizes on the MOSI dataset.
Batch Size
ACC/FI
Training Time
Peak GPU Memory
64
76.6/76.3
429s
5.6GB (5,635MB)
128
77.3/77.0
315s
8.3GB (8,279MB)
256
79.0/78.7
290s
13.2GB (13, 169MB)
288
77.8/77.4
280s
14,6GB (14,621MB)
320
78.1/77.6
280s
15.8GB (15,785MB)
352
77.9/77.2
2738
17.0GB (17,023MB)
29

---

Beyond this point, both accuracy and FI score show a slight decline. This suggests that while larger batches generally benefit learning, there is saturation point beyond which gains diminish and minor degradation may occur.
To provide more practical perspective, also measured training time (over 50 epochs) and peak GPU memory usage under different batch sizes in Table [8] The results reveal three trends: (i) training time decreases as batch size grows, which may due to fewer updates per epoch, (ii) GPU memory usage increases, and (iii) performance improves up to batch size 256, then gradually declines. Overall, batch size of 256 offers the best balance delivering peak performance with acceptable training time and manageable memory consumption.
30


Reviews:
[{"review_id": "mT6laE3ZFs", "reviewer": "Reviewer_Robh", "summary": "This paper introduces Hyper-Modality Enhancement (HME) framework for Multimodal Sentiment Analysis which is robust against missing modalities. The main contribution is a two-stage pipeline. First, the \"Hyper-Modality Representation Generation\" module enriches the representation of each available modality by borrowing semantically relevant information from other samples within the same training batch. This cross-sample enhancement reduces the model's reliance on having complete data for every sample. Second, an \"Uncertainty-Aware Fusion\" mechanism adaptively combines the original modality representations with these newly enhanced ones. It uses Variational Information Bottleneck (VIB), allowing the model to dynamically trust the enhanced features more or less depending on their estimated reliability.", "questions": "- The cross-sample enhancement is a good idea but is dependent on the quality of neighbors in the batch. Have you considered strategies to mitigate the risk of \"negative enhancement\" from misleadingly similar samples?\n- How does the selection of batch size impact the performance?\n- As mentioned in line 135-137, \"If no representation satisfies the threshold $t_s$, or if the modality is missing, we instead average all available representations of the current modality within the batch to capture the modality-specific information\" - What is the reason behind this assumption? If I understood it correctly, the samples are not very similar and thus they can be from different classes. How does averaging them provide an \"Enhanced\" representation?\n- How does the total number of parameters and number of learnable parameters compare with other methods?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- The proposed method looks technically sound and well-motivated. The architecture thoughtfully combines several concepts to enhance robustness to missing modalities.\n- The empirical evaluation is thorough and convincing.\n- The paper is well-written, structured, and easy to follow. \n\n**Weaknesses:**\n- The core enhancement mechanism relies on finding similar samples within a mini-batch. This makes the model's performance potentially sensitive to batch composition and size. \n- The pairwise similarity calculation within a batch has a computational complexity of $O(B^2)$, where $B$ is the batch size. While this is acceptable for the batch sizes used in the paper, it could become a bottleneck for training with very large batches, limiting scalability.\n- The methodology section is presented logically but seemed a little bit overloaded with symbols and equations inside paragraphs. Careful rewriting will enhance readability.", "was_revised": true, "final_justification": "I appreciate the authors’ detailed response and the clarifications provided, as well as the additional experimental data supporting their claims. However, I find that certain limitations and trade-offs between computational cost and performance remain and warrant more thorough analysis. After reviewing the explanation and conducting my own assessment, I have decided to maintain my current rating.", "rebuttal": "We sincerely thank Reviewer Robh for the time and effort dedicated to reviewing our paper and for the insightful comments and constructive suggestions. We are encouraged by the positive feedback, especially the recognition that our method is \"technically sound and well-motivated.\" Below, we address the concerns and questions raised in detail.\n\n### **Weaknesses & Questions:**\n\n**W1&Q2: Sensitivity to Batch Composition and Size**  \n\nWe appreciate your observation regarding the model’s sensitivity to batch composition and size. To analyze this, we evaluated the performance of HME under different batch sizes ({64, 128, 256}) on the MOSI dataset using the random missing modality protocol. The results in Table 1 and Table 2 below show that although performance slightly varies with batch size, HME consistently outperforms baseline methods. Specifically, as batch size increases, both accuracy and F1-score improve, suggesting that more diverse batches help identify better semantic neighbors.\n\nTable 1: Accuracy performance on MOSI with random missing protocol\n| batch size | 0.1  | 0.2  | 0.3  | 0.4  | 0.5  | 0.6  | 0.7  | avg  |\n| ---------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| 64         | 83.8 | 81.1 | 79.1 | 76.1 | 74.2 | 72.3 | 69.7 | 76.6 |\n| 128        | 83.8 | 81.4 | 79.9 | 76.7 | 76.5 | 72.4 | 70.3 | 77.3 |\n| 256        | 84.9 | 82.9 | 81.1 | 79.9 | 76.4 | 74.5 | 73.5 | 79.0 |\n\nTable 2: F1 performance on MOSI with random missing protocol\n| batch size | 0.1  | 0.2  | 0.3  | 0.4  | 0.5  | 0.6  | 0.7  | avg  |\n| ---------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| 64         | 83.8 | 81.2 | 78.6 | 76.1 | 73.9 | 71.2 | 69.2 | 76.3 |\n| 128        | 83.8 | 81.4 | 79.6 | 76.3 | 75.9 | 72.1 | 69.7 | 77.0 |\n| 256        | 84.7 | 82.9 | 81.0 | 80.0 | 76.4 | 74.4 | 71.7 | 78.7 |\n\nThese experiments will be added to the appendix to clarify this sensitivity.\n\n**W2: Scalability and Pairwise Similarity Overhead**  \n\nYou're absolutely right that computing pairwise similarity within the batch introduces a computational overhead, especially as batch size grows. While larger batches improve the chance of finding semantically similar representations, they also increase computation. We acknowledge this trade-off and plan to explore more scalable solutions in future work, such as approximate neighbor search or memory-efficient batch construction strategies.\n\n**W3: Overloaded Mathematical Presentation**  \n\nThank you for pointing out the readability issue in Sections 3.3 and 3.4. We will revise the presentation by simplifying the notation and separating key equations from the main text for better clarity. For instance, auxiliary variables such as $\\sigma + \\epsilon$ will be redefined or renamed to reduce reading load for readers.\n\n**Q1: Risk of Negative Enhancement**  \n\nWe greatly appreciate this insightful observation. Indeed, as you pointed out, semantically similar but sentimentally dissimilar samples within a batch can introduce conflicting signals—what is referred to as “negative enhancement.” In Section A.2.4 and Table 10 of the appendix, we conduct a detailed analysis of this issue. Empirically, we observed that around 12.9% of selected neighbors in a batch could have sentiment labels opposite to the target sample, which may lead to unintended distortion of the enhanced representation.\n\nTo mitigate this, (1) we implement **a similarity threshold** during the neighbor selection phase. Specifically, only samples whose feature similarity exceeds a threshold are considered valid neighbors. This helps **reduce the likelihood of injecting misleading information**. When the similarity falls below this threshold, the model either defaults to using the average batch representation or excludes the neighbor entirely. In addition, (2) the VIB module plays a crucial role in further **suppressing noise introduced by unreliable enhancements**. By promoting compressed and task-relevant representations, the VIB mechanism helps **filter out semantic inconsistencies** and ensures that the fused features remain robust. We have observed that incorporating this strategy, especially with the VIB component, **improves performance across datasets and missing-modality scenarios** (as shown in Tables 6 and 7 of the appendix). We will revise the main manuscript to explicitly describe this mechanism and include additional analysis to highlight its effectiveness.\n\nFurthermore, inspired by  frameworks such as MoCo, we are exploring the use of a memory bank to retrieve reliable neighbors from a broader pool of samples beyond a single batch. This approach could further reduce the risk of negative enhancement and is a promising direction for future work.\n\n**Q3: Averaging Modality Representations When No Good Matches Exist**  \n\nThank you for this excellent question. In scenarios where no high-similarity representations are found or when a modality is entirely missing, we fall back on averaging all available representations of that modality within the batch. While these samples may come from different classes, this fallback provides a modality-level common representation that helps stabilize performance and avoid drastic degradation. Furthermore, VIB plays a crucial role in reducing irrelevant or noisy information during fusion, helping minimize the negative impact of such averaging.\n\n**Q4: Parameter Count Comparison**  \n\nWe appreciate the suggestion to report model complexity. As shown in Table 3 below, we report the parameter count and runtime comparison on the MOSI dataset. HME has approximately 118M parameters—slightly more than DiCMoR (113M), but fewer than most other baselines. Importantly, HME is also efficient in runtime. HME requires only 684 seconds to train, significantly less than DiCMoR’s 12,792 seconds. This reflects the practical efficiency of our approach despite the enhanced representational capacity.\n\nTable 3:Parameter count comparison on MOSI dataset\n| Model   | Parameters      | Runtime (s) |\n| ------- | --------------- | ----------- |\n| MPLMM   | 128,335,507     | 3,433s      |\n| DiCMoR  | 113,007,057     | 12,792s     |\n| IMDER   | 122,484,049     | 175,138s    |\n| **HME** | **118,542,893** | **684s**    |\n\nOnce again, we sincerely thank the reviewer for the thoughtful and constructive feedback. We are committed to improving the clarity, robustness, and scalability of HME, and we will incorporate all the suggested revisions in our updated manuscript and supplementary material."}, {"review_id": "Xep4D20ZEZ", "reviewer": "Reviewer_5Y4d", "summary": "The authors propose Hyper-Modality Enhancement (HME)  that addresses the challenge of missing modalities in real-world scenarios. Instead of reconstructing missing modalities for MM sentiment analysis. The main ideas is that HME enriches missing modalities with semantically relevant information from other samples in the dataset. The main components are a Hyper-Modality Representation Generation Module that uses cross-sample enhancement and bottleneck for noise reduction, and an uncertainty-based fusion module that weights modality contribution based on variance estimates.", "questions": "1. How do you compare with SOTA for your baseline system?\n2. Is your method applicable to other / any MSA architecture?\n3. Variance based fusion is very flaky. Can you quantify your claims that for missing modalities your variance estimates are more robust?\n4. It would be interesting to note how your hyper-parameters transfer to a new database?\n5. How much data to you really need to identify relevant samples and to tune your hyperparameters?\n6. How would your approach work with more tightly integrated MM MSA frameworks, e.g., https://arxiv.org/pdf/2504.11082 or https://lab-msp.com/MSP-Podcast_Competition/IS2025/\n\n[I will revisit my review after some of these questions and issues are clarified by the authors]", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1.  Addresses an interesting and practical problem in a novel way: the cross-sample enhancement strategy is innovative leveraging information from semantically similar samples across the dataset.\n2. Good evaluation section: Extensive experiments on three benchmark datasets (CMU-MOSI, CMU-MOSEI, IEMOCAP) with both fixed and random missing protocols. Thorough analysis.\n3. Good result: HME achieves notable performance gains  - albeit there is quite a bit of tuning going on \n4. Uncertainty-aware fusion (while not very novel) seem to fit well in this framework\n5. The details in the appendix are much appreciated.\n\n\nWeaknesses\n\n1. Unclear to me how the baseline system compares to SOTA. \n2. Not clear how the proposed method could be plug n play to any MSA architecture.\n3. Scalability/performance concerns to a new dataset with very limited data and unknown distribution\n4. Hyperparameter Sensitivity: The method introduces several hyperparameters (similarity threshold, uncertainty bounds etc.) that that appear to require careful tuning for different datasets (see appendix they seem very spiky)", "was_revised": true, "final_justification": "The extensive additional experiments and clarifications significantly strengthen the paper's contributions and practical applicability.", "rebuttal": "We sincerely thank Reviewer 5Y4d for the time and thoughtful feedback provided. We are encouraged by your recognition that our work addresses a practical and important problem with a novel cross-sample enhancement strategy. Below, we respond point-by-point to your concerns and suggestions.\n### **Weaknesses & Questions:**\n\n**W1&Q1: Comparison with State-of-the-Art (SOTA) Methods** \n\nThank you for highlighting the need for clarification. Our baseline systems were selected from the latest state-of-the-art methods specifically designed for multimodal sentiment analysis under missing modality scenarios. These include GCNet (TPAMI 2023), MPLMM (ACL 2024), DiCMoR (ICCV 2023), IMDer (NeurIPS 2023), and LNLN (NeurIPS 2024). We will clearly state this selection criterion in the revised manuscript to avoid confusion and better contextualize our comparisons.\n\n**W2&Q2: Generalizability and Plug-and-Play Applicability of HME**\n\nWe appreciate your suggestion regarding the plug-and-play applicability of HME. This inspired us to further evaluate the modularity of our HME framework. To demonstrate its generality, we integrated our Hyper-Modality Representation Generation (HMG) module into the MPLMM architecture and evaluated it under random missing protocols on the MOSI dataset. As shown in Tables 1 and 2 below, incorporating HMG improved performance in most situations, with average accuracy and F1 gains of 1.2 and 1.8 across various missing rates. These results demonstrate the plug-and-play compatibility of HME with existing architectures and confirm its general effectiveness. We will include these findings in our revised paper.\n\nTable 1: Accuracy performance on MOSI with random missing protocol\n| methods  | 0.1  | 0.2  | 0.3  | 0.4  | 0.5  | 0.6  | 0.7  | avg  |\n| -------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| MPLMM    | 81.4 | 78.7 | 74.7 | 70.3 | 67.1 | 62.5 | 61.0 | 70.8 |\n| +HMG     | 81.9 | 78.5 | 75.0 | 72.1 | 69.2 | 65.2 | 61.9 | 72.0 |\n| $\\Delta$ | +0.5 | -0.2 | +0.3 | +1.8 | +2.1 | +2.7 | +0.9 | +1.2 |\n\nTable 2: F1 performance on MOSI with random missing protocol\n| methods  | 0.1  | 0.2  | 0.3  | 0.4  | 0.5  | 0.6  | 0.7  | avg  |\n| -------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| MPLMM    | 81.4 | 78.7 | 74.8 | 69.9 | 66.3 | 60.5 | 58.3 | 70.0 |\n| +HMG     | 81.9 | 78.6 | 75.1 | 72.3 | 69.3 | 64.5 | 61.1 | 71.8 |\n| $\\Delta$ | +0.5 | -0.1 | +0.3 | +2.4 | +3.0 | +4.0 | +2.8 | +1.8 |\n\n**Q3: Robustness of Variance-Based Fusion**\n\nWe conducted ablation studies to validate the contribution of the variance-based uncertainty fusion module. Specifically, we removed the uncertainty weighting (denoted as w/o $U_m$​) and observed performance drops across all datasets and settings, which is detailed in Tables 6 and 7 in appendix, confirming that our uncertainty estimates improve the model’s robustness in handling missing modalities.\n\n**W3&W4&Q4: Hyperparameter Sensitivity and Generalization to New Datasets**\n\nThank you for pointing out the concern regarding hyperparameter sensitivity and potential challenges in generalizing to new datasets. Inspired by your suggestion, we conducted additional experiments on two external datasets—**UR-FUNNY** and **MUStARD**—using the **same** hyperparameters originally tuned on MOSI. As reported in Tables 3 through 6 (to be included in the revised manuscript), **HME consistently outperformed the MPLMM baseline across various missing modality settings**, even without re-tuning. These results suggest that the proposed hyperparameters generalize reasonably well across datasets, demonstrating both the **scalability** and **robustness** of our method.\n\nTable 3: Accuracy performance on MUStARD with fixed missing protocol\n| methods | L    | A    | V    | LA   | LV   | AV   | LAV  | avg  |\n| ------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| MPLMM   | 67.6 | 61.8 | 61.8 | 67.6 | 67.6 | 61.8 | 67.6 | 65.1 |\n| HME     | 70.6 | 63.2 | 61.7 | 75.0 | 72.0 | 63.2 | 75.0 | 68.7 |\n\nTable 4: F1 performance on MUStARD with fixed missing protocol\n| methods | L    | A    | V    | LA   | LV   | AV   | LAV  | avg  |\n| ------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| MPLMM   | 68.1 | 61.8 | 61.8 | 68.1 | 68.1 | 61.8 | 68.1 | 65.4 |\n| HME     | 70.6 | 63.9 | 67.2 | 75.0 | 73.1 | 64.6 | 75.0 | 69.9 |\n\nTable 5: Accuracy performance on UR-FUNNY with fixed missing protocol\n| methods | L    | A    | V    | LA   | LV   | AV   | LAV  | avg  |\n| ------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| MPLMM   | 71.7 | 52.3 | 53.0 | 72.4 | 71.7 | 54.8 | 72.4 | 64.0 |\n| HME     | 72.9 | 53.3 | 52.5 | 72.8 | 72.9 | 55.5 | 72.6 | 64.6 |\n\nTable 6: F1 performance on UR-FUNNY with fixed missing protocol\n| methods | L    | A    | V    | LA   | LV   | AV   | LAV  | avg  |\n| ------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| MPLMM   | 71.8 | 54.3 | 54.9 | 72.5 | 71.8 | 55.2 | 72.5 | 64.7 |\n| HME     | 73.0 | 56.6 | 56.8 | 73.0 | 73.0 | 55.8 | 72.7 | 65.8 |\n\n**Q5: Data Requirements for Sample Retrieval and Hyperparameter Tuning**\n\nThank you for raising this important question. We address it from two perspectives:\n- **Relevant Sample Selection**: The number of semantically similar samples selected by the HME is closely tied to the batch size used during training. To provide a concrete analysis, we examined this on the MOSI dataset under a missing rate (MR) of 0.3. The Table 7 below summarizes the average number of selected relevant samples (with cosine similarity > 0.9) and the corresponding model performance. Even with a batch size of 64, HME is able to **retrieve nearly 8 highly similar samples** on average, which **already results in competitive performance**. At a batch size of 128, HME outperforms all baselines, and with 256, further gains are observed. These results demonstrate that HME does not require a large batch size to find meaningful enhancements—moderate-sized batches suffice.\n- **Hyperparameter Tuning**: Hyperparameter tuning is performed using the validation set. Taking MOSI as an example, the validation set contains 229 samples. With a batch size of 128 during validation, the effective tuning is conducted on batches of 128. We applied grid search for key hyperparameters using this relatively small validation set. Despite the limited size, **the resulting configuration generalized well across both validation and test scenarios**, as shown in our main experiments.\nIn conclusion, HME is able to identify relevant samples and perform effective tuning without requiring large-scale data. This aligns well with real-world constraints and supports the practicality of our HME.\n\nTable 7: Performance and average number of selected samples of HME in MOSI with MR=0.3\n| batchsize | Avg samples | Acc/F1    |\n| --------- | ----------- | --------- |\n| 64        | 7.84        | 79.1/78.6 |\n| 128       | 13.38       | 79.9/79.6 |\n| 256       | 25.15       | 81.1/81.0 |\n\n**Q6: Integration with More Tightly Coupled MSA Frameworks**\n\nWe appreciate your references to more tightly integrated multimodal frameworks (e.g., DeepMLF, MSP-Podcast Challenge). HME can be flexibly incorporated into these frameworks in three ways:\n1. **Representation-Level Integration**: HME can be applied after unimodal encoders to enhance feature robustness before fusion. Our integration with MPLMM already validates this design.\n2. **Fusion-Level Integration**: Many tightly coupled models still treat all modalities equally. Our uncertainty-aware fusion can dynamically adjust weights based on modality reliability, which is especially useful given the noisier nature of audio and video compared to text.\n3. **Broader Applicability to Missing Modalities**: These frameworks often assume all modalities are available. HME enables them to operate in **real-world missing-modality scenarios**, extending their practical utility.\nWe will elaborate on these integration strategies and provide implementation guidance in the updated manuscript.\n\nThank you once again for your constructive feedback. Your comments have helped us strengthen both the clarity and generalizability of our work. We hope our responses address your concerns and demonstrate the potential of HME to serve as a robust and flexible solution in real-world scenarios."}, {"review_id": "UhCWmL1gAa", "reviewer": "Reviewer_QBVh", "summary": "The paper is focused on the task of multimodal sentiment analysis with missing modalities. To address this task, the authors propose an approach, called Hyper-Modality Enhancement, which leverages cross-sample information to enrich each observed modality with semantic information from other samples and thus prevent performance degradation when some data modalities are missing. Furthermore, the approach uses an uncertainty-aware fusion mechanism that balances the original and enriched representations to improve model robustness. Experimental results on several benchmark datasets show that the proposed approach performs better than the baselines considered under several missing modality settings.", "questions": "Are there any existing multimodal datasets that have missing modalities already, so that one does not have to simulate that using the fixed missing and random missing protocols?\n\nIs the fixed missing protocol used here for the first time? What is the motivation for it?", "limitations": "The authors discuss several limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The approach proposed is intuitive and at the same time original. The two modules are well motivated and the authors do a good job at both providing the big picture and explaining the details. \n\nThe authors compare the performance of the proposed approach with multiple existing baselines on multiple datasets. Experimental results show that the proposed approach performs better than the baselines under various missing modality settings. \n\nAblation studies show that all components of the proposed approach contribute to the performance of the model. \n\nThere are a few components of the model that rely on prior works and approaches. The authors could provide some additional details about those approaches in the Appendix, if not in the main manuscript, to make it clearer what is novel to their approach. Otherwise, the proposed approach may be seen as rather incremental. \n\nThe authors evaluate the baseline models by comparison with the proposed approach under two missing modality protocols (fixed missing and random missing). At least one of the protocols (random missing) has been used in prior works. The authors used existing implementations to get results for the baselines. It is not clear if the exact data used in prior works (with missing modalities) was not available or why the authors were not able to compare directly with results from prior works. It may be that the results of the baselines are not fully optimized, although the authors followed the specific guidelines from prior works. Also, the authors did not justify well the fixed missing protocol, which may not have been used in prior works.", "was_revised": true, "final_justification": "I had several questions for the authors. They satisfactorily answered my questions. The inclusion of those clarifications in the final version of the paper would be useful.", "rebuttal": "We sincerely thank Reviewer QBVh for the time and effort invested in reviewing our paper, as well as for the constructive feedback provided. We are particularly encouraged that you found our approach both intuitive and original. Below, we address each of your comments in detail.\n### **Weaknesses & Questions:**\n\n**W1: Clarification of Components Based on Prior Work**\n\nThank you for pointing out the need to clarify which components of our HME build on prior work. We agree that distinguishing the novel aspects of our approach is important for transparency. In response, we will add more details to the Appendix—such as a clearer description of frameworks like Perceiver—to better contextualize our contributions. This will help readers more easily identify what is new in our HME.\n\n**Q1: On Existing Datasets with Naturally Missing Modalities**\n\nWe appreciate your insightful question regarding datasets that contain naturally missing modalities. To the best of our knowledge, such multimodal sentiment datasets are currently unavailable. As a result, the standard practice across the literature—including the baseline methods we compare against—is to simulate missing modalities by selectively dropping them from fully observed datasets. One of the key distinctions of our HME, is that it does not rely on the ground-truth representations of missing modalities during simulation. This design choice aims to more closely reflect real-world conditions and enhance the practical relevance of our method.\n\n**Q2: Motivation for the Fixed Missing Protocol**\n\nThank you for raising the question regarding the fixed missing protocol. This is not the first time such a protocol has been used. Notable examples include baseline models like DiCMoR (ICCV 2023), IMDer (NeurIPS 2023), and LNLN (NeurIPS 2024). The motivation behind the fixed missing setting is to simulate realistic scenarios where specific modalities may be consistently unavailable—for example, when an audio sensor fails and only textual and visual information is recorded. Evaluating under this protocol helps better understand how models perform under modality-specific failures that could occur in deployment environments.\n\n**W2: Clarification on Baseline Results**\n\nWe also appreciate your concern regarding the reproducibility and optimization of baseline results. In some cases, we were unable to directly compare with results reported in prior work under identical settings—either because the specific missing modality protocol was not used, or the results for certain datasets (such as IEMOCAP) were not reported for those conditions. In such cases, we carefully followed the official implementations and experimental guidelines provided by the original authors to ensure fair and meaningful comparisons. The implementation details are shown in Lines 459-464 in appendix: \"To identify the hyper-parameter of the re-implemented baselines, we adhered to the hyper-parameter guidelines provided in the official implementations of these models. For models like IMDer and DiCMoR, which included pre-trained weights, we used the publicly available checkpoints and configurations. For baselines without available hyper-parameters for specific datasets, we aligned their hyper-parameters with HME’s settings where possible. When unique parameters were required, we referred to their recommended settings from other publicly available implementations.\" \n\nOnce again, we are grateful for your thoughtful feedback, which has helped us further improve the clarity and rigor of our work."}, {"review_id": "X8Z0MnQDQL", "reviewer": "Reviewer_py7h", "summary": "In this paper, authors introduce Hyper-Modality Enhancement (HME), a novel framework for robust multimodal sentiment analysis under missing modality conditions. By leveraging cross-sample semantic cues to enrich observed modalities without explicit reconstruction, HME addresses limitations of prior methods that rely on pseudo-missing simulations or teacher-student frameworks. The approach comprises three key components: (1) a hyper-modality generation module that distills sentiment-relevant information from other samples via learnable prompts and variational information bottleneck (VIB) for noise filtering, (2) an uncertainty-aware fusion mechanism that adaptively balances original and enhanced representations based on representation reliability, and (3) extensive experiments demonstrating consistent performance gains over SOTA methods on CMU-MOSI, CMU-MOSEI, and IEMOCAP datasets under various missing modalities.", "questions": "1. Authors claim VIB reduces noise and redundancy, but it’s unclear how VIB interacts with the uncertainty-aware fusion mechanism. For example:\n(1)Does VIB’s compression degrade important sentiment signals?\n(2)How does the balance between compression (I(Z,X)) and informativeness (I(Z,Y)) affect performance?\n\n2. Experiments focus on in-dataset missingness (fixed/random masks), but real-world missingness often arises from domain shifts (e.g., sensor failures in deployed systems). How would HME perform on datasets with:\n(1)Natural missingness (e.g., Twitter posts with missing images)?\n(2)Cross-dataset evaluation (e.g., training on CMU-MOSI and testing on IEMOCAP)?\n\n3. Why use cosine similarity for cross-sample selection? Are there alternatives (e.g., contrastive learning)?\n\n4. How does the uncertainty weighting (Eq. 5) relate to uncertainty quantification in Bayesian deep learning?\n\n5.What are the ethical risks of inferring emotions from incomplete data (e.g., misrepresentation of marginalized groups)?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths :\nThe content of this paper presents a technically sound and well-motivated framework for robust multimodal sentiment analysis under missing modalities:\n(1) HME addresses key limitations of prior work by avoiding explicit modality reconstruction and instead leveraging cross-sample semantic cues through learnable prompts and a variational information bottleneck (VIB).\n(2) Three-module design, modality enhancement, hyper-modality generation, and uncertainty-aware fusion, is conceptually clear and addresses both noise reduction and representation enrichment. \n(3) The experimental evaluation is thorough, demonstrating consistent improvements over strong baselines (e.g., IMDer, LNLN) across three datasets (CMU-MOSI, CMU-MOSEI, IEMOCAP) and varying missing-modality scenarios. \n\nWeakness:\nTechnical contributions are solid, but the paper’s evaluation has limitations that temper its impact. \n(1) Experiments focus primarily on within-dataset generalization under controlled missingness patterns (fixed/random missing protocols), but do not explore domain shifts or out-of-distribution robustness. \n(2) Theoretical underpinnings of certain design choices, such as the similarity threshold (tₛ) for cross-sample selection and the interplay between VIB compression and uncertainty weighting, are not fully justified. \n(3) Finally, while the visualizations (e.g., t-SNE plots) aid interpretability, the paper could better articulate how hyper-modality representations qualitatively differ from raw features and why this distinction matters for sentiment prediction.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer py7h for the time and effort dedicated to reviewing our paper. We deeply appreciate your recognition of our work as \"a technically sound and well-motivated framework\", and we value the insightful comments and constructive suggestions you provided. Below, we respond to each of your concerns in detail.\n### **Weaknesses & Questions:**\n\n**W2&Q1: Interaction Between VIB and Uncertainty-Aware Fusion**\n\nThank you for highlighting this important aspect. Your comments helped us further explore the interplay between the variational information bottleneck (VIB) and the uncertainty-aware fusion.  \n\n(1) The VIB module is specifically introduced to filter out misleading semantic signals that may appear during cross-sample enhancement. For example, as shown in **Table 10 (Appendix)**, the selected similar samples may contain sentiment-relevant cues that contradict the target sample’s sentiment label, leading to performance degradation. Rather than aggregating all available signals, VIB helps retain only reliable features. The ablation results (**Tables 6 and 7 in Appendix**) show that removing VIB results in a consistent performance drop across all datasets, indicating its effectiveness in suppressing noise and misleading cues. \n\n(2) Motivated by your question on the trade-off between $I(Z,X)$ and $I(Z,Y)$, we conducted a detailed study by varying their weights individually and jointly. Results (see Table 1 below) show that increasing the $I(Z,X)$ weight (i.e., stronger compression) tends to suppress useful sentiment information, degrading performance. Conversely, while increasing $I(Z,Y)$ initially improves results, excessive emphasis eventually hurts performance—likely due to under-utilization of complementary features or overfitting. \n\nTable 1: Performance comparison under different weights of VIB components\n|   I(Z,X)   |    0.1    |    0.2    |    0.5    |     1     |     2     |     5     |\n| :--------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: |\n|    HME     | 78.0/75.9 | 77.4/75.3 | 76.8/74.9 | 76.5/73.7 | 76.0/72.8 | 75.5/71.3 |\n| **I(Z,Y)** |  **0.1**  |  **0.2**  |  **0.5**  |   **1**   |   **2**   |   **5**   |\n|    HME     | 77.3/74.9 | 77.5/75.4 | 77.6/76.0 | 77.5/75.1 | 77.3/74.9 | 77.1/74.5 |\n|  **VIB**   |  **0.1**  |  **0.2**  |  **0.5**  |   **1**   |   **2**   |   **5**   |\n|    HME     | 77.7/75.1 | 77.8/75.5 | 77.6/76.0 | 76.9/75.0 | 76.5/74.4 | 75.7/73.8 |\n\n**W1&Q2: Evaluation Under Real-World Missingness and Cross-Domain Settings**\n\nWe agree that assessing performance under natural missingness and domain shifts is crucial for real-world applicability. Due to the lack of standard datasets with naturally missing modalities (e.g., social media posts with missing images or audio), we followed existing work (IMDer (NeurIPS 2023) and LNLN (NeurIPS 2024)) in simulating missingness. However, HME differs fundamentally by not relying on ground-truth representations of the missing modalities, aiming for more realistic simulation. \n\nFor domain-shifts scenarios, we have extended our experiments to scenarios using the MUStARD and UR-FUNNY sarcasm detection datasets. Results (Tables 2–5 below) show that HME consistently outperforms baseline across various missing-modality conditions, demonstrating improved robustness in domain-shift settings.\n\nAs for cross-dataset generalization, we faced challenges due to label inconsistency (e.g., binary vs. four-class sentiment in MOSI/MOSEI vs. IEMOCAP) and differing input dimensions (e.g., (5/20 vs. 74/35) of audio/video features in MOSI vs. MOSEI). Nonetheless, we acknowledge its importance and will extend our experiments using the MUStARD and UR-FUNNY sarcasm detection datasets. We will include these findings and additional experiments in our revised manuscript.\n\nTable 2: Accuracy performance on MUStARD with fixed missing protocol\n| methods | L    | A    | V    | LA   | LV   | AV   | LAV  | avg  |\n| ------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| MPLMM   | 67.6 | 61.8 | 61.8 | 67.6 | 67.6 | 61.8 | 67.6 | 65.1 |\n| HME     | 70.6 | 63.2 | 61.7 | 75.0 | 72.0 | 63.2 | 75.0 | 68.7 |\n\nTable 3: F1 performance on MUStARD with fixed missing protocol\n| methods | L    | A    | V    | LA   | LV   | AV   | LAV  | avg  |\n| ------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| MPLMM   | 68.1 | 61.8 | 61.8 | 68.1 | 68.1 | 61.8 | 68.1 | 65.4 |\n| HME     | 70.6 | 63.9 | 67.2 | 75.0 | 73.1 | 64.6 | 75.0 | 69.9 |\n\nTable 4: Accuracy performance on UR-FUNNY with fixed missing protocol\n\n| methods | L    | A    | V    | LA   | LV   | AV   | LAV  | avg  |\n| ------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| MPLMM   | 71.7 | 52.3 | 53.0 | 72.4 | 71.7 | 54.8 | 72.4 | 64.0 |\n| HME     | 72.9 | 53.3 | 52.5 | 72.8 | 72.9 | 55.5 | 72.6 | 64.6 |\n\nTable 5: F1 performance on UR-FUNNY with fixed missing protocol\n\n| methods | L    | A    | V    | LA   | LV   | AV   | LAV  | avg  |\n| ------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| MPLMM   | 71.8 | 54.3 | 54.9 | 72.5 | 71.8 | 55.2 | 72.5 | 64.7 |\n| HME     | 73.0 | 56.6 | 56.8 | 73.0 | 73.0 | 55.8 | 72.7 | 65.8 |\n\n**Q3: Choice of cosine similarity for cross-sample selection**\n\nThank you for this valuable suggestion. We chose cosine similarity due to its robustness to curse of dimensionality and its widespread use in related work. Moreover, many contrastive learning methods—including CLIP—also rely on cosine similarity as a similarity metric.\n\nIn our case, cross-sample selection must be unsupervised, since real-world missingness often occurs in settings without label access. Unlike contrastive learning approaches that require carefully curated positive/negative pairs or rely on known modality alignment (as in CLIP), HME is designed to flexibly select multiple relevant samples without supervision, improving adaptability across diverse applications.\n\n**Q4: Relationship between Eq. 5 and Bayesian Deep Learning (BDL) uncertainty**\n\nEquation 5 draws inspiration from uncertainty quantification principles in BDL, particularly in its use of variance to estimate uncertainty. However, while BDL typically estimates model uncertainty via techniques like Monte Carlo dropout, our method estimates uncertainty at the representation level—specifically, the robustness of VIB-compressed features. Thus, Eq. 5 shares a conceptual link with BDL, but it serves a distinct purpose in modulating the fusion of original and enhanced modalities based on feature reliability.\n\n**Q5: Ethical risks of emotion inference from incomplete data**\n\nWe fully acknowledge the ethical implications of inferring emotions from incomplete inputs. Inaccurate emotion recognition can lead to misrepresentation, particularly of marginalized groups, or raise concerns about profiling and privacy. While our method aims to reduce such risks by improving robustness to missing modalities, we recognize the importance of transparent and responsible deployment. We will explicitly discuss these concerns and potential safeguards in our revised manuscript.\n\n**W3: Visualizing hyper-modality representations**\n\nThank you for pointing out the need for clearer qualitative explanations of our hyper-modality representations. While we include relevant visualizations (e.g., **Figure 3b**, **Figure 5**, and **Figure 6** in the appendix), we agree that these could be more illustrative.\n\nTo address this, we plan to extend our qualitative analysis in the revision, including:\n- Visualization of how varying the similarity threshold affects selected samples,\n- Analysis of attention weights during enhancement,\n- Measurement of distance between original, enhanced, and hyper-modality embeddings.\nThis will help demonstrate how hyper-modality differs meaningfully from raw or enhanced representations and why that distinction is important for sentiment understanding.\n\nWe deeply appreciate your thoughtful feedback, which has significantly improved the clarity and depth of our work. We are enthusiastic about further enhancing our manuscript based on your suggestions and thank you again for your constructive review."}, {"review_id": "StbgaxdeiS", "reviewer": "Reviewer_bLRy", "summary": "This paper proposes a novel framework, Hyper-Modality Enhancement (HME), for Multimodal Sentiment Analysis (MSA) in the presence of missing modalities. The method begins by extracting modality-specific representations, then enhances them using batch-level cross-sample information via a modality enhancement module. These enhanced features are integrated through a hyper-modality generation module that fuses cues across all modalities. To address redundancy and noise, the framework incorporates a Variational Information Bottleneck (VIB). Experimental results on multiple benchmark datasets demonstrate that HME outperforms prior methods, especially under high missing rates, showing strong robustness and generalization.", "questions": "1. Experimental setting clarification: It is unclear whether the missing modality setting is consistent across training and testing. For example, when evaluating under “L missing”, is the language modality also missing during training, or only during inference? This clarification is important for interpreting the robustness of the model.\n\n5. Clarification of Figure 3b (Section 4.4): The authors state that the enhanced hyper-modality representations capture more sentiment-relevant information, forming distinct clusters in the t-SNE plot. Could you elaborate more on this point?", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper tackles the underexplored challenge of missing modalities in MSA using a cross-sample enhancement mechanism, which introduces complementary information from semantically similar samples in the same batch—without relying on full modality reconstruction.\n\n2. The uncertainty-aware fusion strategy is thoughtfully designed, using modality-wise variance to weight the contribution of each representation. The accompanying qualitative analysis of uncertainty weights provides insightful understanding of how different modalities contribute under varying conditions.\n\nWeakness:\n\n1. Batch size sensitivity: Since enhanced features are derived from samples within the same batch, it would be valuable to analyze how performance varies with batch size. This could help understand the dependency of HME on batch-level diversity. Qualitative examples would be helpful to understand why certain samples were chosen for enrichment.\n\n2. Interpretation of prompts: The paper could benefit from further analysis of the learnable prompts used in the Perceiver and hyper-modality interaction modules. What information is being captured or attended to by these prompts?\n\n3. More detailed ablation on VIB: While VIB is shown to reduce redundancy, a more granular study of the influence of the VIB loss weight $\\alpha$ would be useful. Additionally, a comparison of performance with and without VIB (especially under different MR values) would clarify its specific contribution.", "was_revised": true, "final_justification": "The rebuttal addressed all my questions and the experiments are well presented. I have raised my rating.", "rebuttal": "We sincerely thank Reviewer bLRy for your time and insightful feedback. We are especially grateful for your recognition of the thoughtful design of our proposed framework. Your comments have significantly helped us deepen our analysis and improve the clarity of our paper. Below, we address each of your concerns in detail.\n### **Weaknesses & Questions:**\n\n**W1: Batch Size Sensitivity & Qualitative Examples of Enhancement**\n\n(1) Thank you for highlighting the potential impact of batch size on our HME. We agree that this is a crucial aspect that can offer valuable insights into HME’s robustness and practical deployment. To explore this, we conducted experiments on the MOSI dataset under the random missing protocol, evaluating batch sizes of 64, 128, and 256. The results are summarized in Tables 1 and 2 below (to be included in the appendix). We observed that larger batch sizes consistently lead to better performance. This is likely because a larger batch offers more diverse and semantically similar samples, allowing the enhancement module to extract richer information. These findings will be included in the revised appendix.\n\nTable 1: Accuracy performance on MOSI with random missing protocol\n| batch size | 0.1  | 0.2  | 0.3  | 0.4  | 0.5  | 0.6  | 0.7  | avg  |\n| ---------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| 64         | 83.8 | 81.1 | 79.1 | 76.1 | 74.2 | 72.3 | 69.7 | 76.6 |\n| 128        | 83.8 | 81.4 | 79.9 | 76.7 | 76.5 | 72.4 | 70.3 | 77.3 |\n| 256        | 84.9 | 82.9 | 81.1 | 79.9 | 76.4 | 74.5 | 73.5 | 79.0 |\n\nTable 2: F1 performance on MOSI with random missing protocol\n| batch size | 0.1  | 0.2  | 0.3  | 0.4  | 0.5  | 0.6  | 0.7  | avg  |\n| ---------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| 64         | 83.8 | 81.2 | 78.6 | 76.1 | 73.9 | 71.2 | 69.2 | 76.3 |\n| 128        | 83.8 | 81.4 | 79.6 | 76.3 | 75.9 | 72.1 | 69.7 | 77.0 |\n| 256        | 84.7 | 82.9 | 81.0 | 80.0 | 76.4 | 74.4 | 71.7 | 78.7 |\n\n(2) As for qualitative examples of enhancement, We appreciate your suggestion to include more qualitative examples. While Figure 6 in appendix currently shows some case visualizations from the MOSI dataset, we agree that it would be beneficial to expand this analysis. In the revised version, we will include visualizations under varying batch sizes and similarity thresholds, highlighting which samples are selected for enhancement and why. This will provide readers with a more concrete grasp of how and why enhancement contributes to performance.\n\n**W2: Interpretation of Learnable Prompts**  \n\nThank you for your insightful comment regarding the prompts used in the Perceiver and hyper-modality interaction modules. Inspired by your suggestion, we realized that simple visualization (as shown in Figures 3b and 5) is insufficient to fully convey their function. We will enhance our qualitative analysis by including:\n- Visualization of how changing the similarity threshold affects prompt behavior and sample selection,\n- Analysis of attention weights assigned to each prompt during enhancement,\n- Measurements of distances between original, enhanced, and hyper-modality embeddings.\n\nThese analyses will clarify the function and utility of prompts in attending to sentiment-relevant information and will help distinguish the role of hyper-modality from raw or unimodal representations.\n\n**W3: Detailed Ablation on the Variational Information Bottleneck (VIB)**  \n\nWe thank you for encouraging a deeper investigation into the role of the VIB module. Following your suggestion, we conducted a more granular analysis of  $I(Z,X)$ and $I(Z,Y)$. Specifically, we varied their weights individually and jointly. As shown in Table 3 below, increasing the weight of $I(Z,X)$ compresses representations too aggressively, degrading performance by removing sentiment-relevant features. Conversely, increasing $I(Z,Y)$ improves performance up to a point, after which it declines—likely due to overfitting or underemphasizing alignment with input features. Jointly increasing both follows a similar trend.\n\nTable 3: Performance comparison under different weights of VIB components\n|   I(Z,X)   |    0.1    |    0.2    |    0.5    |     1     |     2     |     5     |\n| :--------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: |\n|    HME     | 78.0/75.9 | 77.4/75.3 | 76.8/74.9 | 76.5/73.7 | 76.0/72.8 | 75.5/71.3 |\n| **I(Z,Y)** |  **0.1**  |  **0.2**  |  **0.5**  |   **1**   |   **2**   |   **5**   |\n|    HME     | 77.3/74.9 | 77.5/75.4 | 77.6/76.0 | 77.5/75.1 | 77.3/74.9 | 77.1/74.5 |\n|  **VIB**   |  **0.1**  |  **0.2**  |  **0.5**  |   **1**   |   **2**   |   **5**   |\n|    HME     | 77.7/75.1 | 77.8/75.5 | 77.6/76.0 | 76.9/75.0 | 76.5/74.4 | 75.7/73.8 |\n\nAdditionally, to further demonstrate the effectiveness of VIB, we provide an ablation study comparing HME with and without the VIB component across various missing rates and datasets, as shown in Tables 6 and 7 in the appendix. These results consistently show performance drops when VIB is removed, underscoring its contribution to robustness and representation quality.\n\n**Q1: Experimental Setting Clarification**  \n\nThank you for pointing this out. We clarify that in the _fixed missing modality_ setting (e.g., when “L” is missing), the modality is only removed during validation and inference—not during training. This setup aligns with prior works (GCNet (TPAMI 2023), MPLMM (ACL 2024), DiCMoR (ICCV 2023), IMDer (NeurIPS 2023), and LNLN (NeurIPS 2024)) and is intended to simulate real-world scenarios where certain modalities may be missing only at test time. In contrast, under the _random missing protocol_, the same missing rate is applied uniformly during training, validation, and testing. We will revise the paper to state this more explicitly.\n\n**Q2: Clarification of Figure 3b (Section 4.4)**  \n\nWe apologize for the lack of clarity in our explanation. Figure 3b, when considered alongside Figure 5 (in the appendix), provides a more complete picture. In Figure 3b, the t-SNE plot shows that the hyper-modality representations ($F_m$) form more distinct sentiment-aware clusters than the original unimodal representations ($H_m$), indicating that $F_m$ captures more discriminative information. Figure 5 further demonstrates that after applying uncertainty-aware fusion, the $F_m$ representation is effectively integrated with the original $H_m$, resulting in more structured and separable clusters. Together, these visualizations support our claim that the hyper-modality enhances sentiment-relevant information in a meaningful way.\n\nOnce again, we sincerely thank the reviewer for your detailed and thoughtful feedback, which has helped us improve both the depth and clarity of our paper. We will incorporate these enhancements into our revision."}]

Author rebuttal:

Meta-review: Summary. This paper introduces Hyper-Modality Enhancement (HME) framework for Multimodal Sentiment Analysis, which is robust against missing modalities. The main contribution is a two-stage pipeline: (1) a "Hyper-Modality Representation Generation" module enriches the representation of each available modality by borrowing semantically relevant information from other samples within the same training batch. (2) an "Uncertainty-Aware Fusion" mechanism adaptively combines the original modality representations with these newly enhanced ones. 

Strengths. The proposed method looks technically sound and well-motivated. The empirical evaluation is thorough and convincing. The paper is well-written, structured, and easy to follow.

Weaknesses. The method seems sensitive to batch size and other parameters. Authors plan to clarify this in the final version. 

The paper received 5,5,4,4,4 ratings. Authors submitted a rebuttal to reviewer comments and reviewers remained positive ratings. 

Overall, a good paper with interesting results for robust multimodal sentiment analysis. We expect that authors will revise the paper according to reviewer comments and include all the promised changes in the final version.
Decision: Accept (poster)