Towards Robust Multimodal Sentiment Analysis with Incomplete Data
Authors: Haoyu Zhang
Wenbin Wang
Tianshu Yu
Venue: NeurIPS 2024 poster (2024)
Keywords: Multimodal Sentiment Analysis
Multimodal Learning
Robust MSA
Primary area: human-AI_interaction

Abstract: The field of Multimodal Sentiment Analysis (MSA) has recently witnessed an emerging direction seeking to tackle the issue of data incompleteness. Recognizing that the language modality typically contains dense sentiment information, we consider it as the dominant modality and present an innovative Language-dominated Noise-resistant Learning Network (LNLN) to achieve robust MSA. The proposed LNLN features a dominant modality correction (DMC) module and dominant modality based multimodal learning (DMML) module, which enhances the model's robustness across various noise scenarios by ensuring the quality of dominant modality representations. Aside from the methodical design, we perform comprehensive experiments under random data missing scenarios, utilizing diverse and meaningful settings on several popular datasets (e.g., MOSI, MOSEI, and SIMS), providing additional uniformity, transparency, and fairness compared to existing evaluations in the literature. Empirically, LNLN consistently outperforms existing baselines, demonstrating superior performance across these challenging and extensive evaluation metrics.

Towards Robust Multimodal Sentiment Analysis with Incomplete Data
Haoyu Zhangl.2, Wenbin Wang', Tianshu Yu
School of Data Science, The Chinese University of Hong Kong, Shenzhen
Department of Computer Science, University College London
Schooo of Computer Science, Wuhan University
{zhanghaoyu, yutianshu)@cuhl, ceruuhk.edu.c.. cn
haoyu.zhang 230ucl ac.uk
wangwenbin97ewin @whu... www..wwu.u...e.oeommmooooooeeeee cn
Abstract
The field of Multimodal Sentiment Analysis (MSA) has recently witnessed an emerging direction seeking to tackle the issue of data incompleteness. Recogniz- ing that the language modality typically contains dense sentiment information, we consider it the dominant modality and present an innovative Language- dominated Noisc-resistant Learning Network (LNLN) to achieve robust MSA. The proposed LNLN features a dominant modality correction (DMC) module and dominant modality based multimodal learning (DMML) module, which enhances the model's robustness across various noise scenarios by ensuring the quality dominant modality representations. Aside from the methodical design, we perform comprehensive experiments under random data missing scenarios, utilizing diverse and meaningful settings several popular datasets (e.g.. MOSI, MOSEI, and SIMS), providing additional uniformity, transparency, and fairness compared to existing evaluations in the literature. Empirically, LNLN consistently outperforms existing baselines, demonstrating superior performance across these challenging and extensive evaluation metrics.
Introduction
The field of Multimodal Sentiment Analysis (MSA) is at the vanguard of human sentiment identifica- tion by assimilating heterogeneous data types, such video, audio, and language. Its applicability spans numerous fields, prominent among healthcare and human-computer interaction (Jiang et al., 2020). MSA has become essential in enhancing both the precision and the robustness of sentiment analysis by drawing sentiment cues from diverse perspectives.
Changing trends in recent research has taken a turn towards modeling data in natural scenarios from laboratory conditions (Tsai et al., 2019; Hazarika et al., 2020; Yu et al., 2021; Zhang al., 2023). The shift has created a wider application space in the real-world for MSA, though concerns arise due to problems like sensor failures and problems with Automatic Speech Recognition (ASR), leading to inconsistencies such as incomplete data in real-world deployment.
Numerous impactful solutions have been proposed against this primary concern of incomplete data in multimodal sentiment analysis. For instance, Yuan et al. (2021) introduced a Transformer- based feature reconstruction mechanism, TFR-Net, aiming strengthen the robustness of the model handling random missing in unaligned multimodal sequences via reconstructing missing data.
the corresponding author
'The code is available at: https://gitae http:://gichhb...m/aaaoobb///b/ ha/LNLN
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Furthermore, Yuan et al. (2024) proposed the Noise Intimating- based Adversarial Training (NIAT) model, which superiorly learns unified joint representation between an original-noisy instance pair. utilizing the attention mechanism and adversarial learning. Lastly. Li et al. (2024) design a Unified Multimodal Missing Modality Self-Distillation Framework (UMDF) which leverages single network to learn robust inherent representations from consistent multimodal data distributions. Yet, despite these developments, the evaluation metrics for these models are inconsistent, and the evaluation settings are not sufficiently comprehensive. This inconsistency limits effective comparisons and hinders the dissemination knowledge in the field.
Addressing this gap, our paper aims to offer comprehensive evaluation on three widely- datasets, namely MOSI (Zadeh et al., 2016), MOSEI (Zadeh al., 2018) and SIMS (Yu al., 2020) datasets. We introduce random data missing instances and subsequently compare the performance of existing methods on these datasets. This endeavor seeks to provide an all-encompassing outlook for evaluating the effectiveness and robustness of various methods in the face of incomplete data, thereby sparking new insights in the field. Additionally, inspired by the previous work ALMT (Zhang et al., 2023), we hypothesize that model robustness improves when the integrity of the dominant modality is preserved despite varying noise levels. Therefore, we introduce a novel model, namely Language-dominated Noise-resistant Learning Network (LNLN), to enhance MSA's robustness over incomplete data. LNLN aims to augment the integrity of the language modality's 's features, regarded as the dominant modality due its richer sentiment cues, with the support of other auxiliary modalities. The LNLN's ' robustness against varying levels of data incompleteness is achieved through a dominant modality correction (DMC) module for dominant modality construction, dominant modality based multimodal learning (DMML) module for multimodal fusion and classification, and a reconstructor for reconstructing missing information
shield the dominate modality from noise interference. This approach ensures a high-quality dominant modality feature, which significantly bolsters the robustness of LNLN under diverse noise conditions. Consequently, extensive experimental results demonstrate the LNLN's superior performance across these challenging and evaluation metrics.
In summary, this paper conducts a comprehensive evaluation of existing advanced MSA methods. This analysis highlights the strengths and weaknesses of various methods when contending with incomplete data. We believe, it can improve the understanding of different MSA methods' performance under complex real-world scenarios, thereby informing technology's 's future trajectory. Our proposed LNLN also offers valuable insight and guidelines for thriving in this research space.
2
Related Work
2.1 Multimodal Sentiment Analysis
Multimodal Sentiment Analysis (MSA) methods can be categorized into Context-based MSA and Noise-aware MSA, depending on the modeling approach. Most of previous works (Zadeh et al., 2017; Tsai et al., 2019; Mai et al., 2020; Hazarika et al., 2020; Liang et al., 2020; Rahman et al., 2020; Yu et al., 2021; Han et al., 2021; Lv et al., 2021; Yang et al., 2022: Guo al., 2022: Zhang al., 2023) can be classified to Context-based MSA. This line of work primarily focuses on learning unified multimodal representations by analyzing contextual relationships within or between modalities. For example, Zadeh et al. (2017) explore computing the relationships between different modalities using the Cartesian product. Tsai et al. (2019) utilize pairs of Transformers to model long dependencies between different modalities. Yu et al. (2021) propose generating lacudo-labels for each modality to further mine the information of consistency and discrepancy between different modalities.
Despite these advances, context-based methods are usually suboptimal under varying levels of noise effects (e.g. random data missing). Several recent works (Mittal et al., 2020; Yuan et al., 2021, 2024; Li et al., 2024) have been proposed to tackle this issue. For example, Mittal et al. (2020) introduce a modality check step to distinguish invalid and valid modalities, achieving higher robustness. Yuan et al. (2024) propose learning unified joint representation between constructed "original-noisy" instance pairs. Although there have been some advances in improving the model's robustness under noise scenarios, no extant method has provided comprehensive and in-depth comparative analysis.

---

2.2 Robust Representation Learning in MSA
Context-based MSA and Noise-aware MSA differ in their approaches to robust representation learning. In Context-based MSA, robust representation learning typically relies on modeling intra- and inter-modality relationships. For instance, Hazarika et al. (2020) and Yang et al. (2022) apply feature disentanglement to each modality, modeling multimodal representations from multiple feature subspaces and perspectives. Yu et al. (2021) and Liang et al. (2020) explore self- self-supervised learning and semi-supervised learning to enhance multimodal representations, respectively. Tsai et al. (2019) and Rahman et al. (2020) introduce Transformer to learn the long dependencíes of modalities. Zhang et al. (2023) devise language-guided learning mechanism that uses modalities with more intensive sentiment cues to guide the learning of other modalities. In contrast, Noise-aware MSA focuses more on perceiving and eliminating the noise present in the data. For example, Mittal et al. (2020) design a modality check module based on metric learning and Canonical Correlation Analysis (CCA) to identify the modality with greater noise. Yuan et al. (2021) design a feature reconstruction network predict the location of missing information in sequences and reconstruct it. Yuan et al. (2024) introduce adversarial learning (Goodfellow et al., 2014) to perceive and generate cleaner representations.
In this work, the LNLN belongs to the noise-aware MSA category. Inspired by Zhang et al. (2023), we explore the capability of language-guidee mechanisms in resisting noise and aim to provide new perspectives for the study of MSA in noisy scenarios.
3 Method
3.1
Overview
The overall pipeline for the proposed Language-dominated Noise-resistant Learning Network (LNLN) in robust multimodal sentiment analysis is illustrated in Figure 1. As depicted, crucial initial step involves forming a multimodal input with random data missing, which sets the stage for LNLN training. Once the input is prepared, LNLN first utilizes an embedding layer to standardize the dimension of each modality, ensuring uniformity. Recognizing that language is the dominant modality in MSA (Zhang et al., 2023), a specially designed Dominant Modality Correction (DMC) module employs adversarial learning and dynamic weighted enhancement strategy to mitigate noise impacts. This module first enhances the quality of the dominant feature computed from the language modality and then integrates them with the auxiliary modalities (visual and audio) in the dominant modality based multimodal learning (DMML) module for effective multimodal fusion and classification. This process significantly bolsters LNLN's robustness against various noise levels. Moreover, to refine the network's is capability for fine-grained sentiment analysis, simple reconstructor is implemented to reconstruct missing data, further enhancing the system's 's robustness.
3.2 Input Construction and Multimodal Input
Given the challenges of random data missing, we have constructed data sets that simulate these conditions based on MOSI, MOSEI, and SIMS datasets.
Random Data Missing, Following the previous method (Yuan et al., 2021), for each modality, we randomly erased changing proportions of information (from 0% to 100%). Specifically, for visual and audio modalities, we fill the erased information with zeros. For language modality, we fill the erased information with [UNK] which indicates the unknown word in BERT (Kenton and Toutanova, 2019).
For each sample in the dataset, we incorporate data from three modalities: language, audio, and visual data. Consistent with previous works (Zhang et al., 2023), each modality is processed using widely-used tools: language data is encoded using BERT (Kenton and Toutanova, 2019), audio features are extracted through Librosa (McFee et al., 2015), and visual features are obtained using OpenFace (Baltrusaitis et al., 2018). These pre-processed inputs are represented as sequences, denoted by U TT... xd.. where m {l. a) represents the modality type (1 for language, v for visual, a for audio), Tm indicates the sequence length and dm refers to the dimension of each modality's 's vector. With obtained U apply random data missing to U thus forming the noise-corruptdd multimodal input U1_ E T xdm.
Multimdoal Input.
3

---

Multimodal Sample
Ri
Completeness
Dominate Modality Correction
Reconstruction saper
Reconstructor
Language
Video
Audio
R1
Check
X
Mi
Feature Generation
Pracy Domnent
- 02
Praxy Dominant Feature Generator
Effectiveness Discriminator
112
Feature
Extraction
112
Dominant Modality based Multimodal Learning
07
Language Embedding
Hyper- modal iit Learning
Fusion Transformer
Cross madillty
Data Missing
0g
Adaptiive
NC
Embeddlig
Lia
02
01
81
H2
Embedding
Multimodal Input
Training Pipeline
Figure 1: Overall pipeline. Note: HP. H. Ha, Heer and Hy randomly initialized learnable vectors.
3.3 Dominant Modality based Multimodal Learning
Inspired by previous work ALMT, we hypothesize that model robustness improves when the integrity of the dominant modality is preserved despite varying noise levels. We improve ALMT based on the designed DMC module and Reconstructor, thus implementing dominant modality based DMML module for sentiment analysis under random data missing scenarios. Here, we mainly introduce the parts that differ from ALMT. Further details are available in Zhang et al. (2023).
Modality Embedding. For multimodal input U we employ an Embedding Encoder E,, with two Transformer encoder layers to extract and unify the feature. Each modality begins with randomly initialized low-dimensional token Hº E R xxd.. These tokens are then processed by the Transformer encoder layer, embedding essential modality information and producing unified features, represented as H E RXxd The process formalized by:
H m E,, (concat (H U
(1)
where E!! (-) extracts features for each modality, and concat(-) represents the concatenation operation.
Adaptive Hyper-modality Learning. In the original ALMT, each Adaptive Hyper-modality Learning layer contains Transformer and two multi-head attention (MHA) modules. These are applied to learn language representations at different scales and hyper-modality representations from visual and audio modalities, guided by the language modality. Considering the possibility of severe interference in the language modality (i.e. dominant modality) due random data missing, we designed Dominant Modality Correction (DMC) module to generate the proxy dominant feature H and construct corrected dominate feature H3 (more details can be found in Section 3.4). Specifically, the process of learning corrected dominated representation H at different scales can be described as:
=a= E (H1-1),
where i (2,3) means the i-th layer of Adaptive Hyper-modalin Learning module, E the i-th Transformer encoder layer, H; E RT'xd is corrected dominated feature different scale. To learn the hyper-modality representation, the corrected dominated feature and audio/visual features are used calculate Query and Key/Value, respectively. Briefly, the process be written as follows:
Hi Happer Hayper MHA (H), H.)) MMA(HH H.).
where MHA(-) represents multi-head attention, Hi hyper E Txd is the hyper-modalig, feature. Note that the feature Higper E Txx is random initialized vector.

---

Multimodal Fusion and Prediction. With obtained H3S and H hyyper, a Transformer encoder with classifier at depth of layers employed for multimodal fusion and sentiment prediction:
Cross TrrosTranformee((H H2, H Hygper)s
where i is the sentiment prediction.
3.4 Dominant Modality Correction
This module consists of two steps, i.e. completeness check of dominant modality and proxy dominant feature generation using adversarial learning (Ganin and Lempitsky, 2015; Goodfellow et al., 2014).
Completeness Check. We apply encoder Eec that consists of Transformer encoder with depth of two layers and classifier for completeness check. For example, the missing rate of dominate modality is 0.3, the label of completeness is 0.7. This completeness prediction can be obtained as follows:
w Eec (Concal (Hoes H!)),
(5)
where Hec Rxxx is randomly initialized token for completeness prediction. We optimize this process using L.2 loss:
No Σ A=U
Lee
Ilw
(6)
where N₁ is the number of samples in the training set, uh is the label of completeness of k-th sample.
Proxy Dominant Feature Generation. With the randomly initialized feature H. E gxxd visual feature H2 and audio feature H2 we employ Proxy Dominant Feature Generator EDFG, which consists of two Transformer encoder layers. This setup generates the proxy dominant features H2 E Txx designed to complement and correct the dominant modality. The corrected dominant and the language feature Hi, weighted by the
feature H1 E RTxx calculated by combining H, predicted completeness w:
H EDFG (Concat (HI Ha, H.:) (DFG),
11 (1 w) H1 + H!!
(8)
where PDF denotes the parameters of the Proxy Dominant Feature Generator EDFG-
To ensure that the agent feature offers a distinct perspective from the visual and audio features, we utilize effectiveness discriminator D. This discriminator includes binary classifier and a Gradient Reverse Layer (GRL) (Ganin and Lempitsky, 2015) and is tasked with identifying the origin of agent features:
0p D (H, H H 0p).
(9)
where Op represents the parameters of the effectiveness discriminator D. and Üp indicates the prediction of whether the input feature originates from the language modality.
In practice, the generator and the discriminator engage in an adversarial learning strueture. The discriminator aims to identify whether the features derived from the language modality, while the generator's 's objective is to challenge the discriminator's 's ability to make accurate predictions. This dynamic encapsulated in the adversarial learning objective:
No
min max Ludo @p EDFO
Σ 1 log d0 k=0
(10)
No
where N₂ the number of samples in the training set, and D indicates the label determining whether the input feature for the k-th sample originates from the visual or audio modality.
3.5 Reconstructor
Our experiments demonstrate that reconstructing missing information can significantly enhance regression metrics. More details about this are shown in Table 4. To address this, we have developed
5

---

reconstructor, denoted as Erec, which comprises two Transformer layers designed to effectively rebuild missing information each modality. The operational equation for the reconstructor is:
in = Em (Um)
(11)
where U the reconstructed feature corresponding to the feature U
To optimize the performance of the reconstructor, we apply an L.2 loss function:
Nv
& tro k x
Free
(12)
Nb
m
h=0
where U & and U represent the original and reconstructed features with missing information for the k-th sample, respectively. This loss function helps minimize the discrepancies between the original and reconstructed features, thereby improving the accuracy of other components, such as Dominant Modality Correction and final sentiment prediction.
3.6 Overall Learning Objectives
To sum up, our method involves four learning objectives, including a completeness check loss Leas an adversarial learning loss Lado for proxy dominant feature generation, reconstruction loss Leee and one final sentiment prediction loss Lep- The sentiment prediction loss Lep can be described as:
N. Σ n=0
Sp
11y"
12
(13)
No
Therefore, the overall loss L can be written as:
C aho + 3Ladv + +frre + 8Lxp.
(14)
where a, 3. y and 6 hyperparameters. On MOSI and MOSEI datasets, we empirically them to 0.9, 0.8, 0.1, and 1.0, respectively. On the SIMS dataset, we empirically set them to 0.9, 0.6, 0.1, and 1.0, respectively.
4 Experiments and Analysis
In this section, we provide comprehensive and fair comparison between the proposed LNLN and previous representative MSA methods on MOSI (Zadeh et al., 2016), MOSEI (Zadeh et al., 2018) and SIMS (Yu et al., 2020) datasets.
4.1 Datasets
MOSI. The dataset includes 2,199 multimodal samples, integrating visual, audio, and language modalities. It is divided into training set of 1,284 samples, validation set of 229 samples, and test set of 686 samples. Every single sample has been given sentiment score, varying from -3, indicating strongly negative sentiment, to 3, signifying strongly positive sentiment.
MOSEL. The dataset consists of 22,856 video clips sourced from YouTube. The sample is divided into 16,326 clips for training, 1,871 for validation, and 4,659 for testing. Each clip is labeled with score, ranging from -3, denoting the strongly negative, 3, denoting the strongly positive.
The dataset is a Chinese multimodal sentiment dataset that includes 2,281 video clips sourced from different movies and TV series. It has been partitioned into 1.368 samples for training. 456 for validation, and 457 for testing. Each sample has been manually annotated with a sentiment score ranging from 11 (negative) (positive).
SIMS.
4.2 Evaluation Settings and Criteria
For a fair and comprehensive evaluation, we experiment ten times, setting the missing rates r to predefined values from 0 to 0.9 with increment of O.I. For instance, 50% of the information is

---

randomly erased from each modality in the test data when 0.5. Unlike previous works (Yuan et al., 2021, 2024), we did not evaluate at r 1.0, as this would imply complete data erasure from each modality, rendering the experiment non-informative. With the obtained results of each missing rate, we compute the average value as the model's s overall performance under different levels of noise.
For evaluation criteria, we report the binary classification accuracy (Acc-2), the FI score associated with Acc-2, and the mean absolute error (MAE). For Acc-2, we calculated accuracy and F1 in two ways: negative/positive (left-side value of /) and negativeenon--negative (right-side value of /) on the MOSI and MOSEI datasets, respectively. Additionally, we provide the three-class accuracy (Ace-3), the reven-class accuracy (Acc-7), and the correlation of the model's prediction with humans (Cort) on the MOSI dataset. For the SIMS dataset, we report Acc-3, the five-class accuracy (Acc-5), and Corr. Due to the distinct focus of regression and classification metrics on different aspects of model performance, a model achieving the lowest error on regression metries may not necessarily exhibit optimal performance on classification metrics. To comprehensively reflect the model capabilities, we select the best- pest-performing checkpoint for each type of metric across all models in the comparisons, thus capturing the peak performance of both regression and classification aspects independently.
4.3 Implementation Details
We used PyTorch 2.2.1 to implement the method. The experiments were conducted PC with an AMD EPYC 7513 CPU and an NVIDIA Tesla A40. To ensure consistent and fair comparisons across all methods, we conducted each experiment three times using fixed random seeds of 1111, 1112, and 1113. Details of the hyperparameters are shown in Table 1. In addition, the result of MISA, Self-MM, MMIM, CENET, TETFN, and TFR-Net is reproduced by the authors from open source code in the MMSA (Mao et al., 2022), which is unified framework for MSA, using default hyperparameters. The result of ALMT reproduced by the authors from open source code on Github
Table 1: Hyperparameters of LNLN we use on the different datasets
MOSI
MOSEI
SIMS
Vector Length T
8 128
8
8
Vector Dimension d Batch Size Initial Learning Rate
128
128
64
64
le-4
lc-4
le-4
Loss Weight a, 3, y. 8
0.9, 0.8, 0.1, 1.0 AdamW
0.9, 0.8, 0.1, 1.0 AdamW
0.9. 0.6, 0.1. 1.0 AdamW
Optimizer
Epochs
200
200
200
Warm Up
Cosine Annealing Early Stop
Seed
1111,1112,1113
1111,1112.1113
1111,1112.1113
4.4 Robustness Comparison
Tables and 3 show the robustness evaluation results on the MOSI, MOSEI, and SIMS datasets. As shown in Table 2, LNLN achieves state-of-the-an performance on most metrics, On the MOSI dataset, LNLN achieved a relative improvement of 9.46% on 7cc-7 compared the sub-optimal result obtained by MMIM, demonstrating the robustness of LNLN in the face of different noise effects. However, on the MOSEI dataset, LNLN achieves only sub-optimal performance on metrics such as Acc-7 and Ace-5. After analyzing the data distribution (see Appendix A.3) and the confusion matrix (see Appendix A.6), we believe that this because LNLN does not overly bias the neutral category, which has disproportionate number of samples in noisy scenarios. As shown in Table 3, LNLN achieves an improvement in F1 on the SIMS dataset, with a relative improvement of 99.17% on Fl compared to the sub-optimal result obtained by ALMT. Similar to CENET's performance on the MOSEI dataset, TETFN on the SIMS dataset has a tendency to predict inputs as weakly negative categories with high sample sizes, resulting in seemingly better performance some metrics. Additionally, as shown in Figure 2, we present the performance curves of several advanced
2 MMSA: http:: //githu/ www/tthiaat ar /MSSA
ALMT: https: http:p:://ggabbb....///a/ooooo haa ALMT
7

---

methods under varying missing rates. The results demonstrate that the proposed LNLN consistently outperforms others across most scenarios, showing its robustness under different missing rates.
Overall, LNLN attempts to make predictions the face of highly noisy inputs without the severe lazy behavior observed in other models, which often leads predictions heavily biased towards certain category. This demonstrates that LNLN shows strong robustness and competitivenes. across various datasets and noise levels, highlighting its effectiveness multimodal sentiment analysis.
Table 2: Robustness comparison of overall performance on MOSI and MOSEI datasets. Note: The smaller MAE indicates the better performance.
Method
MOSI
MOSEI
Acc-7
Acc-5
Acc-2
MAR
Corr
Acc-7
Acc-5
Acc-2
F1
MAF
Con
SeIf-MM
34.67
66.60 67.55
1.070
0.512
77.42
72.31
0.695
0.498
MMIM
33.77
67.06
64.044
0.507
73:32
08.72/
0.739
0.489
CENET
30.38
37.25
71.46/67.775
68.41764.85
1.080
0.504
47.18
47.83
74.67 77.34
70.68
74,08
0.685
0.535
TETFN
30.30
34.34
69.76/
65.69
1.087
0.507
30.30
47.70
69.70
67.66
65.69/63.29
1.087
0.508
TeR-Net
29.54
34.67
65.157 66.35
61.73/ /6.066
1,200
0.459
46.83
34.67
73.62 77.23
68.80 71,99
0.697
0.489
ALMT
30.30
33:42
70.40/ 68.
72,57 71.80
1.083
0.498
40.92
41 64
76.64 77.54
77.14 78.03
0.674
0.481
38.27
72.73/71.25
76.30/
77.77
77.77/79.99
0.692
Table 3: Robustness comparison of the overall performance on SIMS dataset. Note: The smaller MAE indicates the better performance.
Method
Acc-5
Acc-3
Ace-2
MAE
Corr
MISA 31.53
56.87
72.71
66.30
0.539
0.348
Self-MM
32.28
56.75
72.81
68.43
0.508
0.376
MMIM
31.81
52.76
69.86
66.21
0.544
0.339
CENET
22.29
53.17
68.13
57.90
0.589
0.107
TETEN
33.42
56,91
73.58
68.67
0.505
0.387
TFR-Net
26.52
52.89
68.13
58.70
0.661
0.169
ALMT
20.00
45.36
69.66
72.76
0.561
0.364
LNLN
34.64
57.14
72.73
79.43
0.514
0.397
COIC
TFL
COKT
16.4
2.0
2.5
Missing Rate?"
Missing Rate?"
Missing Rater
(a)
(b)
(c)
10R0
MAE
USA
MAE
3.0
0.8
5.5
5.5
0.
0.2
044
0.6
Missing Rater
Missing Rater
Missing Rater
(d)
(e)
(f)
Figure 2: Performance curves of various missing rates. (a), (b) and (c) are the Fl curves on MOSI, MOSEL, and SIMS, respectively. (d), (e) and are the MAE curves on MOSI, MOSEI, and SIMS. respectively. Note: The smaller MAE indicates the better performance.

---

4.5 Effects of Different Components
To verify the role of different components, we present the results after subtracting each component on the MOSI dataset separately. As shown in Table 4, some metrics show an upward trend in performance when certain modules are removed. On one hand, we believe this is due to randomness in noisy scenarios. On the other hand, the same hyperparameters cannot make the model perform optimally across all metrics. Moreover, the performance of LNLN decreases relatively slightly when the DMC and reconstructor are removed individually. However, there is significant drop in performance when both are removed, e.g., 4.72% decrease in Acc-7 and 5.87% decrease in Acc-5. Performance drops further after replacing DMML with concatenation fusion, which also proves that DMML plays an important role in MSA. Additionally, after removing the noisy data used for training, the performance of LNLN also shows a significant decrease. This observation indicates that LNLN needs to learn to be aware of noise with the help of noisy data. Overall, both the presence of noisy data and well-designed components are crucial for robust MSA in scenarios with random missing data.
Table Effects of different components. Note: The smaller MAE indicates the better performance.
Method
MOSI
SIMS
Acc-7
Acc-5
Acc-2
Acc-5
Acc-3
Acc-2
P1
LNLN
34,26
38.27
72.55
00555/70 70,94
72.73/71.25
1.046
0,527
34.64
57.14
72.73
79.43
0,514
0.397
wio DMC Reconstmictor
29.54
32.40
71.98
70.31
73.83/21.29
1.104
0.478
33.81
54,00
72.57
78.10
0.525
0.376
DMC
33.94
37.59
72.24/ 21.02
21.45/21.45
1.099
0.522
58.30
73.40
77.58
0.518
w/o Reconstrncoo
33.04
38.52
70.81
71.35
71.17/71.69
1.054
0.518
34.02
58.29
73.07
78.71
0.548
0.351
weo DMML
33.22
37.25
69.55 9355/69/ 69.55
70.81
1.084
0,501
34,87
56.06
70.74
76.94
0.553
0.245
Missing Training
30.11
70.62/
72.49/71.03
1.105
0.486
53.09
79.17
0.519
4.6 Effects of Different Regularization
As shown in Table 5, we present the ablation results after subtracting each regularization on the MOSI dataset separately. Obviously, after each regularization is removed, LNLN shows a decrease in performance on most metrics. This demonstrates that regularization contributes to the learning of the model. Similar to the results in Table 4, there are a few metries that improve after the removal of regularization. We believe this is mainly due to the difficulty of optimizing single set of hyperparameters to achieve the best performance across all metrics.
Table 5: Effects of different regularization. Note: The smaller MAE indicate the better performance.
MOSI
SIMS
Method
Acc-7
Acc-5
Acc-2
F1
MAE
Corr Acc-5
Ace-3
Acc-2
F1
MAE
Corr
LNLN
34.26
38.27
72.55 70.94 72.73
71.225
1.046 0.527 34.64
57.14
72.73
79.43 0.514 0.397
w/o Lee
33.61
37.78
72.32 70.72 72.55
/7.002
1.047 0.531 32.87
56.41
70.78
75.30
0.523 0.376
w/o Lade
32.84
37.79
72.28
70.31 72.33
70.40
1.052
0,526
32.47
54.24
73.80
73.91
0,525 0,351
wlo Lree
33.94
38.52 72.46 / 70.81
72.69
77.17
1.048
0.523
34.33
56.12
71.29
74.73
0.525 0.364
4.7 Effects of Different Modalities
To verify the role of different modalities, we present the ablation results after subtracting each modality from the MOSI dataset. As shown in Table 6, all modalities contribute to the final performance. Notably, when the language modality is removed, the performance of LNLN shows a significant decrease, while removing the visual and audio modalities results in smaller decrease. This indicates that the language modality plays a crucial role in the MSA dataset. Moreover, we observe that LNLN converges most metrics even when the visual and audio modalities are removed. This is due to the model's ' lazy behavior, where LNLN tends to fix its predictions to a certain category. More details can be found in Appendix A.3 and A.6.
4.8 Discussion
In this work, we conduct a comprehensive evaluation of current advanced methods, the results of which have triggered some thoughts. Specifically, we find that all methods fail to accurately predict

---

Table Effects of different modalities, Note: The smaller MAE indicates the better performance.
MOSI
Method
Acc-7
Acc-5
Acc-2
MAE
Corr
LNLN
34.26
38.27
72.55
70.94
72.73 71.25
1.046
0.527
w/o language
16.30
16.31
54.88 / 55.79
60.33 03.33/63.10 63.10
1.399
0,033
w/o audio
33.51
38.10
72.31 / 70.54
73.67 771.00
1.064
0.535
w/o vision
33.53
38.15
72.32/ / 70.58
73.68 71.14
1.064
0.536
the inputs in high missing rate scenarios because most of them are designed specifically for complete data (see Appendix A.6 for more details). However, some methods show relatively more robust performance in low missing rate scenarios and some specific scenarios (see Appendix A.8), such as Self-MM, TETEN, and ALMT. We believe that some of the ideas in these methods may be useful for future research on more robust MSA for incomplete data.
For example, the Unimodal Label Generation Module (ULGM) (Yu et al., 2021) used in both Self- MM and TETEN facilitates the modeling of incomplete data. Due to the random data missing, the temporal and structured affective information in the data is corrupted, making difficult for the models to perceive consistency and variability information between different modalities. Using ULGM to generate pseudo-lahel, as an additional supervisory signal for the models may directly enable the model to learn the relationship between different modalities. For ALMT, we believe that its Adaptive Hyper-modality Learning (AHL) module facilitates the modeling of robust affective representations. AHL mainly uses the dominant modality to compute query vectors and applies multi-head attention to query useful sentiment cues from other modalities as a complement to the dominant modality. This process may reduce the difficulty of multimodal fusion by avoiding the introduction of centiment-irrelevan information to some extent. Our proposed LNLN, based on this idea from ALMT, proves the effectiveness of the method.
Additionally, we explore the performance of these methods in modality missing scenarios, which is a special case of random data missing where the partial modality missing rate = 1.0. We find that when language modality is missing, the performance of many models decreases significantly and converges to the same value. More details can be seen in Appendix A.8.
Conclusion and Future Work
In this paper, a novel Language-dominnted dominated Noise-resistant Learning Network (LNLN) is proposed for robust MSA. Due to the collaboration between Dominate Modality Correction (DMC) module, the Dominant Modality based Multimodal Learning (DMML) module, and the Reconstructor, LNLN enhance the dominant modality achieve superior performance in data missing scenarios with varying levels. Extensive evaluation shows that none of the existing methods can effectively model the data with high missing rates. The related analyses better handle robust MSA. In the future, we will focus on improving the generalization of the model handle different types of scenes and varying intensities of noise effectively.
provide suggestions for other researchers to
Limitations
We believe there are several limitations to the LNLN. 1) The LNLN achieves good performance in data missing scenarios, but not always better than all other methods the modality missing scenarios, which demonstrates the lack of multi-scene generalization capability of the LNLN. 2) The data real-world scenarios much more complex. In addition the presence of missing data, other factors need to be considered, such as diverse cultural contexts, varying user behavior patterns, and the influence of platform-specific features on the data. These factors can introduce additional noise and variability, which may require further model adaptation and tuning to handle effectively. 3) Tuning the hyperparameters, particularly those related the loss functions, can challenging and may require more sophisticated methods to achieve optimal performance.
10

---

Acknowledgements
This work is supported by the Guangdong Provincial Key Laboratory of Mathematical Foundations for Artificial Intelligence (202381212010001,
References
Baltrusaitis, T., Zadeh, A., Lim, Y.C., Morency, L., 2018. Openface 2.0: Facial behavior analysis toolkit, in: 2016 IEEE Winter Conference on Applications of Computer Vision, pp. 59-66.
Ganin, Y., Lempitsky, V.S., 2015. Unsupervised domain adaptation by backpropagation, in: Interna- tional Conference on Machine Learning, pp. 1180-1189.
Goodfellow, IJJ., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A.C.,
Bengio, Y., 2014. Generative adversarial nets, in: Advances in Neural Information Processing Systems, pp. 2672-2680.
Guo. J., Tang. J., Dai, W., Ding, Y., Kong, W., 2022. Dynamically adjust word representations using unaligned multimodal information, in: Proceedings the 30th ACM International Conference on Multimedia, pp. 3394-3402.
Han, W., Chen, H., Poria, S., 2021. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis, in: Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pp. 9190-9192.
Hazarika, D., Zimmermann, R., Poria, S., 2020. MISA: modality-invariian and -specific representa- tions for multimodal sentiment analysis, in: Proceedings the 28th ACM International Conference on Multimedia, pp. 1122-1131 131.
Jiang. Y., Li, W. Hossain, M.S., Chen, M., Alelaiwi, A., Al-Hammadi, M., 2020. A snapshot research and implementation of multimodal information fusion for data-driven emotion recognition. Information Fusion 53, 209-221.
Kenton, W.M.C.C.., Toutanova, I.K.,, 2019. Bert: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of NAACL-HLT, pp. 4171-4186.
Li, M., Yang, D., Lei. Y., Wang, S., Wang, S., Su, L., Yang. K., Wang, Y.. Sun, M., Zhang, L., 2024. A unified self-distillation framework for multimodal sentiment analysis with uncertain missing modalities, Proceedings of the AAAI Conference on Artificial Intelligence, pp. 10074-10082.
Liang, J., Li, R., Jin, Q., 2020. Semi-supervised multi-modal emotion recognition with cross-modal distribution matching, in: Proceedings of the 28th ACM International Conference Multimedia, pp. 2852-2861.
F., Chen, X., Huang, Y., Duan, L., Lin, G., 2021. Progressive modality reinforcement for human multimodal emotion recognition from unaligned multimodal sequences, in: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2554-2562.
Mai, S., Xing, S., Hu, H., 2020. Locally confined modality fusion network with a global perspective for multimodal human affective computing. IEEE Transactions on Multimedia 22, 122-137.
Mao, H., Yuan, Z., Xu, H., Yu, W., Liu, Y., Gao, K., 2022. M-SENA: an integrated platform for multimodal sentiment analysis, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 204-213.
McFee, B., Raffel, C., Liang D., Ellis, D.P.W., McVicar, M.. Battenberg, E., Nieto, O., 2015. librosa: Audio and music signal analysis in python, in: Proceedings of the 14th Python in Science Conference, pp. 18-24.
Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., Manocha, D., 2020. M3ER: multiplicative multimodal emotion recognition using facial, textual, and speech cues, Proceedings of the AAAI Conference Artificial Intelligence, pp. 1359-1367.

---

Rahman, W. Hasan, M.K., Lee, S., Zadeh, A.B.. Mao, C., Morency, L., Hoque, M.E., 2020. Integrating multimodal information in large pretrained transformers, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2359-2366
Tsai, Y.H., Bai, S., Liang, P.P.. Kolter, J.Z., Morency, L., Salakhutdinov, R., 2019. Multimodal transformer for unaligned multimodal language sequences, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6558-6569.
Yang, D., Huang, S., Kuang, H., Du, Y., Zhang, L., 2022. Disentangled representation learning for multimodal emotion recognition, in: Proceedings of the 30th ACM International Conference on Multimedia, pp. 1642-1651.
Yu, W. Xu, H., Meng, Zhu, Y., Ma, Y., Wu, J., Zou, Yang, K., 2020. CH-SIMS: chinese multimodal sentiment analysis dataset with fme-grained annotation of modality, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3718-33 3727.
Yu, W., Xu, H., Yuan, Z., Wu, J., 2021. Learning modality-specifee representations with self- supervised multi-task learning for multimodal sentiment analysis, in: Proceedings of the AAAI conference on artificial intelligence, pp. 10790-10797.
Yuan, Z., Li, W., Xu, H., Yu, W., 2021. Transformer-basee feature reconstruction network for robust multimodal sentiment analysis, in: Proceedings of the 29th ACM International Conference Multimedia, pp. 4400-4407
Yuan, Z., Liu, Y., Xu, H., Gao, K., 2024. Noise imitation based adversarial training for robust multimodal sentiment analysis. IEEE Transactions Multimedia 26, 529-539.
Zadeh, A., Chen, M., Poria, S., Cambria, E., Morency, L., 2017. Tensor fusion network for multimodal sentiment analysis, in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1103-1114.
Zadeh, A., Liang, P.P., Poria, S., Cambria, E., Morency, L., 2018. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph, in: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 2236-2246.2
Zadeh, A., Zellers, R., Pincus, E., Morency, L., 2016. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems 31, 82-88.
Zhang, H., Wang, Y., Yin, G., Liu, K., Liu, Y., Yu, T., 2023. Learning language-guin guided adaptive hyper-modality representation for multimodal sentiment analysis, in: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 756-767.
12

---

A Additional Experiments and Analysis
A.1 Effect of Regularization Weight on Model Performance
As shown in Table 7, we empirically tried different combinations of hyperparameters in the loss func- tion. The results show that the selected parameters make the LNLN achieve better performance most metrics. This also demonstrates effectiveness of the optimization objectives of LNLN.
Table 7: Effect of regularization weight on model performance. Note: The smaller MAE indicate the better performance.
MOSI
a, 7, 8
Acc-7
Acc-5
Acc-2
F1
MAE
Corr
0.9, 0.8, 0.1, 1.0
34.26
38.27
70.94
/7.22
1.046
0.527
0.9, 0.8, 0, 1.0
33.94
38.52
72.46
70.81
72.69/71.1
1.148
0.523
0.8, 1.0
32.76
37.33
72.20/700 70.69
72.43/71.2
1.078
0.511
0.9, 0.1,
32.84
37.79
72.28/70.32 70.31
72.83/0..40
70.40
1.052
0.526
0.9, 0.1
32.43
37.23
70.04/70.24
72.59
70.96
1.063
0.511
0.9, 1.0, 0.1,
32.24
37.07
71.61
70.72
73.04
70.89
1.063
0.518
0, 0.8, 1.0
33.61
37.78
72.32 70.72
72.55
71.02
1.047
0.531
0.5, 0.1,
33.29
37.78
72.40 70.95
72.61
71.29
1.067
0.515
1.0, 0.8, 1.0
33.22
36.64
71.66 / 70.57
72.94 71.80
1.067
0.511
SIMS
a. 7,8,y,0 7,
Acc-5
Acc-3
Acc-2
F1
MAE
Corr
0.9, 0.6, 0.1, 1.0
34.64
57.14
72.73
79.43
0.514
0.397
0.9, 0.6, 1.0
32.87
56.41
70.78
75.30
0.523
0.376
0.9, 1.2
33.95
55.11
70.82
73.20
0.517
0.390
0,9,0,01,,1.0
32.47
54.24
73.80
73.91
0.525
0.351
0.9, 0.0
35.16
56.57
72.86
77.54
0.524
0.358
0.9, 1.0,
32.92
54.66
72.50
76.94
0.543
0.363
0, 0.6, 1.0
34.33
56.12
71.29
74.73
0.525
0.364
0.5, 0.6, 1.0
33.96
55.57
70.63
73.06
0.539
0.360
1.0, 0.6, 1.0
33.25
57.03
70.57
72.20
0.522
0.367
A.2 Analysis of Model Stability
To evaluate the stability of the model, selected the MOSI dataset for ablation experiments. Specifically, based on the overall performance (mean) Table 2, we selected several representative methods to additionally compute the standard deviation. For each method, standard deviation of evaluation results for the three random seeds when the missing rate r ranges from 0 to 0.9, and then average the 10 sets of standard deviation. The final result is shown in Table 8. We can see that the standard deviation these methods is not very large in most metrics, indicating that all these methods can guarantee stability in the presence of random noise. It should be noted that LNLN strikes a balance between overall performance and stability.
first calculate the
A.3 Analysis of Data Distribution
Figure 4 illustrates the data distribution on the MOSI, MOSEI, and SIMS datasets. Significant category imbalances can be seen across all datasets. Additionally, distributions of the training set, validation set, and test set not identical on the MOSI and MOSEI datasets. For example, as shown in Figure (a), the percentage of weakly negative samples 15.5% the MOSI training set, while this category reaches 22.7% in the test set, representing the highest percentage among all categories.
13

---

Table Comparison of model stability on MOSI dataset. Note: - means mean The smaller MAE indicates better performance.
Method
Acc-2
MISA
29.85+22.22
33.08+1.77
71.49:1000/ 70.33=0.93
71.28=00.2/ 70.00=1.26
1088::00008
0.524:10.08
SelF-MM
29.55:::.06
34.674:1.87
70.5111,71
69.26.11.07
66.60= /6222/67.252 67.54:12..34
1000:10.13
0,512:+0.07
CENET
30.38:1..27
17.25+
71.4000.00/ 67.73=0.71
68.41= 64.85:22.56
10800+0.14
0,504=0.11
TFR Net
29.54.11.00
34.67.11.75
68.15111.25 66.355 1.09
61.73 60.06.12.33
1.200.10.11
0.459.10.25
LNLN
34.26::1.17
38.27:::.23
72.55:1.117009941..
72.73=0.99 0177::0..077225511.00 71.25:::1066
1.046:::21
0.527:10.17
Similarly, Figure 4 (b) and Figure 4 (c) show that this phenomenon also exists in the MOSEI and SIMS datasets.
The unbalanced data distribution makes it difficult for the model to perform well in data missing scenarios. Specifically, is easy for the model engage in lazy behavior in high-noise scenarios, simply biasing predictions towards categories with a higher proportion in the training set. These unbalanced distributions may further increase the learning difficulty of the model when facing data missing. More details can be found in Appendix A.6.
A.4
Case Study
As shown in Figure 3, visualize several successful and failed predictions made by LNLN and ALMT from the MOSI dataset for the case study. shows that LNLN can perceive sentiment cues in challenging samples, which demonstrates its ability capture sentiment cues in noisy scenarios. However, for inputs with high missing rates (e.g., the third example in the figure). both LNLN and ALMT fail to make correct predictions. We believe this is due to the high loss of valid information in the multimodal input, making accurate predictions difficult.
Input
LOVE THEY ALL THE THE WAR SCENES ALL THE MILITARY AND SHIT
TERRIFIED THROUGH PART OP THIS ORIGINAT. VERSION IS TERRIFY ING
AND WAS STILL BORING
ALMT
Positive x
Positive x
Negative
LNLN
Negative //
Negative
Negative
The input is visualized to facilitate readers' understanding. practice, random data missing applied to the original input sequence, as described in Section 3.
Figure 3: Visualization of successful and failed predictions. Note:
A.5 Details of Robust Comparison
Tables and 10 show the details of robust comparison on the MOSI, MOSEI, and SIMS datasets, respectively. We observe that Self-MM and TETFN have a clear advantage in many evaluation metrics when the missing rate is low. As the missing rate increases, LNLN shows significant improvement all evaluation metrics. This demonstrates LNLN's ability to effectively model data affected by noise of varying intensity. In general, models augmented with noisy data in training usually cannot achieve state-of-ther -ftthe-a-th--rt performance when the noise low due to differences in data distribution. It is also worthwhile to study in the future how to balance the performance of models under different levels of noise.
In addition, when is high, many models converge in their performance on metrics such as Acc-7 and Acc-5 (see Appendix A.6). This is due the fact that these models exhibit lazy behavior, tending to predict inputs categories with higher number of samples in the training set. In such cases,
14

---

believe that the models do not actually learn effective knowledge, even if their performance appears good.
Training Set
Validation set
Test set
Negative
Negative
Weakly Negative
Neutral
Weakly Negative
Negative
Weakly Negative
17.%
25.7%
15.5%
Strongly
Strougly Negative
Neutral
Strongly
Negarive
Negalive
2.%
Stoongly Postive
Strongly
Strongly Positive
11.9
Positive
17.9%
16.%
Positive
Neutral
Weakly Positive
Positive
Weakly Positive
Weakly Positive
Positive
Data distribution of MOSI dataset
Negative
Neutral
Negative
Neutral
Negative
Weakly Negative
Weakly Negative
Weakly
Negarive
Neutral
158
Negative
Scounlyy
5.0%
Negative
Strongly
Negative
Strongy
Strongly Pooltive
Strongly Positive
Stroggy Positive
Weakly Positive
Weakly Positive
Weakly
Positive
Positive
Positive
Positive
(b) Data distribution of MOSEI dataset
Weakly Negative
Weakly Negative
Weakly Negative
Negative
Negative
Negative
22.3
15.2%
15.2%
Weakly Positive
11.1%
Weakly Positive
Weukly Positive
Neutral
Neutral
Neutral
Positive
Positive
Positive
(c) Data distribution of SIMS dataset
Figure 4: Data distribution of MOSI, MOSEL, and SIMS datasets.
A.6 Visualization Confusion Matrix
To verify the effectiveness of the method, we visualized the confusion matrix of several representative methods on the MOSI, MOSEI, and SIMS datasets in Figure Figure 6, and Figure 7, respectively. Obviously, as the missing rate increases, the predictions of all methods on all datasets tend to favor certain categories. This phenomenon indicates that the models hardly learn useful knowledge for prediction but instead ensure high accuracy by being lazy. For example, the MOSI dataset,
15

---

Table 9: Details of robust comparison on MOSI and MOSEI with different random missing rates. Note: The smaller MAE indicates the better performance.
Mathod
MOSI
MOSUI
Asci5
Ase-T
Auc-5
Random Missing Rate
MISA
43.05
50
52.78
0:371
0.777
51.79
53.85
85.10-83.75
83.75
0.552
0.750
Sal-MM
42.51
52.38
85.22
83.24
85.19 M8.26
0.720
0.730
$3.80
55.72
85.34/84.88
85.11 1 (884
0.531
0.764
MMIM
45.92
02.85
53.43/81
83.45 (81
0.744
0.778
50.76
53.04
83.53/811 65
83.30 881.41
0.575
0.724
TETIN
43.00
51.31
$2,02
1.10
82.67
0.719
0.794
48.07
55.30
42.62/8111 10
82.67 /61.09
0.719
0.791
40.83
87.91
$3.64781 68
83.57/8100 61.61
0,803
0.760
53.71
47.91
38.96 79.675
85.71 644.54
0.550
0.245
ALMI
42.37
54.91
85.01 (8.01/2..0
0.752
0.788
52.18
95.62/900
0.542
0.152
Racdom Missing Ratc
0-3
MISA
40:28
06.2
50.15 01
80.21/ 97
0.841
0.721
50.15
51.34
82.21 1821/8233
81.25 11228.003
0.598
0.222
SeIEMM
40:33
89.03
$1.40 88083
91. 19.19:55 80L03
0.811
0.228
51.53 1.30
53.15 53:15
8305/85.00
82.03.100
0.564
0.725
79.
0.715
190.5
CENET
79.91
0.537
03.75180
0.556
0.739
TETEN
40.67
06.84
50.59
735559775440 75.91
80.55 76.
0.845
0.731
-00.5
54.25
S1.59 75 91
50.55
75.75
0.905
0.731
TER-N
38.63
45.55
79.27 77.59
78.70 77.61
0.572
0.705
82.921 63.31
0.573
0.715
15/
INEN
42.37
$1.20 78845
79.00 79.04
0.820
0.724
51.25
83.32185
83.65
0.712
Random Missing Rato
0.2
MISA
36.25
41.55
17.54 77.54/76.34
17.58/ 76.30
0.539
0.654
47.24
47.66
77.84/7000
75.56 76.88
0.659
0.674
SREMI
36.54
63.95
78.15/76.48 76.48
12.76 76.51
01901
0.060
49.44
50.51
30.56: 93.35
79.76 0877-88111 88.17
0.604
0.578
GENET
18.00
43.3
17 49.4111
64
77
74.32 56
0.96
0.644
50.72
51. 85
81. 18.1681160 62
00.28.1 81.11
0.590
0.698
TETEN
15.81
41.79
77
60
72 85
0.10
0.677
15.81
$2.35
77.49
60
22 35
0.10
0.53
0.987
0.773
ALMT
40.33
75.70
0.645
82.71/811
51.83
0.607
LNEN
39.74
45.14
79,22/ 76087
79.53/
77.34
0.891
0.668
48.75
19.95
31.70/31.65
81.95 80.89
0.636
0.677
Missing Rate
MISA
$4.00
33.97
15.76 54
25.82/744 74.51
0,980
0.61%
43.09
43.40
13.32/72 28
68.91 7 22.225
0.724
0.615
48.00
75.69
0.110
MMIM
36.83
90.43
14.08
0.944
0.612
17.08
79.79
0.640
CENET
34.74
33.97
76.83/72.01 72.01
76.56 71.30
0.983
0.605
48.49
19.37
79.05
90.02
77.34 24447 78 94
0.635
0.640
TETTN
33:24
38.56
75.25 73.42
16.77 72.27222
0.982
0.607
33:28
75.25
23.42
25.77
0.932
0.600
ALMI
12.93
15
73.65
0.992
0.345
44.05
$0.90
79
0.632
0.588
INEN
30.00
42.01
77.29
75.46
77.36
75.68
IL953
0.617
47.36
48.40
30.11 M0.45
80.44
0.655
0.620
0.4
MISA
32.65
35.3 37
13.89
15.88/ 22.8/72.48
1.011
0.585
40.87
19.53
10.46
64.02
67.91
0.790
0.561
MMIM
33.38
35. 76
70.84 168 90
69 00699105 17.806 60
1.684
0.576
-01.84
41.86
14.56 16.15
71.98 1118-721 71.40
0.533
0.542
CENET
32.56
36.15
73.38 11.55
72.551 26
1,531
0.574
47.12
41.74
1608/78.55 78.57
23897 75
0.678
0.587
0.571
TER Net
10.17
35
78.6 73.65
0.685
0.565
ALMI
11:44
35.0 03
73. 13.12/2115
35 72.47
1,855
0.390
40.40
41.21
79.407 79 16
7866 79.50
0.681
0.5%6
INEN
36.49
41.11
76.01
76.31 76.67
0.987
0.394
45.99
46.83
76.49
79.70
78.95
89.
0.673
0.392
Randon Missing Ratc
0.55
56
SOK-MM
26.97
31.5 1.999
67.43162
66 (6.8)
1.129
0:503
42.70
43.
31.97
75.81
67 00.00
0.73
0.473
MMIM
28.23
29.89
68.09 08.336 52
66 151666453
1.12%
0.501
18.68
19.3
31.58 45
67. 67.20161
08725
0.370
GENET
26.46/10
71 10/66:90 50
0.495
48.12
69 19.011
0.20
0.15
31.54
1.157
67
TER-NC
25.55
30.71
64.33 63.6663555 62
58.04 03.00/550 56.64
1.270
0.443
45.00
30.71
71.55 75.69
66388 0 70.07
03550
0.471
ALMT
26.42
31.25
08.24 65.94
69.74 66.54
1.138
0.485
37.82
77.401 77.48
77.25 77.25/77.00 77.80
0.683
01361
33.92
1
0.536
45.59
77.23
0.710
Randum Missing Rain
0.6
MISA
24.66
27.12
66.91/65 66.69
0.4-1
65.55
$4.64
0.875
0.415
SOIMMM
24.34
23.31
63.471
63.3166330 36
58.96 (3.91/62.17 62.07
1.209
0.425
41.47
41.75
09.33/ 23 95
63311 165.70
0.762
0.301
MMIM
37:1
0.41%
63.00
0.808
0.40
TETEN
25.12
27 99
63.0
161116133 03
58 56
1.238
0.417
15.12
44.00
63.4/66123 25
58 56.05
13238
0.17
TER-N
34.05
25. 53
61
57.47
32 44 50
1.371
0.363
43.88
78.
68301 05
62 197.07
0.760
0.300
78.71
0.710
INEN
30.37
34.35
67.99
1.147
0.458
44,00
75.03
0.471
Randon: Missing Rato
0.7
MISA
21.14
23.27
65.09 /63
65.07 /6377/63.4
1.257
0.351
34.54
31.21
64.38/3
$1.83 1022/6000 69.65
0.996
0.344
Self MM
20.70
31.81
61.71161
35.11 58.97
1,271
0.339
39.93
40.12
66.791
58.05 663
0.786
0.325
CENET
22.35
39 85
59.45
53 79
1.369
0.335
43.83
44.00
62.50
59
0.776
0.384
TETPN
18:15
35
63 65
31 7 90
0.357
43.15
61 65
33.70 20
90.75
1.201
0.330
36.92
0:276
56
0.512
ALMI
54.95
0.545
INEN
27.79
31.19
65.85
65.01
6595/66
1.219
0.383
42.32
43.56
71.55
11.51/744
74.74
73.49
77.49/777.00
0.762
0.408
Missing Rate
MISA
19.02
20.99
63.56 62.24
63 66166/6667
1.311
0.321
$3.20
20.51
63.43/ 30
49.05 59..00
0.927
0.267
65.07 71
MMIM
20.26
21.77
01287
33.64
54.7666.44
01858
0.260
CENET
21.14
21.67
M191/5755 57
31.81
1:314
0.274
42.71
42.74
65 38.3813
16
5630
69.00
01795
0.316
TETUN
22.01
40/
48 18757
1.337
0.274
22.01
42.11
48.
45.50
1.357
0.274
ALMI
21 65
50 37
65-45 66.14
L3100
0.223
38.01
$4.09
72
0.774
0121)
INEN
26.34
28.23
62.11
62
1.283
0.314
40.76
40.97
68.62/ 72.86
71.83 76.90
0.791
0.335
Random Missing Raic
0.9
MISA
12:08
64 21
56 86 56.
0.226
12.19
62.957
48.90 000010000
0.961
0.180
MMIM
18.95
19.53
55.20 1111111555 51.65
47.35 40.39
1.357
0.136
32.61
32.67
63.69111 10
$1.26 59.99
0.877
0.197
CENET
19.15
19.10
58.99
3001 46.55
1.357
0.181
42.08
42.
64.14110 70.42
34.27 192.33
0.814
0.254
TETPN
20:75
11.67
25.12
43.01 40.15
1.534
0.155
71.34
52.02 19.99
01820
0.175
ALMI
30.31
30.50
$7.32/ 56.66
6492/67.22 67.82
1.399
0.205
34,40
$4,40
61.41 03.65
63.32 19.83
0.019
0.136
22.98
56.30/ 56.51
36 12/7/5644
1.349
0.003
40.10
40.19
6483/ 71.51
70.60
0.30
0731
16

---

Table 10: Details of robust comparison on SIMS with different random missing rates, Note: The smaller MAE indicates the better performance.
Method
5cc-5
Acc.3
Acc-2
F1
MAR
Corr
Method
Acc-5
Acc-3
Acc-2
F1
MAE
Corr
Random Missing Rate
r
Random Missing Rate r
0.5
MISA
40.55
63.38
78.19
77.22
0.449
0.576
MISA
30.56
54.78
71.26
64.16
0.552
0.367
Self-MM
40.77
64.92
78.26
78.00
0.421
0.584
Self-MM
32.02
53.90
71.41
67.11
0.517
0.390
MMIM
37.42
60.69
75.42
73.10
0.475
0,528
MMIM
33.41
52.37
68.49
64.81
0.553
0.336
CENET
23.85
54.05
68.71
57.82
0.578
0.137
CENET
23.12
54.05
68.71
57.92
0.588
0.107
TETEN
41.94
65.86
80.23
79.25
0.424
0.589
TETFN
33.48
56,24
72.43
67.30
0.512
0.394
TFR-Net
33.85
54.12
69.15
58.44
0,562
0.254
TeR-Net
24.65
52.37
67.47
58.66
0.685
0.171
ALMT
23.41
54.78
75.64
76.27
ALMT
47.12
68.27
71.22
0.395
LNLN
38,66
63.97
75,93
79.89
0.458
0,570
LNLN
36.40
57.70
72.72
78.77
0,513
0.412
Random Missing Rate
0.1
Random Missing Rate r
0.6
MISA
38.88
63.02
77.39
75.82
0.461
0.561
MISA
27.72
53.97
70.46
61.81
0.578
0.286
Self-MM
40.26
63.53
77.32
76.76
0.433
0.563
Self-MM
29.10
51.86
70.02
64.21
0,548
0.313
MMIM
37.27
60.90
74.25
72.08
0.473
0.529
MMIM
29.18
49.31
67.91
63.86
0.578
0.270
CENET
22.83
53.98
68.57
57.36
0.580
0.136
CENET
22.46
53.69
69.00
58.64
0.592
0,102
TETEN
41.36
64.62
78.92
77.70
0.432
0.578
TETFN
29.90
53.54
70.97
64.19
0.545
0,309
TFR-Net
30.12
53.25
68.85
59.38
0.596
0,203
TeR-Net
24.80
52.59
67.03
58.30
0.696
0.157
ALMT
22.10
55.14
74,40
75.19
0.530
0.537
ALMT
18.67
43.69
66.81
70.69
0.574
0.322
LNLN
38.51
62.73
76,29
80.07
0.458
0,562
LNLN
33,70
54.63
71.55
79.56
0.535
0.352
Random Missing Rate
0.2
Random Missing Rate
0.7
MISA
38.15
59,23
74,33
71.30
0.489
0.490
MISA
24.87
52.52
69.95
59.54
0.601
0.167
Mc
38.37
61.71
74.98
73.71
0.464
0,500
Self-MM
25.53
50.62
69.58
62.28
0.571
0.198
MMIM
37.27
57.33
72.36
69.80
0,504
0,460
MMIM
28.59
46.53
66.89
62.23
0.595
0.190
CENET
22.25
54,20
68.57
57.64
0.583
0.132
CENET
21.81
53.32
67.69
57.87
0.599
0.070
TETEN
39.46
61.56
75.49
73.59
0.457
0.527
TETEN
27.86
51.06
69.29
61.09
0,572
0.190
TFR-Net
29.03
53.61
68.64
59.74
0.619
0.191
TeR-Net
23.78
52.30
67.18
58.15
0.707
0.163
ALMT
21.08
53.17
72.65
73.90
0.541
0.485
ALMT
18.02
38.66
65.57
70.27
0.586
0.218
LNLN
38.88
61.78
74.76
78.53
0.474
0,513
LNLN
30.27
51.64
69.73
79.37
0.558
0,261
Random Missing Rate
0.3
Random Missing Rate
0.8
MISA
36.40
59.30
74.11
70.40
0.505
0.464
MISA
22.69
52.22
69.37
57.82
0.610
0.092
Self-MM
37.93
59.81
74.76
72.85
0,487
SelF-M
50.77
69.51
60.68
0.138
MMIM
37.71
58.06
72.36
69.52
0.512
0.436
MMIM
22.32
44.35
65.28
60.53
0.607
0.145
CENET
21.44
54.05
68.42
$7.41
0.578
0.175
CENET
21.73
52.15
67.47
58.44
0.599
0.074
TETEN
38.80
61.92
75.86
73.28
0.463
0.521
TEIEN
23.48
49.60
69.88
60.92
0.584
0.154
TFR-Net
27.64
52.30
68.42
59.88
0.640
0,182
TeR-Net
22.97
52.74
67.54
57.55
0.721
0,100
ALMT
20.35
50.62
72.06
73.64
0.546
0.469
ALMT
18.60
34.06
64.19
69.64
0.597
0,133
LNLN
38.37
60.98
74.25
78.60
0.478
0.509
LNLN
27.94
50.47
69.58
80.23
0.580
0,183
Random Missing Rate
0.4
Random Missing Rate
0.9
MISA
34,86
57.33
72.87
67.52
01,23
0.436
MISA
20.64
52.95
69.22
$7.01
(1617
0.041
Self-MM
34.57
58.28
73.30
70.36
0.482
0.479
Self-MM
22.17
52.15
68.92
58.32
0.586
0.111
MMIM
34.57
55.36
69.95
66.49
0.533
0.399
MMIM
20.35
42.67
65.72
59.64
0.610
0.096
CENET
22.54
54.12
68.49
$7.68
0.583
0.141
CENET
20.86
48.07
65.72
58.18
0.609
-0.002
TETEN
35.81
58.93
70.66
0.473
0.504
TETEN
22.10
45.73
68.92
58.75
0.108
TFR-Net
25.31
51.86
67.91
59.16
0.664
0,176
TeR-Net
23.05
53.76
69.08
57.71
0.721
0.088
ALMT
19.91
49.45
70.75
72.97
0.549
0.470
ALMT
19.47
26.91
66.23
73.76
0.596
0.076
LNLN
37.49
60.03
73.81
78.82
0.491
0.481
LNLN
26.19
47.48
68.64
80.42
0.591
0.127
Self-MM predicts all samples weak negative when the missing rate is 0.9. Although the accuracy of Self-MM is high when the missing rate r is high, it does not demonstrate that the model is more robust.
In addition, TETFN and -eR-Net tend to randomly predict within the negative and weak negative categories under high noise, indicating that their lazy behavior is not as severe as methods like Sclf-MM, MMIM, and MISA. Unlike other methods, although LNLN also suffers from the impact of data missing, the model's lazy behavior is less severe. For example, at missing rate of 0.9, the model is still attempting to predict the samples, and there is no case of predictions favoring a particular category an extreme extent.
From Figure and Figure 7, we can see that a similar phenomenon also occurs in the MOSEI and SIMS datasets. These observations demonstrate that methods targeting complete data often fail when modeling highly incomplete data. Therefore, the design of methods specifically for modeling incomplete data is necessary.
17

---

MISA
2.11
CENET
CCT
3/20
6C1
Self-MI
CCN
TETEN
211
TFR-Net-
24
2:41
ALMT
427
CC2
LNLN
2.0
AXT
missing roto 00
missing roto 0.1
missing rote
0.5
missing rato y 0.9
Figure 5: Seven-category confusion matrix of several representative methods on MOSI dataset. Note: 0-6 denote strongly negative, weakly negative, negative, neutral, weakly positive, positive, and strongly positive, respectively.
18

---

3.79
MISA
0:10
CENET
641
0:54
CC1
Self-MM
4:24
LKW
4.1
CCN
TETEN
6:34
6:24
1.00
TeR-Net-
CCD
013
ALMT
4N0
5:00
6:14
221
0C15
LNLN
6:007
CACO
4:00
70
4:1
4:M
100
440
missin mite
missing rate
0.1
missing
mit
missing Tato
-0.9
Figure 6: Seven-category confusion matrix of several representative methods on MOSEI dataset. Note: 0-6 denote strongly negative, weakly negative, negative, neutral, weakly positive, positive, and strongly positive, respectively.
19

---

MISA
8
6444
1.01
5004
300
CENET
1.04
50-3
6407
6.14
9212
CC1
Self-MM
TETFN
640
327
6.44
TFR-Net-
32
CGI
3M
5:11
A LLM
AMT
2.50
1211
LNLN
missing rater
missing rate
missing rate
missing rabe
Figure 7: Five-category confusion matrix of several representative methods on SIMS dataset. Note:
0-4 denote weakly negative, negative, neutral, weakly positive, and positive, respectively.
20

---

A.7 Visualization of Representations
Figure shows the language feature Hi extracted without applying data missing and the corrected dominant feature H. with different data missing rates.
that the model tends learn H. which is consistent with the distribution of the H! indicating that LNLN can complement the information of the language modality when data missing. Additionally, when the missing rate is 1.0 (no valid information for the input), it becomes difficult for the model learn H3 that is consistent with the language feature. This also suggests that the generator indeed trying to learn H3 for fusion in DMML, rather than just guessing.
vortanng
missing
missing 0.2
missing 0.4
R5 my w/l Ainiin
(d) missing rate r 0.6
(c) missing rate r 0.8
missing rate r 1.0
Figure 8: Visualization of the corrected dominant feature H3 with diffcrent data missing rate and the language feature H. without data missing.
A.8 Analysis of the generalization to modality missing scenarios
To explore the generalization of the method, further evaluated it in the special case of data missing, ie.. modality missing scenarios. The overall performance of the model shown in Table 11 and Table 12. The detailed performance of the model is shown in Table 13 and Table 14. It is evident that LNLN, ALMT, TETEN, and Self-MM all exhibit excellent generalization.
We believe that the Dominant Modality Correction (DMC) proposed in LNLN, the Adaptive Hyper- modality Learning (AHL.) proposed in ALMT,, and the Unimodal Label Generation Module (UL.GM) proposed/used in Self-MM and TETEN deserve more research attention in the future. These modules and their corresponding ideas have significant reference value for achieving robust Multimodal Sentiment Analysis (MSA).
However, we also found that many methods converge the same value when the noise is high. This is similar to the phenomenon observed in previous data missing scenarios, where many models exhibited lazy behavior. Therefore, handling modality missing and improving model robustness in different noise scenarios remain critical challenges that need to be addressed in the field of MSA.
21

---

Table 11: Generalization comparison of the overall performance MOSI and MOSEI with random modality missing. Note: the parameters used for evaluation are consistent with those used for testing random data missing. The smaller MAE indicates better performance.
Method
MOSI
MOSEI
Acc-7
Acc-5
Acc-2
MAE
Corr
Acc-7
Acc-5
Acc-2
F1
MAE
Cor
MISA
29.44
31:76
07.62/67.27
64.81 / 66.30
1.096
0.461
41.45
40.32
73.94 / 22.49
66.65/
71.30
0.752
0.438
SeIf.MM
29.74
36.40
71.45 / 68.17
63.71 /61.91
1,052
0.450
45.17
46.05
74.12
77.85
67.00/71.81
0.684
0.447
MMIM
31.28
33.74
68.34
34334455. 66.12
63.38/550
758.88
1.058
0.444
41.36
42.53
73.19
/ 76.14
68.11 / 70.18
0.732
0.448
CENET
30.30
33.94
70.37/
66.67
02.66/58.6
1.068
0.450
47.41
48.19
73.45
74.14 74.14
68.43/ 70.37
0.684
0.480
TETFN
32.66
36.26
70.17/ 68.15
62.47/60.30
60.18
1.061
0.446
47.52
48.41
73.87
77.43
66.70/ 71.45
0.691
0.426
TFR.Net
29.76
34.96
70.49
67.95
62.87
60.08
0.451
47.33
48 08
73.50 / 77.45
66.89
71.28
0.700
0.427
ALMT
LNLN
31.93
34.86
68.53 65.64
72.01 69.65
1.093
0.423
43.85
44.50
73.52/77.04
80.85 / 83.19
0,733
0.425
Table 12: Generalization comparison of the overall performance on SIMS dataset with random modality missing. Note: the parameters used for evaluation consistent with those used for testing in random data missing. The smaller MAE indicate the better performance.
Method
Acc-5
Acc-3
Ace-2
MAE
Corr
MISA
29.62
56,82
73.61
66.32
0.535
0.330
Self-MM
30.36
59.56
75.01
73.13
0.508
0.321
MMIM
27.21
50.79
71.52
65.48
0.546
0,286
CENET
19.95
43.63
64.08
56.96
0.601
0.033
TETFN
30.61
50,98
74.77
68.00
0.514
0.317
TFR-Net
24.69
52.57
69.31
58.03
0.629
0.133
ALMT
21.82
39,39
72.30
78.86
0.563
0,287
LNLN
30.39
54.32
72.67
80.87
0.527
0.287
22

---

Table 13: Generalization comparison on MOSI and MOSEI with random modality missing, Note: The smaller MAE indicates the better performance.
Method
7cc-7
Acc-5
Acc-2
F1
MAE
Corr
Acc-7
Acc-5
Acc-2
F1
MAE
Corr
Language Modality
Missing
MISA
15,60
15.65
52.18/53.74
48.09
48.09/49.42
1.422
0.171
30.93
26.75
62.85/7131
48.51
58.99
0.946
0.128
Self-MM
16.86
19.29
57,77/ 54.37
(2.31843..8
1.381
0.129
36.59
36.59
63.03 1013/713 77.02
49.22 58.99
0.832
0.152
17.20
/4966
46.95/37.63
31.97
62.71/71.01
54.70/59.43
0.886
0.195
CENET
17.54
17.54
57.67/51.80
42.26/35.71
1.387
0.118
41.70
41.70
63.23 / 69.38
52.41 60.67
0.821
0.238
TETEN
21.19
21.14
/55,25
/3.31
1.403
0.097
41.31
41.31
01.82/1100
48.53 58.98
0.831
0.148
TeR-Net
$7.82/ 55.30
42.43/ 39.43
1.563
0.167
41.40
48.88
(1829
0.163
ALMT
19.29
19.68
/ 55.05
66,99/ 66.01
1.394
0.139
21.58
21.58
54.28 /5.38
69.52/ 70.01
0.867
0.071
LNLN
18.80
17.68
52.18/49.05
58.89
10.39/56.33
56.84
1.427
0.075
39.10
39.10
62.85
/7.00
77.19/
83,06
0.847
0.156
Audio Modality Missing
MISA
43.49
48.44
83.74 / 81. .88
83.72 / 81.49
0.766
0.776
$1.59
$3.91
84.94 83.91
84.
83.46
01561
0.759
ScI--MM
52.33
88.24
53.91
84.70
85.06
0.763
MMIM
45.39
49.52
83.08
81.15
83.00/ 81.02
0.744
0.777
51.01
53.26
83.72/ 81.51
83.60/ 81.29
0.572
0.726
CENET
43.05
$0.39
83.08 / 81.54
83.06 / 88.54
0.750
0.785
54.34
$5.98
82.08 85.37
85.35 82.44
0.531
0,770
TETEN
51.31
0.543
TER-Net
37.71
45.48
/ 80.52
82.71 80.45
0.821
0.759
53.75
55,34
84.95
84.65
84.69
84.32
0.550
0.745
ALMT
36.49
41.84
84.60/ 79.88
84.81 / 80.70
0.865
0.767
30.51
33.55
67.13/ 70.05
72.55 71.76
0.535
0.556
LNLN
84.86 82.21
50.63
51.92
83.09
84.71
0.610
0.736
Visual Medality Missing
MISA
43.29
47.09
82.37 /
82.43
0:777
51.76
$3,80
85.25
83.99
85.11 76
0.550
0.757
Scl--MM
42.76
555.47
85.11/ 82.80
85.
82.82
0.722
0.789
53.68
55.47
85.20/ 84.62
84.95 884.9
0.533
0.761
MMIM
44.75
49.85
83.54 / 81.83
83.52 81.78
0.740
0.778
50.45
52.86
83.71 / 81.31
83.53/ 81.09
0.579
0.722
CENET
43.20
50.39
83.08 / 81
83.06
0.748
0.7885
$2.95
54.48
84.99
85.03 56
0.544
0.761
TETEN
44.22
51.46
82.62 81.10
82.67
81.09
0.719
0.794
53.55
84.84 83.78
84.81
0.544
0.761
TER-Net
37.85
45.87
83.54 / 18110
83.41 / $0.95
0.799
0.760
52.94
$4.45
83.97 / 83.48
83.93
83.18
0.577
0.726
ALMT
36.39
41.93
84.81/ 79.99
84.81
80. 76
0.863
0.768
29.
32.95
66.81
71.72/
0.559
0.747
LNLN
45.05
51.99
84.91 82.26
85.17 82.48
0.759
0.772
50.16
51.44
84.09
83.41
84.43 83.76
0.577
0.729
Language & Audio Modality Missing
MISA
15.60
55.29
26.76
58.99
0.961
Self-MM
16.86
19.29
57.77
54.03
42.31 44.31
1.381
0.130
36.61
36.61
63.01 / 77.02
49.18 58.99
0.831
0.144
MMIM
17.59
17.40
52.24 48.20
/31.53
1.389
0.024
31.91
31.92
62.31 / 700.94
54.32
0.886
0.182
CENET
17.40
17.40
57.67
51.85
/ 35.65
1.387
0.110
41.40
41.40
51.22/ 56.68
0.823
0.236
TETEN
21.19
21.14
57.77
55.22
42.31 /39.32
1.403
0.098
41.31
41.31
62.82
71.00
48.51 58.98
0.832
0.144
TER-Ne
21.38
23.96
58.03
55.59
43.37
/ 40.54
1.554
0.107
41.37
41.38
62.81 / 71.02
48.75
58.99
0.829
0.161
ALMT
19.24
56.35
65.86
1.397
0.104
21.57
21.57
57.28
69.52//
0.870
0.091
LNLN
18.80
17.68
52.18/ 49.003
58.89 56,84
1.427
0.072
34.72
34.72
62.85 / 71.02
77.19
83.06
0.900
0.145
Language & Visual Modality Missing
MISA
15.45
15.40
48.98/ 50.87
41.79
/
1.427
0.169
31.57
26.89
62.85
71.02
48.51
58.99
0.936
0.105
SeIE-MM
16.47
19.24
57.77 / 51.75
42.31
35.44
1.386
0.072
36.55
36.55
62.85 / 77.002
48.51
58.99
0.838
0.101
17.40
43.50/36.52
70.94
0.891
0.143
CENET
17.54
17.54
57.67
/5.80
42.26/35.71
1.387
0.118
41.36
41.36
61.78/
/5.67
0.838
0.112
TETEN
21.19
21.14
57.77
/ 58.25
/3.332
1.403
0.997
41.36
41.36
62.81 / 71.02
48.51 58.99
0.840
0.017
THR-Net
22.50
25.07
57.77
55.25
42.31
41.36
41.36
/7.02
0.839
0.039
ALMT
19.29
19.77
56.45/ 55.10
67.09
66.11
1.394
0.136
21.54
21. 54
$4.28
57.00
69.52
70.35
0.874
-0.086
LNLN
18.80
17.68
52.18/449.08
58.89 56.84
1.427
0.075
38.41
38.41
62.85
71.02
77.19 83.06
0.853
0.052
Modality Missing
MISA
43.25
48.40
33.28/811.2
83.29 / $1.17
0.768
0.776
51.67
$3.80
84.88 / 83.97
84.64/ 83.64
0.558
0.756
Self-MM
42.71
52.77
85.06 82,80
85.04
82.82
0.722
0.789
53.67
55.43
85.32 84.74
85.06 84.67
0.535
0.761
MMIM
44.56
51.07
83.48/ 81.29
83.42/ 11.17
0.748
0.777
50.70
53.07
83.79/
81.14
83.61 80.96
0.579
0.722
CENET
43.05
50.39
83.08 / 81.54
83.06/ S1.54
0.750
0.785
52.71
54.22
84.64 80.66
84.71 81.23
0.545
0.760
TETFN
44.12
51.36
82.57 / 81.05
82.62/81..0
0.719
0.794
$3.46
55.22
84.83/ 83.79
84.80/ 83.85
01.499
0.747
TER-Not
36.00
43.44
08
83.01
79.882
0.838
0.758
53.13
54.54
83.97 83.49
83.91 83.18
0.578
0.726
ALMT
36.49
41.93
$4.55 79.83
84.76 80.65
0.864
0.767
30.64
33.57
67.14 69.11
71.77
72.15
0.560
0.748
LNEN
45.10
52.04
84,86 82.26
85.12 82.48
0.760
0.772
50.10
51.38
84.10/ 82.69
84.39 82.86
0.609
0.730
23

---

Table 14: Generalization comparison SIMS with random modality missing. Note: The smaller
MAE indicates the hetter performance.
Method
Acc-5
Acc-3
Acc-2
Fi
MAE
Cotr
Method
Acc-5
Acc-3
Acc-2
FI
MAE
Cor
Language Modality Missing
Language & Audio Modalities Missing
18.60
50.77
56.75
21.23
54.27
Self-MM
19.77
54.20
78.26
78.00
0.594
0.058
Self-MM
19.69
54.20
74.76
72.85
0.594
0.059
MMIM
17.22
41.28
67.76
57.76
11617
11055
MMIM
17.22
41.28
67.91
57.86
11617
0.026
CENET
20.71
49.24
57.67
0.601
CENET
20.42
49.60
66.31
57.49
0.002
TETEN
19.26
36.11
69.37
56.82
0.603
0.045
TETFN
19.26
36.11
69.37
56.82
0.603
0.040
TFR-Net
22.25
54.27
69.30
56.78
0.712
0.008
TFR-Net
21.96
46.98
69.44
57.12
0.635
0.041
ALMT
21.15
24.95
69.37
81.91
0.595
ALMT
21.30
16.05
69.37
81.91
0.594
0.039
LNLN
21.37
45.30
69.37
81.91
0.597
0.005
LNLN
21.37
43.25
69.37
81.91
0.598
-0.008
Audio Modality Missing
Language & Visual Modalities Missing
MISA
38.37
61.92
77.68
75.17
0.466
0.574
19.70
48.58
69.37
56.95
0.600
0.091
SelF-MM
40.92
64.99
77.32
76.76
0.421
01585
SelF-MM
19.48
53.98
73.30
70.36
0.594
0.1156
MMIM
37.27
60.61
75.20
72.96
0.475
0.527
MMIM
17.22
40.99
67.40
57.70
0.617
0.055
CENET
21.59
$7.45
TETEN
41.94
65.86
80.16
79.16
0.424
0,589
TETFN
19.33
36.11
69.37
56.82
0.603
0.045
TFR-Ne Net
26.40
52.45
69.66
59.21
0.569
0,239
TER Net
19.55
54.27
69.37
56.82
0.720
0.027
ALMT
23.05
59.45
75.27
76.36
0.531
ALMT
20.57
69.15
81.08
0.043
LNLN
39.53
64.19
75.78
79.69
0.454
0,570
LNLN
21.44
46.39
69.37
81.91
0.600
0.014
Viusal Modality Missing
Audio & Visual Modalities Missing
MISA
40.84
63.67
78.19
77.07
01.458
0.574
MISA
38.95
61.70
77.83
75.15
(1,778
0.573
SelF-MM
41.21
64,99
73.71
0.421
0.584
SeIF-M
41.06
64.99
71.41
67.11
0.421
0.585
MMIM
37.34
60.47
75.57
73.43
01.755
0.528
MMIM
36.98
60.10
75.27
73.18
0.476
0.527
19.70
38.51
56.38
0.027
36.98
56.46
53.23
0.045
TETEN
41.94
65.86
80.16
79.19
0.424
0.589
TETFN
41.94
65.86
80.16
79.19
0.424
0,589
TER-Net
30.20
54.56
69.15
58.86
0.566
01247
TeR-Net
27.79
52.88
68.93
59.40
0.571
0.234
ALMT
22.83
75.75
ALMT
22.61
59.81
75.27
0.529
LNEN
38.80
63.31
76.22
80.11
0.458
0.569
INLN
39.82
63.46
75.93
79.72
01454
0.569
24

---

B Social Impacts
Our proposed LNLN has wide range of applications in real-world scenarios, such as healtheare and human-compunt interaction. However, it might also be misused to monitor individuals based on their affections for illegal purposes, potentially posing negative social impacts.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made the abstract and introduction accurately reflect the paper's contributions and scope. They clearly outline the capabilities of the proposed LNLN and its performance in handling data missing scenarios, as well as emphasizing the comprehensive evaluation of existing methods and their significance in improving robust Multimodal Sentiment Analysis.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of the work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
26

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The article does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The paper reports the network structure required for reproduction as well as
the hyperparameter settings. In addition, as mentioned in the abstract, the code is available GitHub.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
27

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The datasets used in the thesis open datasets and can be accessed by anyone upon request. As mentioned the abstract, the code available on GitHub.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/ /ttpg//e///oeeubmmiiiii
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips. wwtppcccccgggcueeeeeeeeeeeeeeeee gwt.//gwuee///ee/e http///eoeeueseeiieeeeeeeee ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Optimizer selection, data partitioning, and other details are described. Hyper- parameters are chosen empirically.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: As shown in 8, we report the std/fluctuation of several representative methods on the mosi dataset. It shows the overall stability of the different methods in the face of noise impacts of different levels.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report this information Section Implementation Details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: This study meets the requirements of moral and ethical norms.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We describe this in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The used data has been cited in the paper. As mentioned in the abstract, the code is available on GitHub..
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
The dataset used is public dataset, and the code is available GitHub..
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve above problems.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector:
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This study does not involve the above issues and the datasets used are all publicly available datasets.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "QTyQM4q0Ns", "reviewer": "Reviewer_ugFZ", "summary": "This paper presents LNLN, which aims to address the challenge of data incompleteness in real-world scenarios caused by sensor failures or automatic speech recognition issues. The core idea is that even if other modalities are missing, the system can still work if the information from the dominant modality is complete. Therefore, LNLN aims to augment the dominant modality (text). By focusing on text, the method uses other modalities to reconstruct missing textual information, thereby enhancing robustness.\n\nCore Contributions:\n\n1. Evaluation of MSA methods with missing data.\n2. The LNLN method.", "questions": "Should the citation format be \"cite\" instead of \"citet\"?\n\nIn line 45, is your hypothesis supported by analysis, or is it first hypothesized and then experimentally validated?\n\nHow does the Completeness Check in line 168 work, and what role does the randomly initialized $H_{cc}$ play?", "limitations": "See weakness and question", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper introduces LNLN to improve the robustness of multimodal sentiment analysis (MSA) by reinforcing the language modality. The Dominant Modality Correction (DMC) module and Dominant Modality-based Multimodal Learning (DMML) module enhance the model's robustness in various noisy scenarios by ensuring the quality of dominant modality representations.\n\nComprehensive experiments are conducted on multiple datasets (e.g., MOSI, MOSEI, and SIMS), simulating random data missing scenarios, and comparing with existing methods. The method provides additional consistency and achieves good performance.", "weaknesses": "The evaluation is indeed useful and addresses the gap in existing MSA methods with missing data. However, the innovation is limited and not sufficient to pave the way for future research, resulting in a modest contribution.\n\nThe performance improvement is not significant, and in some cases, it does not achieve state-of-the-art (SOTA) results.\n\nBased on my understanding, Section 3.4 is essentially an application of GAN, which does not highlight significant innovation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer ugFZ\n\n## **Response to W1**\n\nWe believe that our contributions are significant and address the gap in current MSA research. The robustness of MSA models in real-world, noisy environments is an important area of study. Most previous methods [1, 2, 3, 4, 5, 6, 7, 11] are evaluated on standard datasets, which may not accurately represent the challenges posed by real-world data. While some recent works [8, 9, 10] have attempted to explore robust MSA in noisy environments, these efforts often selectively report partial metrics, such as Acc-2 and F1, leading to the performance of various MSA methods in noisy scenarios under-explored. Our work aims to bridge this gap by providing a comprehensive evaluation of advanced MSA methods under noisy conditions, which we believe is a crucial step toward making these models more applicable to real-world scenarios. During this evaluation, we identified several overlooked issues. For example, our evaluation revealed a critical but neglected issue: existing models generally fail under high noise conditions. In such cases, most predictions tend to favor the more numerous categories, making the model appear to achieve SOTA results. However, this behavior is misleading and does not reflect the true performance of the model, leading to an unfair comparison with other approaches. **More details can be seen in Fig. 3, Fig. 4 and Fig. 5 in the appendix of the paper, and Fig. 1 of the uploaded PDF.** \n\n## **Response to W2** \n\nWe would like to clarify and emphasize the following points to address your concern:\n\n1. **As shown in Tab. 1 and Tab. 2 of the paper**, we believe that our method outperforms existing methods in several key aspects. For example, on the MOSI dataset, our method achieves significant improvements in challenging metrics such as Acc-7 and Acc-5, with relative increases of 9.46% and 2.74%, respectively.\n2. The reason for not achieving significant improvements in some cases is that our overall performance results are averaged across different noise levels. As mentioned in the **Response to W1** and in **Section 4.3 of the paper**, some metrics did not show significant improvement. This is because that the predictions of other models were heavily biased towards the more numerous categories under high noise scenarios, resulting those predictions meaningless. **For example**, in **Tab. 8 in the appendix of the paper**, when the data missing rate is 0.9, the Acc-7 of TFR-Net, which is specifically designed for data-missing scenarios, is 41.73%. It is slightly higher than that of LNLN at 40.10%. However, as shown in the confusion matrix in **Fig. 4 of the appendix**, TFR-Net predicted almost all test samples to be in Category 3 (i.e., Neutral). This bias significantly inflated the overall performance of TFR-Net in **Tab. 1 of the paper**, but these predictions are not meaningful in a high-noise scenario. This issue is one reason why TFR-Net's performance appears higher, but it does not reflect genuine robustness. \n3. As mentioned in **Response to W1,** we aimed to provide a fair and comprehensive evaluation, reporting as many relevant metrics as possible rather than selectively reporting only favorable results. It is unrealistic to expect a model to show significant improvements across all fine-grained metrics simultaneously. However, our method consistently maintains a competitive performance level across all reported metrics, highlighting its robustness.\n\n## **Response to W3**\n\nWe would like to clarify that while adversarial learning is employed as a tool within our model, the innovation lies in how we utilize it to address specific challenges in MSA. Similar to how other works employ CNNs and LSTMs as basic layers, we have chosen this pipeline to achieve our specific objectives. The innovation is embodied in the design of the DMC module. This pipeline is carefully designed based on our hypothesis and offers a novel approach to improving the robustness in MSA. Unlike previous approaches [8, 9] that typically treat each modality equally and reconstruct whichever information is missing, our method prioritizes the dominant modality (language) and focuses on improving its completeness in noisy scenarios. Overall, our innovative application of adversarial learning is not just in the use of GANs, but in how it is integrated into our approach to address the specific challenges of robust MSA.\n\n## **Response to Q1**\n\nAs to the format of the references themselves, any style is acceptable as long as it is used consistently. You can refer to the PDF of \"Formatting Instructions For NeurIPS 2024\".\n\n## **Response to Q2**\n\nThe hypothesis is first proposed and then experimentally validated. \n\n## **Response to Q3**\n\nThe Completeness Check mainly used to evaluate the c*ompleteness* of the language features and determine to what extent these features are affected by noise or missing data.\n\n1. $H_{cc}$ is a randomly initialized token used as an initial embedding that is concatenated with language features $H_l^1$. Through learning, it is used to predict the completeness of the $H_l^1$. Random initialization ensures that the feature is free from any a priori bias.\n2. First, the language representation $H^1_l$ and $H_{cc}$ are fed into the encoder. The encoder processes the input and capture the completeness of the $H^1_l$. Then, the encoder outputs a completeness weight ($w$) that indicates the degree of completeness. This weight is used to balance the $H^1_l$ and the proxy features $H^1_p$ generated by the adversarial learning. Finally, the $w$ is used to combine the $H^1_l$ with the generated proxy features. This ensures that the final dominant modality representation maintains high quality and integrity in the presence of noise.\n3. By evaluating and adjusting the dominant modality representation through this process, the Completeness Check helps maintain the robustness and reliability of the model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lkxxJbOTCb", "reviewer": "Reviewer_xa9F", "summary": "The paper addresses data incompleteness in Multimodal Sentiment Analysis (MSA) by presenting the Language-dominated Noise-resistant Learning Network (LNLN). By considering language as the dominant modality, LNLN introduces a Dominant Modality Correction (DMC) module and Dominant Modality-Based Multimodal Learning (DMML) module to enhance robustness against noise. Extensive experiments under random data missing scenarios on datasets like MOSI, MOSEI, and SIMS demonstrate LNLN’s superior performance. The method ensures quality representations of the dominant modality and offers improved uniformity, transparency, and fairness in evaluations, consistently outperforming existing baselines across various challenging metrics.", "questions": "Weakness：\n\n1. The contributions summarized in the introduction are not clearly articulated. Additionally, using language as the primary information carrier to reconstruct incomplete data is a concept frequently employed in multimodal sentiment analysis (MSA). What sets your method apart from other similar approaches?\n\n2. The so-called \"Adaptive Hyper-modality Learning\" essentially involves using transformers to facilitate interactions between different modalities. This interaction process appears to lack algorithmic innovation and seems to adopt a rather naive approach.\n\n3. The primary structure of the Reconstructor is also based on transformers. Why did you choose transformers for reconstruction?\n\n4. The entire algorithm's architecture heavily relies on transformers, but the transformers used in different modules have not been specifically tailored to the characteristics of this task.\n\n5. In the experimental section, although LNLN demonstrates generally good performance, there is no in-depth comparative analysis with the most similar baseline, ALMT. Since LNLN is an improvement based on ALMT, such a detailed analysis is essential.\n\n6. Ablation studies and other experiments were only conducted on the MOSI dataset. Why were these not performed on all datasets?\n\n7. The experimental section lacks visualizations or case study to demonstrate the effectiveness of the algorithm, making it difficult to fully substantiate the algorithm's efficacy.", "limitations": "The algorithm's structure is overly reliant on transformers, and there is a lack of explanation as to why transformers were chosen for various operations, especially in relation to the specific characteristics of this task. Additionally, the paper introduces multiple loss functions; however, the effectiveness and necessity of each loss function are not thoroughly justified or demonstrated. This raises questions about whether each loss is indeed contributing as intended to the overall performance of the model.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strength:\n1. The paper addresses the issue of incomplete data in multimodal sentiment analysis, a prevalent and valuable research problem in real-world applications.\n\n2. The manuscript is well-structured, with a clear and logical flow of ideas. The organization facilitates understanding and ensures that the arguments and methodologies are presented in a coherent manner.", "weaknesses": "Weakness：\n\n1. The contributions summarized in the introduction are not clearly articulated. Additionally, using language as the primary information carrier to reconstruct incomplete data is a concept frequently employed in multimodal sentiment analysis (MSA). What sets your method apart from other similar approaches?\n\n2. The so-called \"Adaptive Hyper-modality Learning\" essentially involves using transformers to facilitate interactions between different modalities. This interaction process appears to lack algorithmic innovation and seems to adopt a rather naive approach.\n\n3. The primary structure of the Reconstructor is also based on transformers. Why did you choose transformers for reconstruction?\n\n4. The entire algorithm's architecture heavily relies on transformers, but the transformers used in different modules have not been specifically tailored to the characteristics of this task.\n\n5. In the experimental section, although LNLN demonstrates generally good performance, there is no in-depth comparative analysis with the most similar baseline, ALMT. Since LNLN is an improvement based on ALMT, such a detailed analysis is essential.\n\n6. Ablation studies and other experiments were only conducted on the MOSI dataset. Why were these not performed on all datasets?\n\n7. The experimental section lacks visualizations or case study to demonstrate the effectiveness of the algorithm, making it difficult to fully substantiate the algorithm's efficacy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer xa9F\n## **Response to Limitations**\n**Regarding the Use of Transformer:**\n\n**First,** our primary focus is on the innovation of the algorithm, not on the innovation of the Transformer. We have utilized Transformer layers in a similar manner to how CNNs and LSTMs are commonly employed in various models. It is only a tool to effectively model sequence data. Extensive MSA-related works [2, 4, 6, 7, 8, 10, 11] has demonstrated the efficacy of Transformers in processing sequential data. Thus, we chose to design the LNLN framework based on the Transformer structure. **Second**, to address concerns about the algorithm’s reliance on Transformers, we evaluated our model by replacing all Transformer layers with LSTM and MLP layers, which are widely used in earlier MSA works [1, 3, 5]. **The Table below** shows that while there is a slight performance degradation, the overall performance remains competitive and still achieves SOTA results across many metrics. It also demonstrates that **our algorithm's structure is not overly reliant on transformers.** In addition, the use of Transformer to implement the reconstructor is also seen in the previous methods [4, 8], which is normal practice in this field. Our focus is on validating the effectiveness of our proposed hypothesis through the LNLN framework.\n\n|MOSI|||||||\n|-|-|-|-|-|-|-|\n|Methods|Acc-7|Acc-5|Acc-2|F1|MAE|Corr|\n|LNLN|**34.26**|**38.27**|**72.55/70.94**|**72.73/71.25**|**1.046**|**0.527**|\n|LNLN(LSTM+MLP)|33.81|38.26|70.86/69.43|70.85/69.52|1.088|0.499|\n|**MOSEI**|||||||\n|Methods|Acc-7|Acc-5|Acc-2|F1|MAE|Corr|\n|LNLN|**45.42**|**46.17**|**76.30/78.19**|**77.77/79.95**|**0.692**|**0.530**|\n|LNLN(LSTM+MLP)|43.48|44.28|76.05/77.18|77.59/79.32|0.716|0.517|\n|**SIMS**|||||||\n|Methods|Acc-5|Acc-3|Acc-2|F1|MAE|Corr|\n|LNLN|**34.64**|**57.14**|**72.73**|79.43|**0.514**|**0.397**|\n|LNLN(LSTM+MLP)|27.87|55.50|70.23|**80.57**|0.560|0.257|\n\n**Regarding the effectiveness and necessity of each loss function:**\n\n**As shown in Tab. 4 of the paper**, we have discussed the effectiveness of each loss. We have additionally conducted our experiments on MOSI and SIMS datasets. In addition, **as shown in Tab. 1 of the uploaded PDF**, the results show that our losses are effectiveness and necessity.\n\n## **Response to W1/Q1**\n**Regarding the contributions**: Please see the general Response.\n\n**Regarding the use of language as the primary information carrier:**\n\nWhile language plays a crucial role in MSA, previous works [1, 2, 3, 4, 5, 6, 8, 9, 10, 11] have not fully emphasized the importance of the language modality in method design. Our proposed LNLN addresses this gap by introducing the DMC module, specifically designed to improve the robustness of language features under varying noise scenarios. \n\n**Regarding the differentiation from existing methods:**\n\n1. Unlike many existing MSA methods [1, 2, 3, 4, 5, 6, 8, 9, 10, 11] that treat all modalities equally, our method prioritizes the integrity of the dominant modality (language). We propose that by ensuring the completeness and quality of this modality, the overall robustness of the model can be significantly improved.\n2. Our method uniquely incorporates a Completeness Check step within the DMC module, which evaluate and adjusts the representation of the dominant modality to maintain its quality. This method is distinct from other methods [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] used in MSA, which do not emphasize the completeness of the dominant modality to this extent.\n3. Our model is specifically designed to tackle a wide range of noise levels, reflecting real-world scenarios where data quality can vary. The ability to maintain robust performance across these conditions is a key differentiator of our approach compared to other methods that may only perform well under limited or ideal scenarios.\n\n## **Response to W2/Q2, W3/Q3 and W4/Q4**\n**As mentioned in the above**, our primary goal is not to introduce a novel Transformer architecture but to explore how to achieve robust MSA. The innovation of our LNLN lies in the overall framework, particularly in the design of the DMC module. The collaboration between AHL and DMC ensures that the model can effectively mitigate the impact of noise, thus achieving robust MSA. For more details, please refer to the Response to Limitations above.\n\n## **Response to W5/Q5 and W7/Q7**\n**Tab. 8 and Tab. 9 of the paper** have detailed the performance of each method under the influence of different noise intensities, demonstrating the superiority of our method compared to ALMT. In addition, **as shown in Fig. 1 and Fig. 2 of the uploaded PDF**, we have compared the confusion matrices of ALMT and LNLN, and plotted their F1 variation curves under different noise intensities. It is clear that ALMT does not perform as well as LNLN in high-noise scenarios, especially when it also tends to favour almost all of its predictions to specific classes. As **shown** **in Fig. 3 of the uploaded PDF,** We conducted a Case Study through visualisation, and we can see that our LNLN is able to perceive the sentiment cues accurately for the hard samples and thus make accurate predictions. Due to word limitations of rebuttal, we will provide a more detailed analysis in the revised version.\n\n## **Response to W6/Q6**\nWe chose to perform our ablation studies on the MOSI dataset because it is widely used in previous works for ablation studies [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]. This allowed us to maintain consistency with existing works. Additionally, due to time limitation of rebuttal, we have tried our best to conduct the ablation studies on the SIMS dataset. The SIMS dataset is a Chinese dataset which differs significantly from MOSI. It allows us to demonstrate the generalizability and robustness of our approach across different languages and cultural contexts. **As shown in Tab. 4 of the uploaded PDF,** the result highlights the effectiveness of our method on this diverse dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AGF0LUXPzo", "reviewer": "Reviewer_LeNg", "summary": "The paper proposes a Language-dominated Noise-resistant Learning Network (LNLN) to enhance the robustness of Multimodal Sentiment Analysis (MSA) under conditions of incomplete data. The model leverages the language modality, considered dominant due to its rich sentiment cues, to correct and reconstruct missing data from other modalities (visual and audio). Comprehensive experiments are conducted on datasets such as MOSI, MOSEI, and SIMS, demonstrating the superior performance of LNLN compared to existing baselines.", "questions": "How to balance the hyperparameters of the loss function is not clear.", "limitations": "Discuss potential real-world applications and scenarios where LNLN can be particularly beneficial, as well as its limitations in these contexts.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Comprehensive Evaluation: The experiments are thorough, covering multiple datasets and varying conditions of data incompleteness. This adds to the validity of the findings and allows for a fair comparison with existing methods.\n\nClear Motivation and Hypothesis: The paper clearly explains the rationale behind focusing on the language modality as the dominant one and provides a logical progression from problem identification to solution proposal.\n\nDetailed Methodology: The methodological section is detailed, providing sufficient information for replication. The incorporation of adversarial learning and dynamic weighting strategies is well-explained.\n\nSignificant Performance Improvement: Empirical results indicate that LNLN consistently outperforms other methods, showcasing its robustness and effectiveness in real-world scenarios.", "weaknesses": "Real-World Application Scenarios: Discussing potential real-world applications and limitations of LNLN would enhance the paper's practical relevance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer LeNg\n\n## **Response to W1**\n\nLNLN is well-suited for applications where multimodal data is often incomplete, which is a common challenge in many real-world s*cenarios*. For example, in platforms like Twitter, Instagram, and TikTok, users often express sentiments through a combination of text, images, videos, and audio. However, due to limitations such as incomplete captions, low-quality images, or background noise in videos, the data collected can be noisy or incomplete. LNLN's ability to prioritize the language modality while robustly integrating information from other modalities makes it ideal for analyzing such content, where data quality is variable. Moreover, in online learning platforms, students interact with content through text, video lectures, and audio discussions. LNLN can be used to analyze student sentiment and engagement, even when certain modalities, such as audio or video, are of poor quality or partially missing. \n\nHowever, limitations also exist. The data in real-world scenarios is much more complex. In addition to the presence of missing data, other factors need to be considered, such as diverse cultural contexts, varying user behavior patterns, and the influence of platform-specific features on the data. These factors can introduce additional noise and variability, which may require further model adaptation and tuning to handle effectively. \n\nIn addition, from the experiments, it can be seen that the LNLN is able to better processing different levels of missing data noise. However, there are still some challenges and areas for improvement. **1)** There are differences in how well the model generalizes across different datasets, which suggests that further work is needed to ensure consistent performance. **2)** Tuning the hyperparameters, particularly those related to the loss functions, can be challenging and may require more sophisticated approaches to achieve optimal performance. **3)** Current MSA datasets are relatively small, and acquiring and annotating such data is difficult. Therefore, improving model performance with limited data is an important area of study. Given the strong performance of large language models in many domains, we believe that future research should explore MSA based on zero-shot and few-shot learning using large language models as a potential research direction.\n\n## **Response to Q1**\n\nFor the balance of hyperparameters, **we empirically selected them through experiments as described in Sections 3.6 and Sections 4.5.** Specifically, the hyperparameters of the loss function were tuned using a combination of grid search and validation performance. We first identified a range of reasonable values for each hyperparameter and then systematically evaluated their impact on a validation set, aiming to find the combination that optimally balances the trade-offs between different objectives.\n\nAdditionally, **as shown in Table 1 of the uploaded PDF**, we have included detailed experimental results on the MOSI and SIMS datasets, further demonstrating the effectiveness of the loss function and the tuning process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response

Dear Reviewers, ACs and SACs,

We would like to express our sincere gratitude for your thoughtful questions and valuable feedback. We greatly appreciate the time and effort you have invested in reviewing our paper. **We are eager to engage in further discussions with you to address your concerns and enhance the quality of our work.**

**First, we would like to restate our contributions:**

1. We provide an extensive evaluation of existing methods, offering a deeper understanding of the performance of current advanced methods in noise scenarios.
2. Our LNLN is designed to improve the overall robustness of MSA by ensuring the quality of the dominant modality (language) across various levels of noise. This is achieved through the novel Dominant Modality Correction (DMC) pipeline, which offers new insights into addressing noise in MSA.
3. Extensive experiments and ablation studies demonstrate the effectiveness of LNLN, showing that it achieves state-of-the-art performance across most metrics, particularly under high levels of noise.

**To better respond to your questions, we have conducted several additional experiments:**

1. We conducted ablation studies on the SIMS dataset, including the model’s dependence on the Transformer structure (**see the Table in Response to Limitations of Reviewer xa9F**), effectiveness and necessity of each loss (**see Tab. 1 of the uploaded PDF**), effects of different regularization on the SIMS dataset (**see Tab. 2 of the uploaded PDF**), and the effect of different component on the SIMS dataset (**see Tab. 2 of the uploaded PDF**).
2. We included more comparisons with previous methods, incorporating visualization (**see Fig. 1 and Fig. 2 of the uploaded PDF**) and a case study (**see Fig. 3 of the uploaded PDF**), with a particular focus on the suboptimal model ALMT.

Specifically, **for the Table in Response to Limitations of Reviewer xa9F**, we replaced the Transformer with LSTM and MLP to build the LNLN model. Although there was a slight decrease in accuracy, the performance still surpassed other advanced methods. This demonstrates that our proposed algorithm does not heavily rely on the Transformer architecture. **For Tab. 1 of the uploaded PDF,** we provided a detailed analysis of the effectiveness of each loss component on the MOSI and SIMS datasets. The results confirm that each part of the loss function is beneficial and contributes to the overall performance of our algorithm. **For Tab. 2 of the uploaded PDF,** we presented ablation results of LNLN on the SIMS dataset. The SIMS dataset, as a Chinese dataset, is quite different from MOSI. The results demonstrate that LNLN has strong generalization capabilities, further demonstrating the effectiveness of our method. **For Fig. 1 of the uploaded PDF,** we compared the confusion matrices of LNLN and ALMT across the three datasets. Consistent with the results shown in **Fig. 3, Fig. 4, and Fig. 5 in the Appendix of the paper**, ALMT suffers from severe invalid predictions in high-noise scenarios, where most samples are predicted to fall into a particular category. Although our proposed LNLN also exhibits this phenomenon to some extent, it is not as severe as with ALMT. This further demonstrates the advantage of our method in processing noise data. **For Fig. 2 of the uploaded PDF,** we compared the F1 accuracy curves of several advanced methods across different random missing rates. The curve shows that LNLN maintains SOTA performance across various random missing rates, especially in high-noise scenarios, demonstrating the robustness of our method. **For Fig. 3 of the uploaded PDF,** we show case studies. In scenarios where key information is masked, such as in the text of Figure (a), our LNLN model can still predict correctly. In Figure (b), both ALMT and LNLN predict incorrectly, highlighting the challenges the model faces in extremely noisy conditions.

Thank you for your patience and support. **We look forward to continuing the discussion and refining our work with your valuable** **insights****.**

Sincerely,

The Authors

## **Reference Used throughout the Rebuttal**

[1] Zadeh, A., Chen, M., Poria, S., Cambria, E., Morency, L., 2017. Tensor fusion network for multimodal sentiment analysis, in EMNLP.

[2] Tsai, Y.H., Bai, S., Liang, P.P., Kolter, J.Z., Morency, L., Salakhutdinov, R., 2019. Multimodal transformer for unaligned multimodal language sequences, in ACL.

[3] Hazarika, D., Zimmermann, R., Poria, S., 2020. MISA: modality-invariant and-specific representations for multimodal sentiment analysis, in ACM MM.

[4] Liang, J., Li, R., Jin, Q., 2020. Semi-supervised multi-modal emotion recognition with cross-modal distribution matching, in ACM MM. 

[5] Yu, W., Xu, H., Yuan, Z., Wu, J., 2021. Learning modality-specific representations with self supervised multi-task learning for multimodal sentiment analysis, in AAAI.

[6] Yang, D., Huang, S., Kuang, H., Du, Y., Zhang, L., 2022. Disentangled representation learning for multimodal emotion recognition, in ACM MM.

[7] Zhang, H., Wang, Y., Yin, G., Liu, K., Liu, Y., Yu, T., 2023. Learning language-guided adaptive hyper-modality representation for multimodal sentiment analysis, in EMNLP.

[8] Yuan, Z., Li, W., Xu, H., Yu, W., 2021. Transformer-based feature reconstruction network for robust multimodal sentiment analysis, in ACM MM.

[9] Yuan, Z., Liu, Y., Xu, H., Gao, K., 2024. Noise imitation based adversarial training for robust multimodal sentiment analysis. IEEE TMM.

[10] Li, M., Yang, D., Lei, Y., Wang, S., Wang, S., Su, L., Yang, K., Wang, Y., Sun, M., Zhang, L., 2024. Aunified self-distillation framework for multimodal sentiment analysis with uncertain missing modalities, in AAAI.

[11] Lv, F., Chen, X., Huang, Y., Duan, L., Lin, G., 2021. Progressive modality reinforcement for human multimodal emotion recognition from unaligned multimodal sequences, in CVPR.

## **Rebuttal PDF**

Meta-review: The paper introduces the Language-dominated Noise-resistant Learning Network (LNLN) for robust Multimodal Sentiment Analysis, designed to address the challenge of incomplete data in real-world applications. LNLN leverages a Dominant Modality Correction module and a Dominant Modality-based Multimodal Learning module to enhance the robustness of sentiment analysis by focusing on the integrity of the language modality, which typically contains the most dense sentiment information. The model is tested across various datasets with random missing data scenarios, consistently outperforming existing methods.

The paper is well-written and easy to follow. The reviewers had some questions and concerns about the evaluation, datasets used in the experiments, technical details, and the motivation behind the paper. During the rebuttal, the authors addressed these issues, and the reviewers were satisfied with the responses and additional experiments. Therefore, I recommend accepting the paper.
Decision: Accept (poster)