Title: Towards Robust Multimodal Sentiment Analysis with Incomplete Data
Venue: NeurIPS 2024
Accepted: yes
Keywords: Multimodal Sentiment Analysis

Multimodal Learning

Robust MSA
Abstract: The field of Multimodal Sentiment Analysis (MSA) has recently witnessed an emerging direction seeking to tackle the issue of data incompleteness. Recognizing that the language modality typically contains dense sentiment information, we consider it as the dominant modality and present an innovative Language-dominated Noise-resistant Learning Network (LNLN) to achieve robust MSA. The proposed LNLN features a dominant modality correction (DMC) module and dominant modality based multimodal learning (DMML) module, which enhances the model's robustness across various noise scenarios by ensuring the quality of dominant modality representations. Aside from the methodical design, we perform comprehensive experiments under random data missing scenarios, utilizing diverse and meaningful settings on several popular datasets (e.g., MOSI, MOSEI, and SIMS), providing additional uniformity, transparency, and fairness compared to existing evaluations in the literature. Empirically, LNLN consistently outperforms existing baselines, demonstrating superior performance across these challenging and extensive evaluation metrics.

Towards Robust Multimodal Sentiment Analysis with Incomplete Data

Abstract

The field of Multimodal Sentiment Analysis (MSA) has recently witnessed an emerging direction seeking to tackle the issue of data incompleteness. Recognizing that the language modality typically contains dense sentiment information, we consider it as the dominant modality and present an innovative Languagedominated Noise-resistant Learning Network (LNLN) to achieve robust MSA. The proposed LNLN features a dominant modality correction (DMC) module and dominant modality based multimodal learning (DMML) module, which enhances the model's robustness across various noise scenarios by ensuring the quality of dominant modality representations. Aside from the methodical design, we perform comprehensive experiments under random data missing scenarios, utilizing diverse and meaningful settings on several popular datasets ( e.g., MOSI, MOSEI, and SIMS), providing additional uniformity, transparency, and fairness compared to existing evaluations in the literature. Empirically, LNLN consistently outperforms existing baselines, demonstrating superior performance across these challenging and extensive evaluation metrics. 1

1 Introduction

The field of Multimodal Sentiment Analysis (MSA) is at the vanguard of human sentiment identification by assimilating heterogeneous data types, such as video, audio, and language. Its applicability spans numerous fields, prominent among healthcare and human-computer interaction (Jiang et al., 2020). MSA has become essential in enhancing both the precision and the robustness of sentiment analysis by drawing sentiment cues from diverse perspectives.

Changing trends in recent research has taken a turn towards modeling data in natural scenarios from laboratory conditions (Tsai et al., 2019; Hazarika et al., 2020; Yu et al., 2021; Zhang et al., 2023). The shift has created a wider application space in the real-world for MSA, though concerns arise due to problems like sensor failures and problems with Automatic Speech Recognition (ASR), leading to inconsistencies such as incomplete data in real-world deployment.

Numerous impactful solutions have been proposed against this primary concern of incomplete data in multimodal sentiment analysis. For instance, Yuan et al. (2021) introduced a Transformerbased feature reconstruction mechanism, TFR-Net, aiming to strengthen the robustness of the model handling random missing in unaligned multimodal sequences via reconstructing missing data.

Furthermore, Yuan et al. (2024) proposed the Noise Intimating-based Adversarial Training (NIAT) model, which superiorly learns a unified joint representation between an original-noisy instance pair, utilizing the attention mechanism and adversarial learning. Lastly, Li et al. (2024) design a Unified Multimodal Missing Modality Self-Distillation Framework (UMDF) which leverages a single network to learn robust inherent representations from consistent multimodal data distributions. Yet, despite these developments, the evaluation metrics for these models are inconsistent, and the evaluation settings are not sufficiently comprehensive. This inconsistency limits effective comparisons and hinders the dissemination of knowledge in the field.

Addressing this gap, our paper aims to offer a comprehensive evaluation on three widely-used datasets, namely MOSI (Zadeh et al., 2016), MOSEI (Zadeh et al., 2018) and SIMS (Yu et al., 2020) datasets. We introduce random data missing instances and subsequently compare the performance of existing methods on these datasets. This endeavor seeks to provide an all-encompassing outlook for evaluating the effectiveness and robustness of various methods in the face of incomplete data, thereby sparking new insights in the field. Additionally, inspired by the previous work ALMT (Zhang et al., 2023), we hypothesize that model robustness improves when the integrity of the dominant modality is preserved despite varying noise levels. Therefore, we introduce a novel model, namely Language-dominated Noise-resistant Learning Network (LNLN), to enhance MSA's robustness over incomplete data. LNLN aims to augment the integrity of the language modality's features, regarded as the dominant modality due to its richer sentiment cues, with the support of other auxiliary modalities. The LNLN's robustness against varying levels of data incompleteness is achieved through a dominant modality correction (DMC) module for dominant modality construction, a dominant modality based multimodal learning (DMML) module for multimodal fusion and classification, and a reconstructor for reconstructing missing information to shield the dominate modality from noise interference. This approach ensures a high-quality dominant modality feature, which significantly bolsters the robustness of LNLN under diverse noise conditions. Consequently, extensive experimental results demonstrate the LNLN's superior performance across these challenging and evaluation metrics.

In summary, this paper conducts a comprehensive evaluation of existing advanced MSA methods. This analysis highlights the strengths and weaknesses of various methods when contending with incomplete data. We believe, it can improve the understanding of different MSA methods' performance under complex real-world scenarios, thereby informing technology's future trajectory. Our proposed LNLN also offers valuable insight and guidelines for thriving in this research space.

2 Related Work

2.1 Multimodal Sentiment Analysis

Multimodal Sentiment Analysis (MSA) methods can be categorized into Context-based MSA and Noise-aware MSA, depending on the modeling approach. Most of previous works (Zadeh et al., 2017; Tsai et al., 2019; Mai et al., 2020; Hazarika et al., 2020; Liang et al., 2020; Rahman et al., 2020; Yu et al., 2021; Han et al., 2021; Lv et al., 2021; Yang et al., 2022; Guo et al., 2022; Zhang et al., 2023) can be classified to Context-based MSA. This line of work primarily focuses on learning unified multimodal representations by analyzing contextual relationships within or between modalities. For example, Zadeh et al. (2017) explore computing the relationships between different modalities using the Cartesian product. Tsai et al. (2019) utilize pairs of Transformers to model long dependencies between different modalities. Yu et al. (2021) propose generating pseudo-labels for each modality to further mine the information of consistency and discrepancy between different modalities.

Despite these advances, context-based methods are usually suboptimal under varying levels of noise effects ( e.g. random data missing). Several recent works (Mittal et al., 2020; Yuan et al., 2021, 2024; Li et al., 2024) have been proposed to tackle this issue. For example, Mittal et al. (2020) introduce a modality check step to distinguish invalid and valid modalities, achieving higher robustness. Yuan et al. (2024) propose learning a unified joint representation between constructed 'original-noisy' instance pairs. Although there have been some advances in improving the model's robustness under noise scenarios, no extant method has provided a comprehensive and in-depth comparative analysis.

2.2 Robust Representation Learning in MSA

Context-based MSA and Noise-aware MSA differ in their approaches to robust representation learning. In Context-based MSA, robust representation learning typically relies on modeling intraand inter-modality relationships. For instance, Hazarika et al. (2020) and Yang et al. (2022) apply feature disentanglement to each modality, modeling multimodal representations from multiple feature subspaces and perspectives. Yu et al. (2021) and Liang et al. (2020) explore self-supervised learning and semi-supervised learning to enhance multimodal representations, respectively. Tsai et al. (2019) and Rahman et al. (2020) introduce Transformer to learn the long dependencies of modalities. Zhang et al. (2023) devise a language-guided learning mechanism that uses modalities with more intensive sentiment cues to guide the learning of other modalities. In contrast, Noise-aware MSA focuses more on perceiving and eliminating the noise present in the data. For example, Mittal et al. (2020) design a modality check module based on metric learning and Canonical Correlation Analysis (CCA) to identify the modality with greater noise. Yuan et al. (2021) design a feature reconstruction network to predict the location of missing information in sequences and reconstruct it. Yuan et al. (2024) introduce adversarial learning (Goodfellow et al., 2014) to perceive and generate cleaner representations.

In this work, the LNLN belongs to the noise-aware MSA category. Inspired by Zhang et al. (2023), we explore the capability of language-guided mechanisms in resisting noise and aim to provide new perspectives for the study of MSA in noisy scenarios.

3 Method

3.1 Overview

The overall pipeline for the proposed Language-dominated Noise-resistant Learning Network (LNLN) in robust multimodal sentiment analysis is illustrated in Figure 1. As depicted, a crucial initial step involves forming a multimodal input with random data missing, which sets the stage for LNLN training. Once the input is prepared, LNLN first utilizes an embedding layer to standardize the dimension of each modality, ensuring uniformity. Recognizing that language is the dominant modality in MSA (Zhang et al., 2023), a specially designed Dominant Modality Correction (DMC) module employs adversarial learning and a dynamic weighted enhancement strategy to mitigate noise impacts. This module first enhances the quality of the dominant feature computed from the language modality and then integrates them with the auxiliary modalities (visual and audio) in the dominant modality based multimodal learning (DMML) module for effective multimodal fusion and classification. This process significantly bolsters LNLN's robustness against various noise levels. Moreover, to refine the network's capability for fine-grained sentiment analysis, a simple reconstructor is implemented to reconstruct missing data, further enhancing the system's robustness.

3.2 Input Construction and Multimodal Input

Given the challenges of random data missing, we have constructed data sets that simulate these conditions based on MOSI, MOSEI, and SIMS datasets.

Random Data Missing. Following the previous method (Yuan et al., 2021), for each modality, we randomly erased changing proportions of information (from 0% to 100%). Specifically, for visual and audio modalities, we fill the erased information with zeros. For language modality, we fill the erased information with [UNK] which indicates the unknown word in BERT (Kenton and Toutanova, 2019).

Multimdoal Input. For each sample in the dataset, we incorporate data from three modalities: language, audio, and visual data. Consistent with previous works (Zhang et al., 2023), each modality is processed using widely-used tools: language data is encoded using BERT (Kenton and Toutanova, 2019), audio features are extracted through Librosa (McFee et al., 2015), and visual features are obtained using OpenFace (Baltrusaitis et al., 2018). These pre-processed inputs are represented as sequences, denoted by U 0 m ∈ R T m × d m , where m ∈ { l, v, a } represents the modality type ( l for language, v for visual, a for audio), T m indicates the sequence length and d m refers to the dimension of each modality's vector. With obtained U 0 m , we apply random data missing to U 0 m , thus forming the noise-corrupted multimodal input U 1 m ∈ R T m × d m .

I am fine.

Language

Video

Audio

Multimodal Sample

Feature

Extraction

…

…

…

…

…

…

Data

Missing

…

…

…

…

…

…

𝐻

!

"

𝐻

#

"

𝐻

$

"

Language

Embedding

Visual

Embedding

Audio

Embedding

Adaptive

Hyper-modality Learning

Cross-modality

Fusion Transformer

𝐻

!

&

𝐻

#

&

𝐻

$

&

Completeness

Check

Feature Generator

𝑤

𝐻

%

&

𝐻

!

&

𝐻

#

&

𝐻

$

&

𝐻

'

&

Effectiveness

Discriminator

ℒ

$'#

ℒ

))

ℒ

*%

Feature Generation

𝑤

1 - 𝑤

𝑈

!

&

𝑈

#

&

𝑈

$

&

Reconstruction Layer

ℒ

+()

Reconstructor

Dominate Modality Correction

(

𝑈

!

"

(

𝑈

#

"

(

𝑈

$

"

% 𝑦

%

% 𝑦

𝐻

!

&

Dominant Modality based Multimodal Learning

𝐻

))

𝐻

%

"

𝑈

!

&

𝑈

#

&

𝑈

$

&

𝑈

!

"

𝑈

#

"

𝑈

$

"

𝑈

!

&

𝑈

#

&

𝑈

$

&

Training Pipeline

3.3 Dominant Modality based Multimodal Learning

Inspired by previous work ALMT, we hypothesize that model robustness improves when the integrity of the dominant modality is preserved despite varying noise levels. We improve ALMT based on the designed DMC module and Reconstructor, thus implementing dominant modality based DMML module for sentiment analysis under random data missing scenarios. Here, we mainly introduce the parts that differ from ALMT. Further details are available in Zhang et al. (2023).

Modality Embedding. For multimodal input U 1 m , we employ an Embedding Encoder E 1 m with two Transformer encoder layers to extract and unify the feature. Each modality begins with a randomly initialized low-dimensional token H 0 m ∈ R T × d m . These tokens are then processed by the Transformer encoder layer, embedding essential modality information and producing unified features, represented as H 1 m ∈ R T × d . The process is formalized by:

where E 1 m ( · ) extracts features for each modality, and concat( · ) represents the concatenation operation.

Adaptive Hyper-modality Learning. In the original ALMT, each Adaptive Hyper-modality Learning layer contains a Transformer and two multi-head attention (MHA) modules. These are applied to learn language representations at different scales and hyper-modality representations from visual and audio modalities, guided by the language modality. Considering the possibility of severe interference in the language modality ( i.e. dominant modality) due to random data missing, we designed a Dominant Modality Correction (DMC) module to generate the proxy dominant feature H 1 p and construct corrected dominate feature H 1 d (more details can be found in Section 3.4). Specifically, the process of learning corrected dominated representation H i d at different scales can be described as:

where i ∈ { 2 , 3 } means the i -th layer of Adaptive Hyper-modality Learning module, E i m ( · ) is the i-th Transformer encoder layer, H i d ∈ R T × d is corrected dominated feature at different scale. To learn the hyper-modality representation, the corrected dominated feature and audio/visual features are used to calculate Query and Key/Value, respectively. Briefly, the process can be written as follows:

where MHA( · ) represents multi-head attention, H i hyper ∈ R T × d is the hyper-modality feature. Note that the feature H 0 hyper ∈ R T × d is a random initialized vector.

Multimodal Fusion and Prediction. With obtained H 3 d and H 3 hyper , a Transformer encoder with a classifier at a depth of 4 layers is employed for multimodal fusion and sentiment prediction:

where ˆ y is the sentiment prediction.

3.4 Dominant Modality Correction

This module consists of two steps, i.e., completeness check of dominant modality and proxy dominant feature generation using adversarial learning (Ganin and Lempitsky, 2015; Goodfellow et al., 2014).

Completeness Check. We apply an encoder E cc that consists of a Transformer encoder with a depth of two layers and a classifier for completeness check. For example, if the missing rate of dominate modality is 0.3, the label of completeness is 0.7. This completeness prediction w can be obtained as follows:

where H cc ∈ R T × d is a randomly initialized token for completeness prediction. We optimize this process using L2 loss:

where N b is the number of samples in the training set, ˆ w k is the label of completeness of k -th sample.

Proxy Dominant Feature Generation. With the randomly initialized feature H 0 p ∈ R T × d , visual feature H 1 v and audio feature H 1 a , we employ a Proxy Dominant Feature Generator E DFG , which consists of two Transformer encoder layers. This setup generates the proxy dominant features H 1 p ∈ R T × d , designed to complement and correct the dominant modality. The corrected dominant feature H 1 d ∈ R T × d is calculated by combining H 1 p and the language feature H 1 l , weighted by the predicted completeness w :

where θ DFG denotes the parameters of the Proxy Dominant Feature Generator E DFG .

To ensure that the agent feature offers a distinct perspective from the visual and audio features, we utilize an effectiveness discriminator D . This discriminator includes a binary classifier and a Gradient Reverse Layer (GRL) (Ganin and Lempitsky, 2015) and is tasked with identifying the origin of the agent features:

where θ D represents the parameters of the effectiveness discriminator D , and ˆ y p indicates the prediction of whether the input feature originates from the language modality.

In practice, the generator and the discriminator engage in an adversarial learning structure. The discriminator aims to identify whether the features are derived from the language modality, while the generator's objective is to challenge the discriminator's ability to make accurate predictions. This dynamic is encapsulated in the adversarial learning objective:

where N b is the number of samples in the training set, and y k p indicates the label determining whether the input feature for the k -th sample originates from the visual or audio modality.

3.5 Reconstructor

Our experiments demonstrate that reconstructing missing information can significantly enhance regression metrics. More details about this are shown in Table 4. To address this, we have developed

a reconstructor, denoted as E rec , which comprises two Transformer layers designed to effectively rebuild missing information of each modality. The operational equation for the reconstructor is:

where ˆ U 0 m is the reconstructed feature corresponding to the feature U 0 m .

To optimize the performance of the reconstructor, we apply an L2 loss function:

where U 0 m k and ˆ U 0 m k represent the original and reconstructed features with missing information for the k -th sample, respectively. This loss function helps minimize the discrepancies between the original and reconstructed features, thereby improving the accuracy of other components, such as Dominant Modality Correction and final sentiment prediction.

3.6 Overall Learning Objectives

To sum up, our method involves four learning objectives, including a completeness check loss L cc , an adversarial learning loss L adv for proxy dominant feature generation, a reconstruction loss L rec and one final sentiment prediction loss L sp . The sentiment prediction loss L sp can be described as:

Therefore, the overall loss L can be written as:

where α , β , γ and δ are hyperparameters. On MOSI and MOSEI datasets, we empirically set them to 0.9, 0.8, 0.1, and 1.0, respectively. On the SIMS dataset, we empirically set them to 0.9, 0.6, 0.1, and 1.0, respectively.

4 Experiments and Analysis

In this section, we provide a comprehensive and fair comparison between the proposed LNLN and previous representative MSA methods on MOSI (Zadeh et al., 2016), MOSEI (Zadeh et al., 2018) and SIMS (Yu et al., 2020) datasets.

4.1 Datasets

MOSI. The dataset includes 2,199 multimodal samples, integrating visual, audio, and language modalities. It is divided into a training set of 1,284 samples, a validation set of 229 samples, and a test set of 686 samples. Every single sample has been given a sentiment score, varying from -3, indicating strongly negative sentiment, to 3, signifying strongly positive sentiment.

MOSEI. The dataset consists of 22,856 video clips sourced from YouTube. The sample is divided into 16,326 clips for training, 1,871 for validation, and 4,659 for testing. Each clip is labeled with a score, ranging from -3, denoting the strongly negative, to 3, denoting the strongly positive.

SIMS. The dataset is a Chinese multimodal sentiment dataset that includes 2,281 video clips sourced from different movies and TV series. It has been partitioned into 1,368 samples for training, 456 for validation, and 457 for testing. Each sample has been manually annotated with a sentiment score ranging from -1 (negative) to 1 (positive).

4.2 Evaluation Settings and Criteria

For a fair and comprehensive evaluation, we experiment ten times, setting the missing rates r to predefined values from 0 to 0.9 with an increment of 0.1. For instance, 50% of the information is

randomly erased from each modality in the test data when r = 0 . 5 . Unlike previous works (Yuan et al., 2021, 2024), we did not evaluate at r = 1 . 0 , as this would imply complete data erasure from each modality, rendering the experiment non-informative. With the obtained results of each missing rate, we compute the average value as the model's overall performance under different levels of noise.

For evaluation criteria, we report the binary classification accuracy (Acc-2), the F1 score associated with Acc-2, and the mean absolute error (MAE). For Acc-2, we calculated accuracy and F1 in two ways: negative/positive (left-side value of /) and negative/non-negative (right-side value of /) on the MOSI and MOSEI datasets, respectively. Additionally, we provide the three-class accuracy (Acc-3), the seven-class accuracy (Acc-7), and the correlation of the model's prediction with humans (Corr) on the MOSI dataset. For the SIMS dataset, we report Acc-3, the five-class accuracy (Acc-5), and Corr. Due to the distinct focus of regression and classification metrics on different aspects of model performance, a model achieving the lowest error on regression metrics may not necessarily exhibit optimal performance on classification metrics. To comprehensively reflect the model capabilities, we select the best-performing checkpoint for each type of metric across all models in the comparisons, thus capturing the peak performance of both regression and classification aspects independently.

4.3 Implementation Details

We used PyTorch 2.2.1 to implement the method. The experiments were conducted on a PC with an AMDEPYC 7513 CPU and an NVIDIA Tesla A40. To ensure consistent and fair comparisons across all methods, we conducted each experiment three times using fixed random seeds of 1111, 1112, and 1113. Details of the hyperparameters are shown in Table 1. In addition, the result of MISA, Self-MM, MMIM, CENET, TETFN, and TFR-Net is reproduced by the authors from open source code in the MMSA 2 (Mao et al., 2022), which is a unified framework for MSA, using default hyperparameters. The result of ALMT is reproduced by the authors from open source code on Github 3 .

4.4 Robustness Comparison

Tables 2 and 3 show the robustness evaluation results on the MOSI, MOSEI, and SIMS datasets. As shown in Table 2, LNLN achieves state-of-the-art performance on most metrics. On the MOSI dataset, LNLN achieved a relative improvement of 9.46% on Acc-7 compared to the sub-optimal result obtained by MMIM, demonstrating the robustness of LNLN in the face of different noise effects. However, on the MOSEI dataset, LNLN achieves only sub-optimal performance on metrics such as Acc-7 and Acc-5. After analyzing the data distribution (see Appendix A.3) and the confusion matrix (see Appendix A.6), we believe that this is because LNLN does not overly bias the neutral category, which has a disproportionate number of samples in noisy scenarios. As shown in Table 3, LNLN achieves an improvement in F1 on the SIMS dataset, with a relative improvement of 9.17% on F1 compared to the sub-optimal result obtained by ALMT. Similar to CENET's performance on the MOSEI dataset, TETFN on the SIMS dataset has a tendency to predict inputs as weakly negative categories with high sample sizes, resulting in seemingly better performance on some metrics. Additionally, as shown in Figure 2, we present the performance curves of several advanced

methods under varying missing rates. The results demonstrate that the proposed LNLN consistently outperforms others across most scenarios, showing its robustness under different missing rates.

Overall, LNLN attempts to make predictions in the face of highly noisy inputs without the severe lazy behavior observed in other models, which often leads to predictions heavily biased towards a certain category. This demonstrates that LNLN shows strong robustness and competitiveness across various datasets and noise levels, highlighting its effectiveness in multimodal sentiment analysis.

MAE

Missing Rate

𝒓

MAE

Missing Rate

𝒓

MAE

Missing Rate

𝒓

(d)

(e)

(f)

F1

Missing Rate

𝒓

F1

Missing Rate

𝒓

F1

Missing Rate

𝒓

(a)

(b)

(c)

4.5 Effects of Different Components

To verify the role of different components, we present the results after subtracting each component on the MOSI dataset separately. As shown in Table 4, some metrics show an upward trend in performance when certain modules are removed. On one hand, we believe this is due to randomness in noisy scenarios. On the other hand, the same hyperparameters cannot make the model perform optimally across all metrics. Moreover, the performance of LNLN decreases relatively slightly when the DMC and reconstructor are removed individually. However, there is a significant drop in performance when both are removed, e.g., 4.72% decrease in Acc-7 and 5.87% decrease in Acc-5. Performance drops further after replacing DMML with concatenation fusion, which also proves that DMML plays an important role in MSA. Additionally, after removing the noisy data used for training, the performance of LNLN also shows a significant decrease. This observation indicates that LNLN needs to learn to be aware of noise with the help of noisy data. Overall, both the presence of noisy data and well-designed components are crucial for robust MSA in scenarios with random missing data.

4.6 Effects of Different Regularization

As shown in Table 5, we present the ablation results after subtracting each regularization on the MOSI dataset separately. Obviously, after each regularization is removed, LNLN shows a decrease in performance on most metrics. This demonstrates that regularization contributes to the learning of the model. Similar to the results in Table 4, there are a few metrics that improve after the removal of regularization. We believe this is mainly due to the difficulty of optimizing a single set of hyperparameters to achieve the best performance across all metrics.

4.7 Effects of Different Modalities

To verify the role of different modalities, we present the ablation results after subtracting each modality from the MOSI dataset. As shown in Table 6, all modalities contribute to the final performance. Notably, when the language modality is removed, the performance of LNLN shows a significant decrease, while removing the visual and audio modalities results in a smaller decrease. This indicates that the language modality plays a crucial role in the MSA dataset. Moreover, we observe that LNLN converges on most metrics even when the visual and audio modalities are removed. This is due to the model's lazy behavior, where LNLN tends to fix its predictions to a certain category. More details can be found in Appendix A.3 and A.6.

4.8 Discussion

In this work, we conduct a comprehensive evaluation of current advanced methods, the results of which have triggered some thoughts. Specifically, we find that all methods fail to accurately predict

the inputs in high missing rate scenarios because most of them are designed specifically for complete data (see Appendix A.6 for more details). However, some methods show relatively more robust performance in low missing rate scenarios and some specific scenarios (see Appendix A.8), such as Self-MM, TETFN, and ALMT. We believe that some of the ideas in these methods may be useful for future research on more robust MSA for incomplete data.

For example, the Unimodal Label Generation Module (ULGM) (Yu et al., 2021) used in both SelfMMand TETFN facilitates the modeling of incomplete data. Due to the random data missing, the temporal and structured affective information in the data is corrupted, making it difficult for the models to perceive consistency and variability information between different modalities. Using ULGM to generate pseudo-labels as an additional supervisory signal for the models may directly enable the model to learn the relationship between different modalities. For ALMT, we believe that its Adaptive Hyper-modality Learning (AHL) module facilitates the modeling of robust affective representations. AHL mainly uses the dominant modality to compute query vectors and applies multi-head attention to query useful sentiment cues from other modalities as a complement to the dominant modality. This process may reduce the difficulty of multimodal fusion by avoiding the introduction of sentiment-irrelevant information to some extent. Our proposed LNLN, based on this idea from ALMT, proves the effectiveness of the method.

Additionally, we explore the performance of these methods in modality missing scenarios, which is a special case of random data missing where the partial modality missing rate r = 1 . 0 . We find that when language modality is missing, the performance of many models decreases significantly and converges to the same value. More details can be seen in Appendix A.8.

5 Conclusion and Future Work

In this paper, a novel Language-dominated Noise-resistant Learning Network (LNLN) is proposed for robust MSA. Due to the collaboration between Dominate Modality Correction (DMC) module, the Dominant Modality based Multimodal Learning (DMML) module, and the Reconstructor, LNLN can enhance the dominant modality to achieve superior performance in data missing scenarios with varying levels. Extensive evaluation shows that none of the existing methods can effectively model the data with high missing rates. The related analyses can provide suggestions for other researchers to better handle robust MSA. In the future, we will focus on improving the generalization of the model to handle different types of scenes and varying intensities of noise effectively.

Limitations

We believe there are several limitations to the LNLN. 1) The LNLN achieves good performance in data missing scenarios, but is not always better than all other methods in the modality missing scenarios, which demonstrates the lack of multi-scene generalization capability of the LNLN. 2) The data in real-world scenarios is much more complex. In addition to the presence of missing data, other factors need to be considered, such as diverse cultural contexts, varying user behavior patterns, and the influence of platform-specific features on the data. These factors can introduce additional noise and variability, which may require further model adaptation and tuning to handle effectively. 3) Tuning the hyperparameters, particularly those related to the loss functions, can be challenging and may require more sophisticated methods to achieve optimal performance.

References

Baltrusaitis, T., Zadeh, A., Lim, Y.C., Morency, L., 2018. Openface 2.0: Facial behavior analysis toolkit, in: 2016 IEEE Winter Conference on Applications of Computer Vision, pp. 59-66.

Ganin, Y., Lempitsky, V.S., 2015. Unsupervised domain adaptation by backpropagation, in: International Conference on Machine Learning, pp. 1180-1189.

Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A.C., Bengio, Y., 2014. Generative adversarial nets, in: Advances in Neural Information Processing Systems, pp. 2672-2680.

Guo, J., Tang, J., Dai, W., Ding, Y ., Kong, W., 2022. Dynamically adjust word representations using unaligned multimodal information, in: Proceedings of the 30th ACM International Conference on Multimedia, pp. 3394-3402.

Han, W., Chen, H., Poria, S., 2021. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 9180-9192.

Hazarika, D., Zimmermann, R., Poria, S., 2020. MISA: modality-invariant and -specific representations for multimodal sentiment analysis, in: Proceedings of the 28th ACM International Conference on Multimedia, pp. 1122-1131.

Jiang, Y., Li, W., Hossain, M.S., Chen, M., Alelaiwi, A., Al-Hammadi, M., 2020. A snapshot research and implementation of multimodal information fusion for data-driven emotion recognition. Information Fusion 53, 209-221.

Kenton, J.D.M.W.C., Toutanova, L.K., 2019. Bert: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of NAACL-HLT, pp. 4171-4186.

Li, M., Yang, D., Lei, Y., Wang, S., Wang, S., Su, L., Yang, K., Wang, Y., Sun, M., Zhang, L., 2024. A unified self-distillation framework for multimodal sentiment analysis with uncertain missing modalities, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 10074-10082.

Liang, J., Li, R., Jin, Q., 2020. Semi-supervised multi-modal emotion recognition with cross-modal distribution matching, in: Proceedings of the 28th ACM International Conference on Multimedia, pp. 2852-2861.

Lv, F., Chen, X., Huang, Y., Duan, L., Lin, G., 2021. Progressive modality reinforcement for human multimodal emotion recognition from unaligned multimodal sequences, in: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2554-2562.

Mai, S., Xing, S., Hu, H., 2020. Locally confined modality fusion network with a global perspective for multimodal human affective computing. IEEE Transactions on Multimedia 22, 122-137.

Mao, H., Yuan, Z., Xu, H., Yu, W., Liu, Y., Gao, K., 2022. M-SENA: an integrated platform for multimodal sentiment analysis, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 204-213.

McFee, B., Raffel, C., Liang, D., Ellis, D.P.W., McVicar, M., Battenberg, E., Nieto, O., 2015. librosa: Audio and music signal analysis in python, in: Proceedings of the 14th Python in Science Conference, pp. 18-24.

Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., Manocha, D., 2020. M3ER: multiplicative multimodal emotion recognition using facial, textual, and speech cues, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 1359-1367.

Rahman, W., Hasan, M.K., Lee, S., Zadeh, A.B., Mao, C., Morency, L., Hoque, M.E., 2020. Integrating multimodal information in large pretrained transformers, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2359-2369.

Tsai, Y.H., Bai, S., Liang, P.P., Kolter, J.Z., Morency, L., Salakhutdinov, R., 2019. Multimodal transformer for unaligned multimodal language sequences, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6558-6569.

Yang, D., Huang, S., Kuang, H., Du, Y., Zhang, L., 2022. Disentangled representation learning for multimodal emotion recognition, in: Proceedings of the 30th ACM International Conference on Multimedia, pp. 1642-1651.

Yu, W., Xu, H., Meng, F., Zhu, Y., Ma, Y., Wu, J., Zou, J., Yang, K., 2020. CH-SIMS: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3718-3727.

Yu, W., Xu, H., Yuan, Z., Wu, J., 2021. Learning modality-specific representations with selfsupervised multi-task learning for multimodal sentiment analysis, in: Proceedings of the AAAI conference on artificial intelligence, pp. 10790-10797.

Yuan, Z., Li, W., Xu, H., Yu, W., 2021. Transformer-based feature reconstruction network for robust multimodal sentiment analysis, in: Proceedings of the 29th ACM International Conference on Multimedia, pp. 4400-4407.

Yuan, Z., Liu, Y., Xu, H., Gao, K., 2024. Noise imitation based adversarial training for robust multimodal sentiment analysis. IEEE Transactions on Multimedia 26, 529-539.

Zadeh, A., Chen, M., Poria, S., Cambria, E., Morency, L., 2017. Tensor fusion network for multimodal sentiment analysis, in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1103-1114.

Zadeh, A., Liang, P.P., Poria, S., Cambria, E., Morency, L., 2018. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph, in: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 2236-2246.

Zadeh, A., Zellers, R., Pincus, E., Morency, L., 2016. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems 31, 82-88.

Zhang, H., Wang, Y., Yin, G., Liu, K., Liu, Y., Yu, T., 2023. Learning language-guided adaptive hyper-modality representation for multimodal sentiment analysis, in: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 756-767.

A Additional Experiments and Analysis

A.1 Effect of Regularization Weight on Model Performance

As shown in Table 7, we empirically tried different combinations of hyperparameters in the loss function. The results show that the selected parameters can make the LNLN achieve better performance in most metrics. This also demonstrates the effectiveness of the optimization objectives of LNLN.

A.2 Analysis of Model Stability

To evaluate the stability of the model, we selected the MOSI dataset for ablation experiments. Specifically, based on the overall performance (mean) in Table 2, we selected several representative methods to additionally compute the standard deviation. For each method, we first calculate the standard deviation of evaluation results for the three random seeds when the missing rate r ranges from 0 to 0.9, and then average the 10 sets of standard deviation. The final result is shown in Table 8. We can see that the standard deviation of these methods is not very large in most metrics, indicating that all these methods can guarantee stability in the presence of random noise. It should be noted that LNLN strikes a balance between overall performance and stability.

A.3 Analysis of Data Distribution

Figure 4 illustrates the data distribution on the MOSI, MOSEI, and SIMS datasets. Significant category imbalances can be seen across all datasets. Additionally, the distributions of the training set, validation set, and test set are not identical on the MOSI and MOSEI datasets. For example, as shown in Figure 4 (a), the percentage of weakly negative samples is 15.5% in the MOSI training set, while this category reaches 22.7% in the test set, representing the highest percentage among all categories.

Similarly, Figure 4 (b) and Figure 4 (c) show that this phenomenon also exists in the MOSEI and SIMS datasets.

The unbalanced data distribution makes it difficult for the model to perform well in data missing scenarios. Specifically, it is easy for the model to engage in lazy behavior in high-noise scenarios, simply biasing predictions towards categories with a higher proportion in the training set. These unbalanced distributions may further increase the learning difficulty of the model when facing data missing. More details can be found in Appendix A.6.

A.4 Case Study

As shown in Figure 3, we visualize several successful and failed predictions made by LNLN and ALMT from the MOSI dataset for the case study. It shows that LNLN can perceive sentiment cues in challenging samples, which demonstrates its ability to capture sentiment cues in noisy scenarios. However, for inputs with high missing rates ( e.g., the third example in the figure), both LNLN and ALMT fail to make correct predictions. We believe this is due to the high loss of valid information in the multimodal input, making accurate predictions difficult.

AND HE WAS STILL BORING

Input

ALMT

LNLN

Positive

×

Negative √

SIGH I LOVE WHEN THEY SHOW

ALL THE THE WAR SCENES ALL

THE THE TIME THEY SPENT IN

THE MILITARYAND SHIT

Negative

×

Negative

×

MY 8

YEAR

OLD

WAS

TERRIFIED THROUGH PART OF

THIS BECAUSE THE ORIGINAL

VERSION IS TERRIFYING

Positive

×

Negative √

A.5 Details of Robust Comparison

Tables 9 and 10 show the details of robust comparison on the MOSI, MOSEI, and SIMS datasets, respectively. We observe that Self-MM and TETFN have a clear advantage in many evaluation metrics when the missing rate r is low. As the missing rate r increases, LNLN shows a significant improvement in all evaluation metrics. This demonstrates LNLN's ability to effectively model data affected by noise of varying intensity. In general, models augmented with noisy data in training usually cannot achieve state-of-the-art performance when the noise is low due to differences in data distribution. It is also worthwhile to study in the future how to balance the performance of models under different levels of noise.

In addition, when r is high, many models converge in their performance on metrics such as Acc-7 and Acc-5 (see Appendix A.6). This is due to the fact that these models exhibit lazy behavior, tending to predict inputs as categories with a higher number of samples in the training set. In such cases, we

believe that the models do not actually learn effective knowledge, even if their performance appears good.

Neutral

Negative

Weakly

Negative

Strongly

Negative

Strongly

Positive

Weakly

Positive

Positive

Neutral

Negative

Weakly

Negative

Strongly

Negative

Strongly

Positive

Weakly

Positive

Positive

Positive

Weakly

Positive

Neutral

Negative

Weakly

Negative

Strongly

Negative

Strongly

Positive

Training Set

Validation set

Test set

(a) Data distribution of MOSI dataset

Negative

Neutral

Positive

Weakly

Positive

Strongly

Positive

Weakly

Negative

Strongly

Negative

Neutral

Negative

Weakly

Positive

Positive

Strongly

Positive

Strongly

Negative

Weakly

Negative

Negative

Neutral

Weakly

Negative

Strongly

Negative

Strongly

Positive

Weakly

Positive

Positive

(b) Data distribution of MOSEI dataset

Neutral

Positive

Weakly

Positive

Weakly

Negative

Negative

Weakly

Positive

Positive

Neutral

Negative

Weakly

Negative

Weakly

Negative

Weakly

Positive

Positive

Neutral

Negative

(c) Data distribution of SIMS dataset

A.6 Visualization of Confusion Matrix

To verify the effectiveness of the method, we visualized the confusion matrix of several representative methods on the MOSI, MOSEI, and SIMS datasets in Figure 5, Figure 6, and Figure 7, respectively. Obviously, as the missing rate r increases, the predictions of all methods on all datasets tend to favor certain categories. This phenomenon indicates that the models hardly learn useful knowledge for prediction but instead ensure high accuracy by being lazy. For example, on the MOSI dataset,

Self-MM predicts all samples as weak negative when the missing rate is 0.9. Although the accuracy of Self-MM is high when the missing rate r is high, it does not demonstrate that the model is more robust.

In addition, TETFN and TFR-Net tend to randomly predict within the negative and weak negative categories under high noise, indicating that their lazy behavior is not as severe as methods like Self-MM, MMIM, and MISA. Unlike other methods, although LNLN also suffers from the impact of data missing, the model's lazy behavior is less severe. For example, at a missing rate of 0.9, the model is still attempting to predict the samples, and there is no case of predictions favoring a particular category to an extreme extent.

From Figure 6 and Figure 7, we can see that a similar phenomenon also occurs in the MOSEI and SIMS datasets. These observations demonstrate that methods targeting complete data often fail when modeling highly incomplete data. Therefore, the design of methods specifically for modeling incomplete data is necessary.

missing rate

𝑟

= 0

missing rate

𝑟

= 0.1

missing rate

𝑟

= 0.5

missing rate

𝑟

= 0.9

CENET

Self-MM

TETFN

TFR-Net

LNLN

ALMT

CENET

Self-MM

TETFN

TFR-Net

LNLN

missing rate

𝑟

= 0

missing rate

𝑟

= 0.1

missing rate

𝑟

= 0.5

missing rate

𝑟

= 0.9

ALMT

CENET

Self-MM

TETFN

TFR-Net

LNLN

missing rate

𝑟

= 0

missing rate

𝑟

= 0.1

missing rate

𝑟

= 0.5

missing rate

𝑟

= 0.9

ALMT

A.7 Visualization of Representations

Figure 8 shows the language feature H 1 l extracted without applying data missing and the corrected dominant feature H 1 d with different data missing rates. we can see that the model tends to learn H 1 d , which is consistent with the distribution of the H 1 l , indicating that LNLN can complement the information of the language modality when data is missing. Additionally, when the missing rate r is 1.0 (no valid information for the input), it becomes difficult for the model to learn H 1 d that is consistent with the language feature. This also suggests that the generator is indeed trying to learn H 1 d for fusion in DMML, rather than just guessing.

(a) missing rate

𝑟

= 0

(b) missing rate

𝑟

= 0.2

(c) missing rate

𝑟

= 0.4

(d) missing rate

𝑟

= 0.6

(e) missing rate

𝑟

= 0.8

(f) missing rate

𝑟

= 1.0

𝐻

!

$

𝐻

"

$

w/o missing

𝐻

!

$

𝐻

"

$

w/o missing

𝐻

!

$

𝐻

"

#

w/o missing

𝐻

!

$

𝐻

"

$

w/o missing

𝐻

!

$

𝐻

"

$

w/o missing

𝐻

!

$

𝐻

"

#

w/o missing

A.8 Analysis of the generalization to modality missing scenarios

To explore the generalization of the method, we further evaluated it in the special case of data missing, i.e., modality missing scenarios. The overall performance of the model is shown in Table 11 and Table 12. The detailed performance of the model is shown in Table 13 and Table 14. It is evident that LNLN, ALMT, TETFN, and Self-MM all exhibit excellent generalization.

We believe that the Dominant Modality Correction (DMC) proposed in LNLN, the Adaptive Hypermodality Learning (AHL) proposed in ALMT, and the Unimodal Label Generation Module (ULGM) proposed/used in Self-MM and TETFN deserve more research attention in the future. These modules and their corresponding ideas have significant reference value for achieving robust Multimodal Sentiment Analysis (MSA).

However, we also found that many methods converge to the same value when the noise is high. This is similar to the phenomenon observed in previous data missing scenarios, where many models exhibited lazy behavior. Therefore, handling modality missing and improving model robustness in different noise scenarios remain critical challenges that need to be addressed in the field of MSA.

B Social Impacts

Our proposed LNLN has a wide range of applications in real-world scenarios, such as healthcare and human-computer interaction. However, it might also be misused to monitor individuals based on their affections for illegal purposes, potentially posing negative social impacts.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. They clearly outline the capabilities of the proposed LNLN and its performance in handling data missing scenarios, as well as emphasizing the comprehensive evaluation of existing methods and their significance in improving robust Multimodal Sentiment Analysis.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations of the work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions ( e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The article does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper reports the network structure required for reproduction as well as the hyperparameter settings. In addition, as mentioned in the abstract, the code is available on GitHub.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The datasets used in the thesis are open datasets and can be accessed by anyone upon request. As mentioned in the abstract, the code is available on GitHub.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Optimizer selection, data partitioning, and other details are described. Hyperparameters are chosen empirically.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: As shown in 8, we report the std/fluctuation of several representative methods on the mosi dataset. It shows the overall stability of the different methods in the face of noise impacts of different levels.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We report this information in Section Implementation Details.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This study meets the requirements of moral and ethical norms.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We describe this in the appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The used data has been cited in the paper. As mentioned in the abstract, the code is available on GitHub..

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: The dataset used is a public dataset, and the code is available on GitHub..

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve above problems.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This study does not involve the above issues and the datasets used are all publicly available datasets.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper proposes a Language-dominated Noise-resistant Learning Network (LNLN) to enhance the robustness of Multimodal Sentiment Analysis (MSA) under conditions of incomplete data. The model leverages the language modality, considered dominant due to its rich sentiment cues, to correct and reconstruct missing data from other modalities (visual and audio). Comprehensive experiments are conducted on datasets such as MOSI, MOSEI, and SIMS, demonstrating the superior performance of LNLN compared to existing baselines.

STRENGTHS:
Comprehensive Evaluation: The experiments are thorough, covering multiple datasets and varying conditions of data incompleteness. This adds to the validity of the findings and allows for a fair comparison with existing methods.

Clear Motivation and Hypothesis: The paper clearly explains the rationale behind focusing on the language modality as the dominant one and provides a logical progression from problem identification to solution proposal.

Detailed Methodology: The methodological section is detailed, providing sufficient information for replication. The incorporation of adversarial learning and dynamic weighting strategies is well-explained.

Significant Performance Improvement: Empirical results indicate that LNLN consistently outperforms other methods, showcasing its robustness and effectiveness in real-world scenarios.

WEAKNESSES:
Real-World Application Scenarios: Discussing potential real-world applications and limitations of LNLN would enhance the paper's practical relevance.

QUESTIONS:
How to balance the hyperparameters of the loss function is not clear.

LIMITATIONS:
Discuss potential real-world applications and scenarios where LNLN can be particularly beneficial, as well as its limitations in these contexts.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses data incompleteness in Multimodal Sentiment Analysis (MSA) by presenting the Language-dominated Noise-resistant Learning Network (LNLN). By considering language as the dominant modality, LNLN introduces a Dominant Modality Correction (DMC) module and Dominant Modality-Based Multimodal Learning (DMML) module to enhance robustness against noise. Extensive experiments under random data missing scenarios on datasets like MOSI, MOSEI, and SIMS demonstrate LNLN’s superior performance. The method ensures quality representations of the dominant modality and offers improved uniformity, transparency, and fairness in evaluations, consistently outperforming existing baselines across various challenging metrics.\n\nSTRENGTHS:\nStrength:\n1. The paper addresses the issue of incomplete data in multimodal sentiment analysis, a prevalent and valuable research problem in real-world applications.\n\n2. The manuscript is well-structured, with a clear and logical flow of ideas. The organization facilitates understanding and ensures that the arguments and methodologies are presented in a coherent manner.\n\nWEAKNESSES:\nWeakness：\n\n1. The contributions summarized in the introduction are not clearly articulated. Additionally, using language as the primary information carrier to reconstruct incomplete data is a concept frequently employed in multimodal sentiment analysis (MSA). What sets your method apart from other similar approaches?\n\n2. The so-called \"Adaptive Hyper-modality Learning\" essentially involves using transformers to facilitate interactions between different modalities. This interaction process appears to lack algorithmic innovation and seems to adopt a rather naive approach.\n\n3. The primary structure of the Reconstructor is also based on transformers. Why did you choose transformers for reconstruction?\n\n4. The entire algorithm's architecture heavily relies on transformers, but the transformers used in different modules have not been specifically tailored to the characteristics of this task.\n\n5. In the experimental section, although LNLN demonstrates generally good performance, there is no in-depth comparative analysis with the most similar baseline, ALMT. Since LNLN is an improvement based on ALMT, such a detailed analysis is essential.\n\n6. Ablation studies and other experiments were only conducted on the MOSI dataset. Why were these not performed on all datasets?\n\n7. The experimental section lacks visualizations or case study to demonstrate the effectiveness of the algorithm, making it difficult to fully substantiate the algorithm's efficacy.\n\nQUESTIONS:\nWeakness：\n\n1. The contributions summarized in the introduction are not clearly articulated. Additionally, using language as the primary information carrier to reconstruct incomplete data is a concept frequently employed in multimodal sentiment analysis (MSA). What sets your method apart from other similar approaches?\n\n2. The so-called \"Adaptive Hyper-modality Learning\" essentially involves using transformers to facilitate interactions between different modalities. This interaction process appears to lack algorithmic innovation and seems to adopt a rather naive approach.\n\n3. The primary structure of the Reconstructor is also based on transformers. Why did you choose transformers for reconstruction?\n\n4. The entire algorithm's architecture heavily relies on transformers, but the transformers used in different modules have not been specifically tailored to the characteristics of this task.\n\n5. In the experimental section, although LNLN demonstrates generally good performance, there is no in-depth comparative analysis with the most similar baseline, ALMT. Since LNLN is an improvement based on ALMT, such a detailed analysis is essential.\n\n6. Ablation studies and other experiments were only conducted on the MOSI dataset. Why were these not performed on all datasets?\n\n7. The experimental section lacks visualizations or case study to demonstrate the effectiveness of the algorithm, making it difficult to fully substantiate the algorithm's efficacy.\n\nLIMITATIONS:\nThe algorithm's structure is overly reliant on transformers, and there is a lack of explanation as to why transformers were chosen for various operations, especially in relation to the specific characteristics of this task. Additionally, the paper introduces multiple loss functions; however, the effectiveness and necessity of each loss function are not thoroughly justified or demonstrated. This raises questions about whether each loss is indeed contributing as intended to the overall performance of the model.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents LNLN, which aims to address the challenge of data incompleteness in real-world scenarios caused by sensor failures or automatic speech recognition issues. The core idea is that even if other modalities are missing, the system can still work if the information from the dominant modality is complete. Therefore, LNLN aims to augment the dominant modality (text). By focusing on text, the method uses other modalities to reconstruct missing textual information, thereby enhancing robustness.\n\nCore Contributions:\n\n1. Evaluation of MSA methods with missing data.\n2. The LNLN method.\n\nSTRENGTHS:\nThe paper introduces LNLN to improve the robustness of multimodal sentiment analysis (MSA) by reinforcing the language modality. The Dominant Modality Correction (DMC) module and Dominant Modality-based Multimodal Learning (DMML) module enhance the model's robustness in various noisy scenarios by ensuring the quality of dominant modality representations.\n\nComprehensive experiments are conducted on multiple datasets (e.g., MOSI, MOSEI, and SIMS), simulating random data missing scenarios, and comparing with existing methods. The method provides additional consistency and achieves good performance.\n\nWEAKNESSES:\nThe evaluation is indeed useful and addresses the gap in existing MSA methods with missing data. However, the innovation is limited and not sufficient to pave the way for future research, resulting in a modest contribution.\n\nThe performance improvement is not significant, and in some cases, it does not achieve state-of-the-art (SOTA) results.\n\nBased on my understanding, Section 3.4 is essentially an application of GAN, which does not highlight significant innovation.\n\nQUESTIONS:\nShould the citation format be \"cite\" instead of \"citet\"?\n\nIn line 45, is your hypothesis supported by analysis, or is it first hypothesized and then experimentally validated?\n\nHow does the Completeness Check in line 168 work, and what role does the randomly initialized $H_{cc}$ play?\n\nLIMITATIONS:\nSee weakness and question", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-04"}