Towards Robust Multimodal Sentiment Analysis with Incomplete Data

Authors: haoyu  zhang
wenbin  wang
tianshu  yu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 00:26:03

Abstract: The field of Multimodal Sentiment Analysis (MSA) has recently witnessed an emerging direction seeking to tackle the issue of data incompleteness. Recognizing that the language modality typically contains dense sentiment information, we consider it as the dominant modality and present an innovative Language-dominated Noise-resistant Learning Network (LNLN) to achieve robust MSA. The proposed LNLN features a dominant modality correction (DMC) module and dominant modality based multimodal learning (DMML) module, which enhances the model's robustness across various noise scenarios by ensuring the quality of dominant modality representations. Aside from the methodical design, we perform comprehensive experiments under random data missing scenarios, utilizing diverse and meaningful settings on several popular datasets (e.g., MOSI, MOSEI, and SIMS), providing additional uniformity, transparency, and fairness compared to existing evaluations in the literature. Empirically, LNLN consistently outperforms existing baselines, demonstrating superior performance across these challenging and extensive evaluation metrics.

Main paper: 
 introduction: 
 The field of Multimodal Sentiment Analysis (MSA) is at the vanguard of human sentiment identification by assimilating heterogeneous data types, such as video, audio, and language. Its applicability spans numerous fields, prominent among healthcare and human-computer interaction (Jiang et al., 2020) . MSA has become essential in enhancing both the precision and the robustness of sentiment analysis by drawing sentiment cues from diverse perspectives.Changing trends in recent research has taken a turn towards modeling data in natural scenarios from laboratory conditions (Tsai et al., 2019; Hazarika et al., 2020; Yu et al., 2021; Zhang et al., 2023) . The shift has created a wider application space in the real-world for MSA, though concerns arise due to problems like sensor failures and problems with Automatic Speech Recognition (ASR), leading to inconsistencies such as incomplete data in real-world deployment.Numerous impactful solutions have been proposed against this primary concern of incomplete data in multimodal sentiment analysis. For instance, Yuan et al. (2021) introduced a Transformerbased feature reconstruction mechanism, TFR-Net, aiming to strengthen the robustness of the model handling random missing in unaligned multimodal sequences via reconstructing missing data.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. They clearly outline the capabilities of the proposed LNLN and its performance in handling data missing scenarios, as well as emphasizing the comprehensive evaluation of existing methods and their significance in improving robust Multimodal Sentiment Analysis.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
multimodal sentiment analysis: 
 Multimodal Sentiment Analysis (MSA) methods can be categorized into Context-based MSA and Noise-aware MSA, depending on the modeling approach. Most of previous works (Zadeh et al., 2017; Tsai et al., 2019; Mai et al., 2020; Hazarika et al., 2020; Liang et al., 2020; Rahman et al., 2020; Yu et al., 2021; Han et al., 2021; Lv et al., 2021; Yang et al., 2022; Guo et al., 2022; Zhang et al., 2023) can be classified to Context-based MSA. This line of work primarily focuses on learning unified multimodal representations by analyzing contextual relationships within or between modalities. For example, Zadeh et al. (2017) explore computing the relationships between different modalities using the Cartesian product. Tsai et al. (2019) utilize pairs of Transformers to model long dependencies between different modalities. Yu et al. (2021) propose generating pseudo-labels for each modality to further mine the information of consistency and discrepancy between different modalities. Despite these advances, context-based methods are usually suboptimal under varying levels of noise effects (e.g. random data missing). Several recent works (Mittal et al., 2020; Yuan et al., 2021 Yuan et al., , 2024;; Li et al., 2024) have been proposed to tackle this issue. For example, Mittal et al. (2020) introduce a modality check step to distinguish invalid and valid modalities, achieving higher robustness. Yuan et al. (2024) propose learning a unified joint representation between constructed "original-noisy" instance pairs. Although there have been some advances in improving the model's robustness under noise scenarios, no extant method has provided a comprehensive and in-depth comparative analysis.Context-based MSA and Noise-aware MSA differ in their approaches to robust representation learning. In Context-based MSA, robust representation learning typically relies on modeling intraand inter-modality relationships. For instance, Hazarika et al. (2020) and Yang et al. (2022) apply feature disentanglement to each modality, modeling multimodal representations from multiple feature subspaces and perspectives. Yu et al. (2021) and Liang et al. (2020) explore self-supervised learning and semi-supervised learning to enhance multimodal representations, respectively. Tsai et al. (2019) and Rahman et al. (2020) introduce Transformer to learn the long dependencies of modalities. Zhang et al. (2023) devise a language-guided learning mechanism that uses modalities with more intensive sentiment cues to guide the learning of other modalities. In contrast, Noise-aware MSA focuses more on perceiving and eliminating the noise present in the data. For example, Mittal et al. (2020) design a modality check module based on metric learning and Canonical Correlation Analysis (CCA) to identify the modality with greater noise. Yuan et al. (2021) design a feature reconstruction network to predict the location of missing information in sequences and reconstruct it. Yuan et al. (2024) introduce adversarial learning (Goodfellow et al., 2014) to perceive and generate cleaner representations.In this work, the LNLN belongs to the noise-aware MSA category. Inspired by Zhang et al. (2023) , we explore the capability of language-guided mechanisms in resisting noise and aim to provide new perspectives for the study of MSA in noisy scenarios.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We have discussed the limitations of the work.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
overview: 
 The overall pipeline for the proposed Language-dominated Noise-resistant Learning Network (LNLN) in robust multimodal sentiment analysis is illustrated in Figure 1 . As depicted, a crucial initial step involves forming a multimodal input with random data missing, which sets the stage for LNLN training. Once the input is prepared, LNLN first utilizes an embedding layer to standardize the dimension of each modality, ensuring uniformity. Recognizing that language is the dominant modality in MSA (Zhang et al., 2023) , a specially designed Dominant Modality Correction (DMC) module employs adversarial learning and a dynamic weighted enhancement strategy to mitigate noise impacts. This module first enhances the quality of the dominant feature computed from the language modality and then integrates them with the auxiliary modalities (visual and audio) in the dominant modality based multimodal learning (DMML) module for effective multimodal fusion and classification. This process significantly bolsters LNLN's robustness against various noise levels. Moreover, to refine the network's capability for fine-grained sentiment analysis, a simple reconstructor is implemented to reconstruct missing data, further enhancing the system's robustness.Given the challenges of random data missing, we have constructed data sets that simulate these conditions based on MOSI, MOSEI, and SIMS datasets.Random Data Missing. Following the previous method (Yuan et al., 2021) , for each modality, we randomly erased changing proportions of information (from 0% to 100%). Specifically, for visual and audio modalities, we fill the erased information with zeros. For language modality, we fill the erased information with [UNK] which indicates the unknown word in BERT (Kenton and Toutanova, 2019) .Multimdoal Input. For each sample in the dataset, we incorporate data from three modalities: language, audio, and visual data. Consistent with previous works (Zhang et al., 2023) , each modality is processed using widely-used tools: language data is encoded using BERT (Kenton and Toutanova, 2019) , audio features are extracted through Librosa (McFee et al., 2015) , and visual features are obtained using OpenFace (Baltrusaitis et al., 2018) . These pre-processed inputs are represented as sequences, denoted by U 0 m ∈ R Tm×dm , where m ∈ {l, v, a} represents the modality type (l for language, v for visual, a for audio), T m indicates the sequence length and d m refers to the dimension of each modality's vector. With obtained U 0 m , we apply random data missing to U 0 m , thus forming the noise-corrupted multimodal input U 1 m ∈ R Tm×dm .I am fine.Feature Extraction… … … … … … Data Missing … … … … … … Multimodal Input 𝐻 ! " 𝐻 # " 𝐻 $ " Language Embedding Visual Embedding Audio Embedding Adaptive Hyper-modality Learning Effectiveness Discriminator ℒ $'# ℒ )) ℒ *% Proxy Dominant Feature Generation 𝑤 1 -𝑤 𝑈 ! Figure 1: Overall pipeline. Note: H 0 l , H 0 v , H 0 a , H cc, and H 0 p are randomly initialized learnable vectors.Inspired by previous work ALMT, we hypothesize that model robustness improves when the integrity of the dominant modality is preserved despite varying noise levels. We improve ALMT based on the designed DMC module and Reconstructor, thus implementing dominant modality based DMML module for sentiment analysis under random data missing scenarios. Here, we mainly introduce the parts that differ from ALMT. Further details are available in Zhang et al. (2023) .Modality Embedding. For multimodal input U 1 m , we employ an Embedding Encoder E 1 m with two Transformer encoder layers to extract and unify the feature. Each modality begins with a randomly initialized low-dimensional token H 0 m ∈ R T ×dm . These tokens are then processed by the Transformer encoder layer, embedding essential modality information and producing unified features, represented as H 1 m ∈ R T ×d . The process is formalized by:EQUATIONwhere E 1 m (•) extracts features for each modality, and concat(•) represents the concatenation operation. Adaptive Hyper-modality Learning. In the original ALMT, each Adaptive Hyper-modality Learning layer contains a Transformer and two multi-head attention (MHA) modules. These are applied to learn language representations at different scales and hyper-modality representations from visual and audio modalities, guided by the language modality. Considering the possibility of severe interference in the language modality (i.e. dominant modality) due to random data missing, we designed a Dominant Modality Correction (DMC) module to generate the proxy dominant feature H 1 p and construct corrected dominate feature H 1 d (more details can be found in Section 3.4). Specifically, the process of learning corrected dominated representation H i d at different scales can be described as:EQUATIONwhere i ∈ {2, 3} means the i-th layer of Adaptive Hyper-modality Learning module, E i m (•) is the i-th Transformer encoder layer, H i d ∈ R T ×d is corrected dominated feature at different scale. To learn the hyper-modality representation, the corrected dominated feature and audio/visual features are used to calculate Query and Key/Value, respectively. Briefly, the process can be written as follows:EQUATIONwhere MHA(•) represents multi-head attention, H i hyper ∈ R T ×d is the hyper-modality feature. Note that the feature H 0 hyper ∈ R T ×d is a random initialized vector.Multimodal Fusion and Prediction. With obtained H 3 d and H 3 hyper , a Transformer encoder with a classifier at a depth of 4 layers is employed for multimodal fusion and sentiment prediction:EQUATIONwhere ŷ is the sentiment prediction.This module consists of two steps, i.e., completeness check of dominant modality and proxy dominant feature generation using adversarial learning (Ganin and Lempitsky, 2015; Goodfellow et al., 2014) .Completeness Check. We apply an encoder E cc that consists of a Transformer encoder with a depth of two layers and a classifier for completeness check. For example, if the missing rate of dominate modality is 0.3, the label of completeness is 0.7. This completeness prediction w can be obtained as follows:EQUATIONwhere H cc ∈ R T ×d is a randomly initialized token for completeness prediction. We optimize this process using L2 loss:EQUATIONwhere N b is the number of samples in the training set, ŵk is the label of completeness of k-th sample.Proxy Dominant Feature Generation. With the randomly initialized feature H 0 p ∈ R T ×d , visual feature H 1 v and audio feature H 1 a , we employ a Proxy Dominant Feature Generator E DF G , which consists of two Transformer encoder layers. This setup generates the proxy dominant features H 1 p ∈ R T ×d , designed to complement and correct the dominant modality. The corrected dominant feature H 1 d ∈ R T ×d is calculated by combining H 1 p and the language feature H 1 l , weighted by the predicted completeness w:EQUATIONEQUATIONwhere θ DF G denotes the parameters of the Proxy Dominant Feature Generator E DF G .To ensure that the agent feature offers a distinct perspective from the visual and audio features, we utilize an effectiveness discriminator D. This discriminator includes a binary classifier and a Gradient Reverse Layer (GRL) (Ganin and Lempitsky, 2015) and is tasked with identifying the origin of the agent features:EQUATIONwhere θ D represents the parameters of the effectiveness discriminator D, and ŷp indicates the prediction of whether the input feature originates from the language modality.In practice, the generator and the discriminator engage in an adversarial learning structure. The discriminator aims to identify whether the features are derived from the language modality, while the generator's objective is to challenge the discriminator's ability to make accurate predictions. This dynamic is encapsulated in the adversarial learning objective:EQUATIONwhere N b is the number of samples in the training set, and y k p indicates the label determining whether the input feature for the k-th sample originates from the visual or audio modality.Our experiments demonstrate that reconstructing missing information can significantly enhance regression metrics. More details about this are shown in Table 4 . To address this, we have developed a reconstructor, denoted as E rec , which comprises two Transformer layers designed to effectively rebuild missing information of each modality. The operational equation for the reconstructor is:Û 0 m = E rec m U 1 m (11)where Û 0 m is the reconstructed feature corresponding to the feature U 0 m . To optimize the performance of the reconstructor, we apply an L2 loss function:EQUATION)where U 0 m k and Û 0 m k represent the original and reconstructed features with missing information for the k-th sample, respectively. This loss function helps minimize the discrepancies between the original and reconstructed features, thereby improving the accuracy of other components, such as Dominant Modality Correction and final sentiment prediction.To sum up, our method involves four learning objectives, including a completeness check loss L cc , an adversarial learning loss L adv for proxy dominant feature generation, a reconstruction loss L rec and one final sentiment prediction loss L sp . The sentiment prediction loss L sp can be described as:EQUATIONTherefore, the overall loss L can be written as:EQUATIONwhere α, β, γ and δ are hyperparameters. On MOSI and MOSEI datasets, we empirically set them to 0.9, 0.8, 0.1, and 1.0, respectively. On the SIMS dataset, we empirically set them to 0.9, 0.6, 0.1, and 1.0, respectively. 
experiments and analysis: 
 In this section, we provide a comprehensive and fair comparison between the proposed LNLN and previous representative MSA methods on MOSI (Zadeh et al., 2016) , MOSEI (Zadeh et al., 2018) and SIMS (Yu et al., 2020) For a fair and comprehensive evaluation, we experiment ten times, setting the missing rates r to predefined values from 0 to 0.9 with an increment of 0.1. For instance, 50% of the information is randomly erased from each modality in the test data when r = 0.5. Unlike previous works (Yuan et al., 2021 (Yuan et al., , 2024)) , we did not evaluate at r = 1.0, as this would imply complete data erasure from each modality, rendering the experiment non-informative. With the obtained results of each missing rate, we compute the average value as the model's overall performance under different levels of noise.For evaluation criteria, we report the binary classification accuracy (Acc-2), the F1 score associated with Acc-2, and the mean absolute error (MAE). For Acc-2, we calculated accuracy and F1 in two ways: negative/positive (left-side value of /) and negative/non-negative (right-side value of /) on the MOSI and MOSEI datasets, respectively. Additionally, we provide the three-class accuracy (Acc-3), the seven-class accuracy (Acc-7), and the correlation of the model's prediction with humans (Corr) on the MOSI dataset. For the SIMS dataset, we report Acc-3, the five-class accuracy (Acc-5), and Corr. Due to the distinct focus of regression and classification metrics on different aspects of model performance, a model achieving the lowest error on regression metrics may not necessarily exhibit optimal performance on classification metrics. To comprehensively reflect the model capabilities, we select the best-performing checkpoint for each type of metric across all models in the comparisons, thus capturing the peak performance of both regression and classification aspects independently.We used PyTorch 2.2.1 to implement the method. The experiments were conducted on a PC with an AMD EPYC 7513 CPU and an NVIDIA Tesla A40. To ensure consistent and fair comparisons across all methods, we conducted each experiment three times using fixed random seeds of 1111, 1112, and 1113. Details of the hyperparameters are shown in Table 1 . In addition, the result of MISA, Self-MM, MMIM, CENET, TETFN, and TFR-Net is reproduced by the authors from open source code in the MMSA2 (Mao et al., 2022) , which is a unified framework for MSA, using default hyperparameters.The result of ALMT is reproduced by the authors from open source code on Github3 . 1111,1112,1113 1111,1112,1113 1111,1112,1113 Tables 2 and 3 show the robustness evaluation results on the MOSI, MOSEI, and SIMS datasets.As shown in Table 2 , LNLN achieves state-of-the-art performance on most metrics. On the MOSI dataset, LNLN achieved a relative improvement of 9.46% on Acc-7 compared to the sub-optimal result obtained by MMIM, demonstrating the robustness of LNLN in the face of different noise effects. However, on the MOSEI dataset, LNLN achieves only sub-optimal performance on metrics such as Acc-7 and Acc-5. After analyzing the data distribution (see Appendix A.3) and the confusion matrix (see Appendix A.6), we believe that this is because LNLN does not overly bias the neutral category, which has a disproportionate number of samples in noisy scenarios. As shown in Table 3 , LNLN achieves an improvement in F1 on the SIMS dataset, with a relative improvement of 9.17% on F1 compared to the sub-optimal result obtained by ALMT. Similar to CENET's performance on the MOSEI dataset, TETFN on the SIMS dataset has a tendency to predict inputs as weakly negative categories with high sample sizes, resulting in seemingly better performance on some metrics. Additionally, as shown in Figure 2 , we present the performance curves of several advanced methods under varying missing rates. The results demonstrate that the proposed LNLN consistently outperforms others across most scenarios, showing its robustness under different missing rates.Overall, LNLN attempts to make predictions in the face of highly noisy inputs without the severe lazy behavior observed in other models, which often leads to predictions heavily biased towards a certain category. This demonstrates that LNLN shows strong robustness and competitiveness across various datasets and noise levels, highlighting its effectiveness in multimodal sentiment analysis. To verify the role of different components, we present the results after subtracting each component on the MOSI dataset separately. As shown in Table 4 , some metrics show an upward trend in performance when certain modules are removed. On one hand, we believe this is due to randomness in noisy scenarios. On the other hand, the same hyperparameters cannot make the model perform optimally across all metrics. Moreover, the performance of LNLN decreases relatively slightly when the DMC and reconstructor are removed individually. However, there is a significant drop in performance when both are removed, e.g., 4.72% decrease in Acc-7 and 5.87% decrease in Acc-5. Performance drops further after replacing DMML with concatenation fusion, which also proves that DMML plays an important role in MSA. Additionally, after removing the noisy data used for training, the performance of LNLN also shows a significant decrease. This observation indicates that LNLN needs to learn to be aware of noise with the help of noisy data. Overall, both the presence of noisy data and well-designed components are crucial for robust MSA in scenarios with random missing data. As shown in Table 5 , we present the ablation results after subtracting each regularization on the MOSI dataset separately. Obviously, after each regularization is removed, LNLN shows a decrease in performance on most metrics. This demonstrates that regularization contributes to the learning of the model. Similar to the results in Table 4 , there are a few metrics that improve after the removal of regularization. We believe this is mainly due to the difficulty of optimizing a single set of hyperparameters to achieve the best performance across all metrics. To verify the role of different modalities, we present the ablation results after subtracting each modality from the MOSI dataset. As shown in Table 6 , all modalities contribute to the final performance. Notably, when the language modality is removed, the performance of LNLN shows a significant decrease, while removing the visual and audio modalities results in a smaller decrease. This indicates that the language modality plays a crucial role in the MSA dataset. Moreover, we observe that LNLN converges on most metrics even when the visual and audio modalities are removed. This is due to the model's lazy behavior, where LNLN tends to fix its predictions to a certain category. More details can be found in Appendix A.3 and A.6.In this work, we conduct a comprehensive evaluation of current advanced methods, the results of which have triggered some thoughts. Specifically, we find that all methods fail to accurately predict the inputs in high missing rate scenarios because most of them are designed specifically for complete data (see Appendix A.6 for more details). However, some methods show relatively more robust performance in low missing rate scenarios and some specific scenarios (see Appendix A.8), such as Self-MM, TETFN, and ALMT. We believe that some of the ideas in these methods may be useful for future research on more robust MSA for incomplete data.For example, the Unimodal Label Generation Module (ULGM) (Yu et al., 2021) used in both Self-MM and TETFN facilitates the modeling of incomplete data. Due to the random data missing, the temporal and structured affective information in the data is corrupted, making it difficult for the models to perceive consistency and variability information between different modalities. Using ULGM to generate pseudo-labels as an additional supervisory signal for the models may directly enable the model to learn the relationship between different modalities. For ALMT, we believe that its Adaptive Hyper-modality Learning (AHL) module facilitates the modeling of robust affective representations. AHL mainly uses the dominant modality to compute query vectors and applies multi-head attention to query useful sentiment cues from other modalities as a complement to the dominant modality. This process may reduce the difficulty of multimodal fusion by avoiding the introduction of sentiment-irrelevant information to some extent. Our proposed LNLN, based on this idea from ALMT, proves the effectiveness of the method.Additionally, we explore the performance of these methods in modality missing scenarios, which is a special case of random data missing where the partial modality missing rate r = 1.0. We find that when language modality is missing, the performance of many models decreases significantly and converges to the same value. More details can be seen in Appendix A.8. 
conclusion and future work: 
 In this paper, a novel Language-dominated Noise-resistant Learning Network (LNLN) is proposed for robust MSA. Due to the collaboration between Dominate Modality Correction (DMC) module, the Dominant Modality based Multimodal Learning (DMML) module, and the Reconstructor, LNLN can enhance the dominant modality to achieve superior performance in data missing scenarios with varying levels. Extensive evaluation shows that none of the existing methods can effectively model the data with high missing rates. The related analyses can provide suggestions for other researchers to better handle robust MSA. In the future, we will focus on improving the generalization of the model to handle different types of scenes and varying intensities of noise effectively. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The used data has been cited in the paper. As mentioned in the abstract, the code is available on GitHub.. 
 
 Appendix: We believe there are several limitations to the LNLN. 1) The LNLN achieves good performance in data missing scenarios, but is not always better than all other methods in the modality missing scenarios, which demonstrates the lack of multi-scene generalization capability of the LNLN.2)The data in real-world scenarios is much more complex. In addition to the presence of missing data, other factors need to be considered, such as diverse cultural contexts, varying user behavior patterns, and the influence of platform-specific features on the data. These factors can introduce additional noise and variability, which may require further model adaptation and tuning to handle effectively. 3) Tuning the hyperparameters, particularly those related to the loss functions, can be challenging and may require more sophisticated methods to achieve optimal performance.As shown in Table 7 , we empirically tried different combinations of hyperparameters in the loss function. The results show that the selected parameters can make the LNLN achieve better performance in most metrics. This also demonstrates the effectiveness of the optimization objectives of LNLN. To evaluate the stability of the model, we selected the MOSI dataset for ablation experiments. Specifically, based on the overall performance (mean) in Table 2 , we selected several representative methods to additionally compute the standard deviation. For each method, we first calculate the standard deviation of evaluation results for the three random seeds when the missing rate r ranges from 0 to 0.9, and then average the 10 sets of standard deviation. The final result is shown in Table 8 . We can see that the standard deviation of these methods is not very large in most metrics, indicating that all these methods can guarantee stability in the presence of random noise. It should be noted that LNLN strikes a balance between overall performance and stability.Figure 4 illustrates the data distribution on the MOSI, MOSEI, and SIMS datasets. Significant category imbalances can be seen across all datasets. Additionally, the distributions of the training set, validation set, and test set are not identical on the MOSI and MOSEI datasets. For example, as shown in Figure 4 (a), the percentage of weakly negative samples is 15.5% in the MOSI training set, while this category reaches 22.7% in the test set, representing the highest percentage among all categories. The unbalanced data distribution makes it difficult for the model to perform well in data missing scenarios. Specifically, it is easy for the model to engage in lazy behavior in high-noise scenarios, simply biasing predictions towards categories with a higher proportion in the training set. These unbalanced distributions may further increase the learning difficulty of the model when facing data missing. More details can be found in Appendix A.6.As shown in Figure 3 , we visualize several successful and failed predictions made by LNLN and ALMT from the MOSI dataset for the case study. It shows that LNLN can perceive sentiment cues in challenging samples, which demonstrates its ability to capture sentiment cues in noisy scenarios. However, for inputs with high missing rates (e.g., the third example in the figure), both LNLN and ALMT fail to make correct predictions. We believe this is due to the high loss of valid information in the multimodal input, making accurate predictions difficult. In practice, random data missing is applied to the original input sequence, as described in Section 3.Tables 9 and 10 show the details of robust comparison on the MOSI, MOSEI, and SIMS datasets, respectively. We observe that Self-MM and TETFN have a clear advantage in many evaluation metrics when the missing rate r is low. As the missing rate r increases, LNLN shows a significant improvement in all evaluation metrics. This demonstrates LNLN's ability to effectively model data affected by noise of varying intensity. In general, models augmented with noisy data in training usually cannot achieve state-of-the-art performance when the noise is low due to differences in data distribution. It is also worthwhile to study in the future how to balance the performance of models under different levels of noise.In addition, when r is high, many models converge in their performance on metrics such as Acc-7 and Acc-5 (see Appendix A.6). This is due to the fact that these models exhibit lazy behavior, tending to predict inputs as categories with a higher number of samples in the training set. In such cases, we believe that the models do not actually learn effective knowledge, even if their performance appears good. To verify the effectiveness of the method, we visualized the confusion matrix of several representative methods on the MOSI, MOSEI, and SIMS datasets in Figure 5 , Figure 6 , and Figure 7 , respectively. Obviously, as the missing rate r increases, the predictions of all methods on all datasets tend to favor certain categories. This phenomenon indicates that the models hardly learn useful knowledge for prediction but instead ensure high accuracy by being lazy. For example, on the MOSI dataset, Self-MM predicts all samples as weak negative when the missing rate is 0.9. Although the accuracy of Self-MM is high when the missing rate r is high, it does not demonstrate that the model is more robust.In addition, TETFN and TFR-Net tend to randomly predict within the negative and weak negative categories under high noise, indicating that their lazy behavior is not as severe as methods like Self-MM, MMIM, and MISA. Unlike other methods, although LNLN also suffers from the impact of data missing, the model's lazy behavior is less severe. For example, at a missing rate of 0.9, the model is still attempting to predict the samples, and there is no case of predictions favoring a particular category to an extreme extent.From Figure 6 and Figure 7 , we can see that a similar phenomenon also occurs in the MOSEI and SIMS datasets. These observations demonstrate that methods targeting complete data often fail when modeling highly incomplete data. Therefore, the design of methods specifically for modeling incomplete data is necessary. Figure 8 shows the language feature H 1 l extracted without applying data missing and the corrected dominant feature H 1 d with different data missing rates. we can see that the model tends to learn H 1 d , which is consistent with the distribution of the H 1 l , indicating that LNLN can complement the information of the language modality when data is missing. Additionally, when the missing rate r is 1.0 (no valid information for the input), it becomes difficult for the model to learn H 1 d that is consistent with the language feature. This also suggests that the generator is indeed trying to learn H 1 d for fusion in DMML, rather than just guessing. To explore the generalization of the method, we further evaluated it in the special case of data missing, i.e., modality missing scenarios. The overall performance of the model is shown in Table 11 and Table 12 . The detailed performance of the model is shown in Table 13 and Table 14 . It is evident that LNLN, ALMT, TETFN, and Self-MM all exhibit excellent generalization.We believe that the Dominant Modality Correction (DMC) proposed in LNLN, the Adaptive Hypermodality Learning (AHL) proposed in ALMT, and the Unimodal Label Generation Module (ULGM) proposed/used in Self-MM and TETFN deserve more research attention in the future. These modules and their corresponding ideas have significant reference value for achieving robust Multimodal Sentiment Analysis (MSA).However, we also found that many methods converge to the same value when the noise is high. This is similar to the phenomenon observed in previous data missing scenarios, where many models exhibited lazy behavior. Therefore, handling modality missing and improving model robustness in different noise scenarios remain critical challenges that need to be addressed in the field of MSA. Our proposed LNLN has a wide range of applications in real-world scenarios, such as healthcare and human-computer interaction. However, it might also be misused to monitor individuals based on their affections for illegal purposes, potentially posing negative social impacts.NeurIPS Paper Checklist• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [No] Justification: The dataset used is a public dataset, and the code is available on GitHub.. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.& 𝐻 $ && 𝑈 $ &MMSA: https://github.com/thuiar/MMSAALMT: https://github.com/Haoyu-ha/ALMT 


Decision: The paper introduces the Language-dominated Noise-resistant Learning Network (LNLN) for robust Multimodal Sentiment Analysis, designed to address the challenge of incomplete data in real-world applications. LNLN leverages a Dominant Modality Correction module and a Dominant Modality-based Multimodal Learning module to enhance the robustness of sentiment analysis by focusing on the integrity of the language modality, which typically contains the most dense sentiment information. The model is tested across various datasets with random missing data scenarios, consistently outperforming existing methods.

The paper is well-written and easy to follow. The reviewers had some questions and concerns about the evaluation, datasets used in the experiments, technical details, and the motivation behind the paper. During the rebuttal, the authors addressed these issues, and the reviewers were satisfied with the responses and additional experiments. Therefore, I recommend accepting the paper.

Reviews: review: paper_summary: paper_summary: This paper presents LNLN, which aims to address the challenge of data incompleteness in real-world scenarios caused by sensor failures or automatic speech recognition issues. The core idea is that even if other modalities are missing, the system can still work if the information from the dominant modality is complete. Therefore, LNLN aims to augment the dominant modality (text). By focusing on text, the method uses other modalities to reconstruct missing textual information, thereby enhancing robustness.

Core Contributions:

1. Evaluation of MSA methods with missing data.
2. The LNLN method.
strength_weakness: strength_weakness: The evaluation is indeed useful and addresses the gap in existing MSA methods with missing data. However, the innovation is limited and not sufficient to pave the way for future research, resulting in a modest contribution.

The performance improvement is not significant, and in some cases, it does not achieve state-of-the-art (SOTA) results.

Based on my understanding, Section 3.4 is essentially an application of GAN, which does not highlight significant innovation.
questions: questions: Should the citation format be "cite" instead of "citet"?

In line 45, is your hypothesis supported by analysis, or is it first hypothesized and then experimentally validated?

How does the Completeness Check in line 168 work, and what role does the randomly initialized $H_{cc}$ play?
limitations: limitations: See weakness and question
paper_summary: paper_summary: The paper addresses data incompleteness in Multimodal Sentiment Analysis (MSA) by presenting the Language-dominated Noise-resistant Learning Network (LNLN). By considering language as the dominant modality, LNLN introduces a Dominant Modality Correction (DMC) module and Dominant Modality-Based Multimodal Learning (DMML) module to enhance robustness against noise. Extensive experiments under random data missing scenarios on datasets like MOSI, MOSEI, and SIMS demonstrate LNLN’s superior performance. The method ensures quality representations of the dominant modality and offers improved uniformity, transparency, and fairness in evaluations, consistently outperforming existing baselines across various challenging metrics.
strength_weakness: strength_weakness: Weakness：

1. The contributions summarized in the introduction are not clearly articulated. Additionally, using language as the primary information carrier to reconstruct incomplete data is a concept frequently employed in multimodal sentiment analysis (MSA). What sets your method apart from other similar approaches?

2. The so-called "Adaptive Hyper-modality Learning" essentially involves using transformers to facilitate interactions between different modalities. This interaction process appears to lack algorithmic innovation and seems to adopt a rather naive approach.

3. The primary structure of the Reconstructor is also based on transformers. Why did you choose transformers for reconstruction?

4. The entire algorithm's architecture heavily relies on transformers, but the transformers used in different modules have not been specifically tailored to the characteristics of this task.

5. In the experimental section, although LNLN demonstrates generally good performance, there is no in-depth comparative analysis with the most similar baseline, ALMT. Since LNLN is an improvement based on ALMT, such a detailed analysis is essential.

6. Ablation studies and other experiments were only conducted on the MOSI dataset. Why were these not performed on all datasets?

7. The experimental section lacks visualizations or case study to demonstrate the effectiveness of the algorithm, making it difficult to fully substantiate the algorithm's efficacy.
questions: questions: Weakness：

1. The contributions summarized in the introduction are not clearly articulated. Additionally, using language as the primary information carrier to reconstruct incomplete data is a concept frequently employed in multimodal sentiment analysis (MSA). What sets your method apart from other similar approaches?

2. The so-called "Adaptive Hyper-modality Learning" essentially involves using transformers to facilitate interactions between different modalities. This interaction process appears to lack algorithmic innovation and seems to adopt a rather naive approach.

3. The primary structure of the Reconstructor is also based on transformers. Why did you choose transformers for reconstruction?

4. The entire algorithm's architecture heavily relies on transformers, but the transformers used in different modules have not been specifically tailored to the characteristics of this task.

5. In the experimental section, although LNLN demonstrates generally good performance, there is no in-depth comparative analysis with the most similar baseline, ALMT. Since LNLN is an improvement based on ALMT, such a detailed analysis is essential.

6. Ablation studies and other experiments were only conducted on the MOSI dataset. Why were these not performed on all datasets?

7. The experimental section lacks visualizations or case study to demonstrate the effectiveness of the algorithm, making it difficult to fully substantiate the algorithm's efficacy.
limitations: limitations: The algorithm's structure is overly reliant on transformers, and there is a lack of explanation as to why transformers were chosen for various operations, especially in relation to the specific characteristics of this task. Additionally, the paper introduces multiple loss functions; however, the effectiveness and necessity of each loss function are not thoroughly justified or demonstrated. This raises questions about whether each loss is indeed contributing as intended to the overall performance of the model.
paper_summary: paper_summary: The paper proposes a Language-dominated Noise-resistant Learning Network (LNLN) to enhance the robustness of Multimodal Sentiment Analysis (MSA) under conditions of incomplete data. The model leverages the language modality, considered dominant due to its rich sentiment cues, to correct and reconstruct missing data from other modalities (visual and audio). Comprehensive experiments are conducted on datasets such as MOSI, MOSEI, and SIMS, demonstrating the superior performance of LNLN compared to existing baselines.
strength_weakness: strength_weakness: Real-World Application Scenarios: Discussing potential real-world applications and limitations of LNLN would enhance the paper's practical relevance.
questions: questions: How to balance the hyperparameters of the loss function is not clear.
limitations: limitations: Discuss potential real-world applications and scenarios where LNLN can be particularly beneficial, as well as its limitations in these contexts.

Comments: comment: Dear All,

We would like to express our sincere thanks for the time and effort you have invested in reviewing our paper. We have responded to all of the comments and questions raised during the review process and hope that our responses address your concerns adequately.

Thank you for your valuable feedback.

Best regards,

The Authors
Thank you for adjusting your rating and for your support of our work. The feedback is important. We are grateful for the opportunity to improve our paper, and appreciate your time and effort.
Thank you for your response and clarifications. I have adjusted my rating to 7.