KEBR: Knowledge Enhanced Self-Supervised Balanced  Representation for Multimodal Sentiment Analysis

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 04:27:22

Abstract: Multimodal sentiment analysis (MSA) aims to integrate multiple modalities of information to better understand human sentiment. The current research mainly focuses on conducting multimodal fusion and representation learning, which neglects the under-optimized modal representations generated by the imbalance of unimodal performances in joint learning. Moreover, the size of labeled datasets limits the generalization ability of existing supervised models used in MSA. To address the above issues, this paper proposes a knowledge-enhanced self-supervised balanced representation approach (KEBR) to capture common sentimental knowledge in unlabeled videos and explore the optimization issue of information imbalance between modalities. First, a text-based cross-modal fusion method (TCMF) is constructed, which injects the non-verbal information from the videos into the semantic representation of text to enhance the multimodal representation of text. Then, a multimodal cosine constrained loss (MCC) is designed to constrain the fusion of non-verbal information in joint learning to balance the representation of multimodal information. Finally, with the help of sentiment knowledge and non-verbal information, KEBR conducts sentiment word masking and sentiment intensity prediction, so that the sentiment knowledge in the videos is embedded into the pre-trained multimodal representation in a balanced manner. Experimental results on two publicly available datasets MOSI and MOSEI show that KEBR significantly outperforms the baseline, achieving new state-of-the-art results.

Main paper: 
 introduction: 
 Multimodal Sentiment Analysis (MSA) offers a comprehensive understanding of human sentiment by integrating information from text, audio, and visual modalities, which are closer to real-life scenarios where human beings process sentiment [2] . The widespread use of smart devices has led to a significant increase in user opinion videos, which offer abundant data resources and application scenarios for studying MSA tasks, such as depression detection, e-commerce, smart customer service, and human-computer, interaction, among others. With the generation of large amounts of multi-source information, MSA has received increasing attention, and richer information can also help to improve model performance [42] . However, due to the heterogeneous modal gap, achieving human-comparable MSA performance remains still challenging [44] .The majority of previous MSA approaches have concentrated on developing multimodal fusion and representation learning. RNNbased models [6, 20, 21, 51] connect each modal feature to a fusion vector input for subsequent classification or regression. Models based on Transformers [35, 36] are employed to simulate multimodal interactions for reducing the effect of inter-modal differences. The BERT [12, 29, 40, 45] pre-trained language model is employed as an encoder for text modality, thereby enhancing the performance of the MSA task by leveraging BERT's exceptional language representation capabilities. However, a joint model for uniform objective optimization may have its unimodal encoders converging at different rates [40] . One dominant modality could dominate the optimization process, leading to the neglect of other modalities and causing under-optimized representations. This results in modal bias that fails to fully utilize the capabilities of multimodal. To address this issue, some studies control the learning rate of different modalities [24, 26, 38] . However, all of these methods only mitigate the imbalance between the audio-visual modalities, and they all require additional training costs. Numerous studies have demonstrated that text plays a more central role than non-verbal (audio, vision) information in MSA tasks [10, 22, 30, 31, 33, 37, 49] . To explore the dynamic relationship between modalities, HyCon [22] and ConKI [49] conducted intra-modal and inter-modal comparative learning to explore the interaction between modalities, but heavily relied on labeled datasets. Due to the limited availability of MSA sentimentannotated datasets and the increased parameter capacity of fused models, supervised models tend to experience significant overfitting [8, 38] , thus reducing the generalization ability of existing MSA models.Through the above analysis, to improve the quality and performance of fusion representations for MSA, two key issues need to be addressed: 1) How to avoid modal imbalance (mainly nonverbal) due to modal gaps and efficiently construct text-centered non-verbal joint multimodal representations? 2) How to address the problem of overfitting and poor generalization ability in supervised models when using limited labeled datasets?Inspired by knowledge-enhanced pre-training models on text sentiment analysis [32, 34, 54, 55] , we found that a large number of unlabeled opinion videos on the Internet contain valuable sentiment knowledge. This knowledge can guide the fusion of visual, audio, and text modalities to express sentiment patterns or combine sentiment semantics. Learning this knowledge will enhance sentiment representation in MSA tasks. This can enhance further learning on a restricted MSA dataset. To this end, we propose a knowledge-enhanced self-supervised balanced representation (KEBR) method. Sentiment knowledge helps predict the sentiment intensity of masked sentiment words in unlabeled opinion videos using contextual and non-verbal information to learn common sentiment patterns. Specifically, we propose a text-based cross-modal fusion method (TCMF). Different from the previous work, we employ multi-layer cross-modal fusion to inject low-level features of non-verbal modalities into the semantic representation of text, while keeping the fused parameter capacity unchanged. This approach preserves the original affective semantics of the non-verbal information and enhances the text-based multimodal representation. In addition, a multimodal cosine constrained loss (MCC) was designed to mitigate the modal imbalance. Different from previous methods, MCC is capable of optimizing the imbalance among the three modalities. Its design restricts the injected non-verbal information in the fused multimodal representations to mitigate modal bias. Furthermore, MCC is designed as an external constraint with almost no additional training cost and is independent of the model or architecture.As shown in Fig. 1 , first, the most significant sentiment word in the text is masked according to the pre-specified sentiment lexicon. Then, employing the proposed text-based cross-modal fusion method, the injection fusion of non-verbal information and the enhancement of text representation is accomplished. At the same time, the designed multimodal cosine constrained loss is applied to avoid the problem of under-optimization of non-verbal information due to modal gaps in the joint multimodal representation. Finally, the masked word representations are used to predict the joint loss of sentiment intensity and multimodal cosine constraints, embedding word-level sentiment knowledge from the video into pre-trained multimodal representations in a balanced manner.After pre-training KEBR, to evaluate its effectiveness, we finetuned it on two benchmark datasets: MOSI [52] and MOSEI [1] . The experimental results demonstrate that KEBR outperforms the baseline and achieves state-of-the-art performance. The code has been released at https://github.com/*****/KEBR/.The primary contributions of this paper are as follows:• We propose a knowledge-enhanced multimodal self-supervised balanced representation method, which leverages the sentiment knowledge from large-scale unlabeled videos to enhance multimodal sentiment representation learning. • We propose a text-based cross-modal fusion approach that injects low-level features from audio and visual modalities into text to enhance the multimodal information representation of text. This method highlights the dominance of text and the assistance of non-verbal information in modal fusion, which preserves the original affective semantics of the non-verbal information while enhancing the text-based multimodal representation. • We propose a multimodal cosine constrained loss function to optimize the imbalance of unimodal in joint representation, which avoids the problem of certain modalities being neglected in fusion unable to exploit their capabilities. 
related work 2.1 knowledge enhanced pre-training language models: 
 In NLP, Transformer-based pre-trained language models have been widely studied and applied to extract contextual semantic features.Considerable attention has been focused on pre-training models on large-scale unlabeled datasets [9, 18, 46] to capture linguistic information, followed by fine-tuning the model for specific downstream tasks.It is effective to introduce domain-specific knowledge into the process of pre-training language models [53] . The domain-specific knowledge can be the common knowledge of entity type and relationship classification tasks [17, 27, 53] , legal knowledge of extracting legal elements [56] , sentiment knowledge of sentiment analysis [15, 34, 47] , and biomedical knowledge for health question-andanswer and medical reasoning [13] . Several studies [17, 27, 53, 54] integrate sentiment knowledge, encompassing sentiment words, word polarity, and aspect-emotion pairs, into text representations to improve specific sentiment analysis tasks. Knowledge-enhanced pre-trained language models have made significant progress in text sentiment analysis. However, few studies have utilized the abundant sentiment knowledge presented in unlabeled videos to improve sentiment representation learning for MSA tasks.MSA collects and processes data from multiple sources of audio, visual, and text information to comprehensively understand human sentiment [11] . Early MSA studies integrate multimodal representations obtained from different feature extraction networks [28] . The Tensor Fusion Network (TFN) [50] learns the intra-modaldynamics through modal embedding sub-networks. Low-rank multimodal fusion (LMF) [19] reduces the computational cost of TFN by utilizing a low-rank tensor. The multimodal transformer (MulT) [35] applies cross-modal attention to transform one modality to another. Different from MulT, our KEBR model injects low-level features of non-verbal information into the text feature representation through multimodal interaction. It uses the enhanced text representation as the joint representation for MSA tasks.Given the remarkable success of pre-trained language models in NLP, MISA [12] learns invariant and specific representations of each modality. The self-supervised multitask multimodal Sentiment Analysis Network (Self-MM) [48] introduced a self-supervised label generation module for acquiring additional unimodal labels. MAG-BERT [29] introduced a multimodal adaptation gate that enables BERT to accept representations of non-verbal modalities. MMIM [11] maximizes mutual information in the multimodal fusion pipeline to preserve the fusion of task-relevant information among modalities. To explore the dynamic relationship between modalities, HyCon [22] performs comparative learning intra-modal and inter-modal to explore cross-modal interactions. Furthermore, ConKI uses the dataset from one MSA task as domain-specific knowledge trained on a different dataset, leading to the model's performance becoming overly reliant on the labeled dataset. CENet [37] utilizes visual and audio sentiment information to enrich the text representation in the pre-trained language model, while still relying on the labeled data.Different from existing MSA models, KEBR not only uses audio and vision of MSA task labeled data to enhance the corresponding text representation but also uses sentiment knowledge from large-scale unlabeled videos to promote multimodal sentiment representation learning, which facilitates further training on limited MSA datasets.By integrating various sensory inputs, multimodal methods contribute to enhancing overall task comprehension and performance [2] . However, in practice, it has been found that even if a multimodal model outperforms its unimodal counterpart, one modality will dominate the optimization process because the unimodal converges at different speeds. Thus, causing the other modalities to be neglected and unable to play their ability [38] . To address this issue, some studies have proposed methods such as gradient mixing [38] , dynamic gradient modulation [26] , and an information-sharing multimodal fusion strategy [24] . MMCosine [43] proposes a multimodal cosine loss, which performs a modal L2 normalization of the features and weights. Inspired by this, we propose a multimodal cosine constrained loss function (MCC), which extends from audiovisual bimodality to address imbalance optimization among text, audio, and visual modalities compared to previous imbalance mitigation methods. Our method is self-supervised and independent of the model architecture. 
introduction methodology: 
 As shown in Fig. 2 , the framework of KEBR consists of four main components: sentiment knowledge-guided masking and prediction, text-based cross-modal fusion (TCMF), multimodal cosine constraint loss function (MCC), and sentiment intensity prediction.The task of MSA is to calculate the sentiment information in videos based on multimodal signals (𝑀 = {𝑡𝑒𝑥𝑡 (𝑡), 𝑎𝑢𝑑𝑖𝑜 (𝑎), 𝑣𝑖𝑠𝑢𝑎𝑙 (𝑣)}). The feature sequences of these signals can be denoted as 𝑋 𝑡 ∈ R 𝑙 𝑡 ×𝑑 𝑡 , 𝑋 𝑎 ∈ R 𝑙 𝑎 ×𝑑 𝑎 and 𝑋 𝑣 ∈ R 𝑙 𝑣 ×𝑑 𝑣 . 𝑙 𝑀 ∈ {𝑡,𝑎,𝑣 } denotes the sequence length of each modality, and 𝑑 𝑀 denotes the dimension of KEBR captures common sentiment knowledge in unlabeled videos by predicting the sentiment value of masked sentiment words, which is helpful for further learning with the limited labeled data of MSA. Specifically, given an opinion video without sentiment labels, Automatic Speech Recognition (ASR) is used to obtain the transcribed text of the video. Subsequently, the words with the highest sentiment intensity in the transcribed text are identified to be masked 𝑦 ∈ {𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒, 𝑛𝑒𝑢𝑡𝑟𝑎𝑙, 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒} based on a predetermined sentiment lexicon [14] . At the same time, the sentiment score of the word and its position in the text are recorded. The masked text be represented as follows.EQUATIONwhere 𝑤 𝑚𝑎𝑠𝑘 denotes the masked word. For text sentences that do not contain sentiment words, a word in the sentence is randomly masked, and its sentiment score is set to 𝑦 𝑚𝑎𝑠𝑘 = 0.0 .Masked text 𝑇 ′ with accompanying non-verbal information is used as input for KEBR pre-training. The sentiment score serves as the label to predict the sentiment value of the masked words. This prediction is assisted by text-based cross-modal fusion and a multimodal cosine constraint loss function. Thereby, word-level sentiment information is embedded into the pre-trained multimodal representation to enhance the model's performance on MSA-specific tasks.In MSA, different non-verbal information may convey different sentiments for the same word. Therefore, the precise sentiment semantics should be determined by the word itself and the accompanying non-verbal behavior [39] . However, many studies have shown that text plays a more central role in MSA than non-verbal information [10, 12, 30, 31, 33, 37, 49] . Inspired by MulT [35] , which focuses on low-level features in other modalities to fuse multimodal information, a text-based cross-modal fusion method (TCMF) is designed. In this method, low-level features from audio and visual modalities are repeatedly injected into the text feature space (K times) to enhance the multimodal representation of the text. The capacity of the fused feature parameters remains unchanged, avoiding overfitting.In 2, the feature sequence 𝑋 𝑀 0 (𝑀 ∈ {𝑡, 𝑎, 𝑣 }) is used as the input to TCMF. The TCMF module consists of a total of K-th layers. The calculation for the TCMF at layer can be expressed as:EQUATIONWhere 0≤ 𝑘 ≤ 𝐾 , if 𝑘 = 1 , 𝑋 𝑡 (𝑎𝑣) 1 = 𝑇𝐶𝑀𝐹 1 (𝑋 𝑎 0 , 𝑋 𝑡 0 , 𝑋 𝑣 0 ).The main difference is that the audio and vision inputs of different layers of TCMF 𝑋 𝑎 0 are and 𝑋 𝑣 0 , while the text is input as 𝑋 𝑡 0 (𝑘 = 1) or 𝑋𝑘 -1 (1 < 𝑘 ≤ 𝐾) . Non-verbal information and text within the TCMF module interact through cross-modal attention (CMA). Taking text and audio as examples, the Q (query), K (key), and V (value) of CMA are represented as follows:EQUATION)EQUATION)EQUATION)Where𝑊 𝑡 Q ∈ R 𝑑 𝑡 ×𝑑 𝑡 , 𝑊 𝑎 𝐾 ∈ R 𝑑 𝑎 ×𝑑 𝑡 and 𝑊 𝑎 𝑉 ∈ R 𝑑 𝑎 ×𝑑 𝑡 are weighs. 𝑋 𝑡 𝑘 -1 = 𝑋 𝑡 0 (𝑘 = 1) , 𝑋 𝑡 𝑘 -1 = 𝑋 𝑡 (𝑎𝑣) 𝑘 -1 (1 < 𝑘 ≤ 𝐾).Further, the CMA for audio and text can be expressed as follows.EQUATIONWhere 𝑀 )) + 𝑀 𝑡 (𝑎𝑣) 𝑘 (8) Where 𝑋 𝑡 (𝑎𝑣) 𝑘 ∈ R 𝑁 ×𝑑 𝑡 denotes the output of the TCMF module at layer k.In Section 3.3, we employ the low-level features of both audio and visual modalities to enhance the multimodal representation of the text via TCMF. However, in the process of information fusion, we find that when a non-verbal modality gains a relative advantage in the fusion, it tends to quickly strengthen its advantages, thus precluding the fusion of other non-verbal information within the main modality. To optimize the imbalance of modality in the multimodal joint representation, we design a multimodal cosine constraint loss function (MCC), which constrains the fusion of non-verbal information by utilizing the cosine constraint. This approach facilitates the synchronous convergence of different non-verbal information and exploits the capabilities of different modalities.As shown in Fig. 3 , the sample feature sequences are firstly normalized by L2. Subsequently, the cosines of the fused multimodal features 𝑋 EQUATIONWhere Δ𝜃 𝑖 = arccos 𝜃 When 𝑓 𝑖 𝑚𝑐𝑐 converges, the value of the cosine function 𝑦 (𝑥 𝑖 ) value is assumed to be ŷ (𝑥 𝑖 ) . To induce 𝑦 (𝑥 𝑖 ) convergence ŷ (𝑥 𝑖 ) , both 𝜃 3 . Simultaneously, an angle penalty g is introduced to regulate the convergence value of the cosine function 𝑦 (𝑥 𝑖 ) . Through this joint regulation, the convergence of the multimodal cosine constrained loss 𝑓 𝑖 𝑚𝑐𝑐 for the sample 𝑖 can be expressed as follows.𝑓 𝑖 𝑚𝑐𝑐 = lim (𝑦 (𝑥 𝑖 )+𝑔)→ ŷ (𝑥 𝑖 ) 𝑠 × (| cos(𝜋 (𝑦 (𝑥 𝑖 ) + 𝑔))|) = 0 (13)In the pre-training phase of KEBR, the main task is to predict the intensity of sentiment words that have been masked. This is to use the sentiment knowledge from large, unlabeled videos to improve multimodal sentiment representation learning. Masked text encoded by BERT [9] and enhanced with low-level features from audio and visual modalities by TCMF, can be represented as follows: EQUATIONWhere 𝑥𝑀𝐴𝑆𝐾 denotes the masked word that has been reinforced by non-verbal information. To predict the sentiment value of the masked word, we employ a multilayer perceptron (MLP) with a ReLU activation function acting as a classifier.EQUATIONWhere 𝜃 𝐹𝐶 denotes the parameters of the fully connected network, and 𝑦 𝑝𝑟𝑒𝑑 is the predicted sentiment value.EQUATION)EQUATION)where n is the batch size. L is the total loss of the model pre-training. 𝐿 𝑡𝑎𝑠𝑘 is the masked sentiment word sentiment value prediction loss. Our task in the testing phase is also affective computing. Therefore, in the testing phase, we only added an output layer to the pre-trained language model as well as the multimodal fusion module to generate task-specific predictions. Then, we fine-tuned the labeled multimodal dataset to validate the performance of KEBR in the MSA task.Table 1 : Results on MOSI and MOSEI. In Acc-2 and F1, the left of the "/" is "negative/nonnegative" and the right is "negative/positive". "*" denotes the result is from ConKI [49] . "Δ" denotes the reimplementation of the non-verbal feature mentioned in 4.2 .Mark the best results in bold.  
experiments 4.1 datasets and evaluation metrics: 
 Pre-training datasets. KEBR was pre-trained on the VoxCeleb 1 dataset, which is a sizable dataset of English speaker recognition with rich sentiment. The dataset has two parts: VoxCeleb1 [23] and VoxCeleb2 [7] . VoxCeleb1 has over 100,000 discourses from 1251 celebrities from YouTube videos. VoxCeleb2 has more than one million discourses from 6112 speakers. According to previous research [41] , video clips without English were excluded.Fine-tuning datasets. We fine-tuned the pre-trained KEBR on two benchmark datasets in the MSA field: MOSI [52] and MOSEI [1] . The details of the datasets are shown in Appendix A.Following previous works [7, 12, 22, 33, 49] , we present our experimental results in both regression and classification. For regression, we present the mean absolute error (MAE) and Pearson correlation (Corr). For classification, we calculate the Acc-2 and F1 scores for both the negative positive (zero excluded) and nonnegative positive (zero included) settings. Additionally, In addition, we calculate Acc-7, which shows the percentage of predictions correctly classified into seven intervals between -3 and +3. Except for MAE, higher values indicate better performance for all metrics.For the text, audio, and visual information in the dataset, we use BERT [9] , librosa [4] , and OpenFace [3] to perform the feature extraction of the corresponding information. To be consistent with the sentiment values of the test dataset, we linearly scale the sentiment scores shown in the sentiment lexicon [14] from [-4, +4] to [-3, +3].Training Details: All models are built on the Pytorch [25] toolbox with the NVIDIA RTX A100 GPU. The batch size is set to 32 and the epoch is set to 200. The initial learning rate is set to 5e-6 for BERT and 1e-4 for other parameters. Adam [16] is used as the 1 https://mm.kaist.ac.kr/datasets/voxceleb optimizer. The angle penalty g is 0.2. Appendix B describes the details of video feature extraction and hyper-parameter setting.Given that the pre-training dataset lacks explicit word timestamps, our model does not need to align text words with vision and audio. We performed a comprehensive comparative analysis of KEBR with baselines and state-of-the-art models on an unaligned setup, with the comparative model as follows: TFN [50] , LMF [19] , MISA [12] , MAG-BERT [29] , HyCon [22] , MMIM [11] , ConKI [49] , MulT [35] , CENet [37] , Self-MM [48] . Please refer to Appendix C for more details on these baselines.Following previous works [8, 12, 22, 49] , we run our model five times with the same hyper-parameter settings and report the average performance of all the metrics in Table 1 . It is evident that KEBR produces better or more competitive results in MOSI and MOSEI.In the experiments based on the re-implementation of non-verbal feature extraction in Section 4.2 (Marked with "Δ" in Table 1 ), our KEBR significantly outperforms the other models on all experimental metrics for two datasets. The baseline code is from publicly available repositories 2 .Compared with the experimental metrics in the latest research papers, KEBR achieved better or competitive results, except for Acc-7 and MAE on MOSI are slightly lower than ConKI. This may be because, ConKI utilizes adapters to inject a larger amount of data from MOSEI as MOSI-specific domain knowledge into the learning process. However, KEBR was pre-trained on the unlabeled dataset VoxCeleb to learn more general sentiment patterns, so the metrics of the more refined multi-classification tasks were slightly lower than ConKI. However, on the MOSEI dataset, KEBR outperforms ConKI in all metrics, with an average performance improvement of 1.72%. Considering that the size of MOSEI is much larger than To study the impact of learning from external data on the model, we pre-trained KEBR using the VoxCeleb1 (132K) and VoxCeleb2 (947K) datasets.The results are presented in Table 2 . A substantial amount of pre-training data results in a more significant performance improvement. Compared with VoxCeleb1, the average performance of MOSI and MOSEI pre-trained on VoxCeleb2 is improved by 0.99% and 1.37%, respectively. The performance improvement of MOSEI exceeds that of MOSI, possibly due to the relatively small size of the MOSI dataset, which might have been more effectively enhanced using the VoxCeleb1 pre-trained model. It may also contain labels with noise, which has been found in some previous studies [37] . In Section 4.6, when the effects of different g on performance are analyzed on two data sets, it is further demonstrated that the model trained on MOSI has greater instability than MOSEI.To investigate the impact of backbone language models with different parameters on KEBR, we conducted comparative analyses on VoxCeleb1 using Bert-base and Bert-large models with 110M and 340M parameters in MOSI and MOSEI.As shown in Table 3 , Bert-large enhances the average performance of MOSI and MOSEI by 1.88% and 1.79%, respectively, compared to Bert-base. By comparing Tables 2 and 3 , we observe that larger language models may outperform models trained on larger pre-training datasets. Bert-large has stronger language comprehension and representation ability. This is consistent with the earlier conclusion [5] that scaling up can significantly enhance performance. Moreover, the text encoded by a larger pre-trained language model can acquire a broader representation space for non-verbal information under the same cosine constraint, while retaining the text features, thereby facilitating further improvement in performance.We conduct an associated ablation study in the MOSI dataset. Pretraining in VoxCeleb2 is costly, so we conduct an ablation analysis in the VoxCeleb1 dataset. Bert-large serves as the backbone model.In this paper, the performance of the KEBR is primarily influenced by a sentiment knowledge-enhanced pre-training task (Pre-t) and two model components: text-based cross-modal fusion (TCMF) We evaluated the impact of not including the KEBR components on performance using the average degradation rate of all the metrics. As shown in Table 4 , it can be observed that excluding different parts of the KEBR will lead to performance degradation. Further analysis shows that without TCMF(C2) KEBR performance decreases by 8.3%. This decrease is significantly larger compared to the exclusion of Pre-t(C2) and MCC(C4). This suggests that a textbased approach in the MAS task can significantly improve model performance. The average performance of KERB without Pre-t(C2) only decreases by 0.51%, which may be attributed to conducting our ablation experiments on the smaller VoxCeleb1 dataset (132 K). Comparing the results in Table 2 to VoxCeleb2 (947 K), the overall performance without Pre-t(C2) drops to 1.49%. This suggests that the impact of pre-training on model performance is limited by the amount of pre-training data. Meanwhile, we observed that the average performance without MCC(C4) decreased by 1.55%, even surpassing the 1.49% decline observed with the larger pre-training dataset VoxCeleb2. This suggests the presence of a potential modal imbalance issue in MSA, and incorporating additional constraints of MCC to balance the modal representation can significantly enhance the model's performance.We analyze the effects of different g on the model and the imbalance phenomena in the MSA task (provided by Appendix D.1), and further, visualize the reasons for the imbalance phenomena in multimodal fusion(provided by Appendix D.2).Analyze the angle penalty g. In Section 4.5, experiments demonstrate that MCC is effective and necessary in KEBR. Next, we analyze the effect of different values of the angle penalty g in MCC on the MOSI and MOSEI datasets. The pre-training dataset used is VoxCeleb1.It is observed in Fig. 4 that the variation trend of F1 on the two datasets with g is similar. However, MOSI exhibits relatively higher instability, likely due to its smaller dataset size and the potential presence of noisy labels. For 𝑔 ∈ [-0.5, -0.05] , F1 is low and the performance is very unstable. For 𝑔 ∈ [-0.05, 0.25] , F1 is relatively high and stable. For 𝑔 ∈ [0.25, 0.5] , F1 is relatively low and unstable. Further analysis with Eq.11, d considering the periodic nature of the cosine function, suggests that the variation of F1 with g should also display a weak periodicity with a period of 1. Due to the nonlinearity of the cosine function, g varies within the same step interval, resulting in differing effects on the cosine values. For 𝑔 ∈ [-0.5, 0] , the same change has a greater impact on F1 than that 𝑔 ∈ [0, 0.5] , so there is a tendency for F1 to be destabilized. Furthermore, when 𝑔 ∈ [-0.5, 0] , the cosine function needs to converge to a larger cosine value. This will make the proportion of non-verbal info in the fused features larger, which destroys the dominance of the text modality and causes performance degradation and instability. On the contrary, when 𝑔 ∈ [0, 0.5], the value that 𝑦 (𝑥 𝑖 ) needs to converge decreases as g increases. The fused feature has less nonverbal information and cannot utilize the ability of all modalities, so the performance will be decreased. However, in this scenario, text still dominates, resulting in a slight decline in performance. Considering different non-verbal information may convey different sentiments for the same word, non-verbal dominant modal fusion tends to be more unstable than text. Thus, we can guess boldly that for different datasets, when 𝑔 ∈ [0, 0.25], it will bring a large performance improvement while maintaining a robust multimodal joint representation.To better comprehend the modal imbalance issue in MSA, a comparative experiment is conducted in Appendix D.1 to observe the convergence of various modalities at different learning rates. In Appendix D.2, modal feature distribution is visualized and analyzed with and without MCC to elucidate the underlying cause of the modal imbalance problem. 
conclusion: 
 In this paper, we propose KEBR, which leverages the sentiment knowledge from large-scale unlabeled videos to enhance multimodal sentiment representation learning. This approach facilitates learning on limited MSA datasets. KEBR highlights the dominance of text modality in MSA tasks by employing multi-layer cross-modal cross-fusion to inject non-verbal information into text representations to enhance multimodal representation in text. Furthermore, we design the multimodal cosine constrained loss to optimize the imbalance of unimodal in joint representation and utilize the capability of different modalities. We validate the effectiveness of the KEBR model and the importance of each module through comprehensive experiments and ablation studies conducted on two benchmark datasets. The experiments show that the larger the amount of unlabeled video data and the stronger the language modeling ability, the better the performance. We believe this work can inspire further exploration of multimodal representations for various scenarios and tasks, as well as investigations into optimizing modal imbalance in joint learning.  
 
 Appendix: https://github.com/thuiar/MMSA 


Decision: This paper studies multimodal sentiment analysis and proposes a self-supervised learning method to address the imbalance in unimodal performances by two modules, including text-based cross-modal fusion and multimodal cosine-constrained loss. This paper received four reviews with initial ratings of one borderline reject and three borderline accept. The authors provided rebuttal to address the reviewers' concerns. After discussion, the final ratings became four borderline accept. The reviewers are satisfied with the rebuttal and agree most of their concerns are addressed in the rebuttal. I recommend acceptance of this paper and request the authors to take the reviews and their rebuttal into consideration when preparing the camera-ready version.

Reviews: review: paper_summary: paper_summary: This paper introduces a self-supervised learning method for multimodal sentiment analysis. The proposed method addresses the imbalance in unimodal performances by two modules, including text-based cross-modal fusion and multimodal cosine constrained loss. Experiments on MOSI and MOSEI are conducted.
strength_weakness: strength_weakness: + The idea of building masked affective modeling is interesting.
+ Experimental results demonstrate the effectiveness of the proposed method.
+ Ablation studies verify that each component is an optimal solution.
limitations: limitations: -	While the experimental results on MOSI and MOSEI are promising, these datasets are widely studied and may not fully capture the complexity and diversity of real-world multimodal sentiment analysis tasks. To strengthen the validity and generalizability of the proposed KEBR method, I recommend evaluating its performance on a broader range of datasets, such as SIMS, SEWA, Recola, Affwild, EmotiW, IEMOCAP, and LIRIS-ACCEDE.
-	Extra pretraining data. The authors use quite large-scale datasets as extra data for pretraining, which leads to unfair comparison. Models pre-trained on extensive datasets are naturally expected to perform better, which could overshadow the true effectiveness of the proposed method. Therefore, it should verify two points for clarity: one is for the effectiveness of the proposed method in the current small-sized datasets (i.e., MOSI and MOSEI).
-	The results on dimensional emotion recognition are omitted. There are various datasets for VA/VAD estimation such as AFEW.
-	Missing SOTA comparison. There are some self-supervised pretraining methods [A,B,C,D], which align with the proposed method. However, the authors provide no discussion or comparison.
-	Some Typos. 
a.	The title of sec 3 should be more rigid.
b.	The decimal places should be uniform in Tab. 2, 3, and 4.

[A] Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition, Information Fusion, 2024.

[B] Masked Affective Representation Learning via Masked Temporal Distribution Distillation, CVPR, 2024.

[C] Contrastive Audio-Visual Masked Autoencoder, ICLR, 2023.

[D] Audiovisual Masked Autoencoders, ICCV, 2023.
paper_summary: paper_summary: This paper proposes a knowledge-enhanced multi-modal self-supervised balanced representation learning method KEBR. It utilizes large-scale unlabeled video data's emotional knowledge to enhance multi-modal emotional representation learning. The paper also proposes a text-based cross-modal fusion method and a multi-modal cosine constraint loss function to address the imbalance problem among different modalities during the joint optimization process. Experimental results show that KEBR achieves state-of-the-art performance on benchmark datasets.
strength_weakness: strength_weakness: 1. The article is well-structured with a clear line of reasoning.
2. The authors use clear language and provide sufficient background information for readers who may not be familiar with MSA or imbalanced optimization. 
3. The experiments are thoroughly and comprehensively conducted.
limitations: limitations: 1. The manuscript contains several grammatical and spelling errors. The authors are requested to thoroughly revise the grammar and spelling throughout the document. Specific issues include, but are not limited to: the text in line 438, an extraneous symbol in line 469, and the incorrect use of 'Ave' instead of 'Avg' in Table 2.
2. The schematic diagram of the overall framework in the article is not clearly drawn. It is indicated in the text that the framework consists of four parts; the authors should clearly label the name of each part in the diagram.
3. In the section on text-based cross-modal fusion, the authors frequently mention the use of low-level audio and visual modalities, which are injected into the text features. Could the authors clarify the rationale behind using low-level features? Additionally, how might the use of high-level features impact subsequent operations of the model?
paper_summary: paper_summary: This paper proposes a knowledge-enhanced self-supervised balanced representation approach (KEBR). KEBR uses a text-based cross-modal fusion method to enhance text representations with non-verbal information and a multimodal cosine constrained loss to balance multimodal information. By incorporating sentiment knowledge and non-verbal cues, KEBR achieves state-of-the-art results on the MOSI and MOSEI datasets.
strength_weakness: strength_weakness: 1. This paper is well-structured, presenting a clear and logical flow of ideas. The organization of content makes it easy to read and understand, facilitating the reader's comprehension of the complex concepts and methodologies discussed.

2. The design of the Multimodal Cosine Constraint (MCC) loss is both innovative and mathematically interpretable. This approach effectively alleviates the imbalance in multimodal representation, providing a robust framework for enhancing the integration of diverse information modalities.

3. The experiments conducted are relatively comprehensive, covering a wide range of scenarios and validating the proposed approach thoroughly. 

4. This method achieves state-of-the-art (SOTA) performance on several datasets, showcasing its superior effectiveness and robustness compared to existing approaches.
limitations: limitations: 1. The prediction of opinion words heavily relies on the sentence text. In multimodal scenarios, various cases such as sarcasm and implicit sentiment clues need to be considered from non-verbal information. Although the language modality plays a crucial role in this task, the pre-training task seems to overly emphasize the textual modality, addressing the problem in a "hard" manner. This approach neglects the potential implicit sentiment clues provided by other modalities, which could lead to a less comprehensive analysis. A more balanced consideration of all modalities would enhance the model's effectiveness.


2. The sentiment knowledge utilized in this study is primarily based on a sentiment lexicon, which neglects other rich sentiment expressions conveyed by the semantic content of the sentence. This reliance on lexicon-based knowledge may overlook the nuanced and context-dependent nature of sentiment, potentially limiting the model's ability to fully capture and interpret the diverse ways sentiment can be expressed. Integrating a more context-aware approach could enhance the model's comprehensiveness and accuracy.

3. There are instances where opinion words may not appear in the text, necessitating sentiment polarity prediction based solely on acoustic and visual content. This method struggles to handle such cases effectively, as it primarily relies on textual information. Addressing this limitation by enhancing the model's ability to interpret non-verbal cues would improve its robustness and accuracy in predicting sentiment in multimodal contexts.

4. In line 458, "Likewise, the text sequence enhanced by audio modal injection can be denoted as," the term "audio" should be "visual." Manuscripts submitted to top-tier conferences should avoid such errors, as they undermine the professionalism and credibility of the research.

5. The Text-based Cross-Modal Fusion method is deemed too simplistic and lacks novelty. The fusion process fails to address the semantic gap that exists between different modalities, which is a critical aspect of effective multimodal fusion.

6.Including a case study to illustrate the efficacy of sentiment knowledge pre-training would enhance the paper's academic value. A detailed case study could provide concrete examples and insights into how the sentiment pre-training approach improves sentiment analysis performance, making the research more practical and applicable to real-world scenarios.
paper_summary: paper_summary: This study delves into the Multimodal Sentiment Analysis task, presenting a knowledge-enhanced self-supervised balanced representation approach (KEBR) designed to capture shared sentiment-related knowledge in unlabeled videos. It addresses the optimization challenge stemming from information imbalances between modalities. Additionally, the paper introduces a text-based cross-modal fusion method (TCMF) and a multimodal cosine-constrained loss function to more effectively represent multimodal information in text and counteract the imbalance of unimodal representations in joint representation.
strength_weakness: strength_weakness: 1. This paper is well-written, easy to read and detailed. The figures are well-designed.

2. The experimental results are clearly demonstrated.
limitations: limitations: 1. The proposed pretraining stage is not strong. It requires larger datasets and more computational resources, but the improvements are not too significant. Furthermore, the evaluation of only 2 datasets (MOSI and MOSEI) could not show a clear contribution of this approach.

2. Figure 3 is quite confusing and has many details, leading to overlapping text and changes in resolution and scale.

3. This work focuses on representation learning, especially multimodal text representation. Thus, the paper should include the visualization of the representation vector in feature space compared with other baselines (or without the proposed method).