# Cmdar: Cross-Modality Dual-Attention Fusion Network for Multimodal Sentiment Analysis
## 1 Introduction  
Multimodal fusion learning aims to create fine-grained fusion representations from different modalities, which in turn enhances the overall performance of the model. In recent years, online social media platforms like Facebook, YouTube, TikTok, and Twitter have experienced rapid growth. This growth has led to an explosion of multimodal data (e.g., textual, acoustic, and visual) that captures users' emotional expressions. This influx of data not only motivates advancements in multimodal fusion learning but also presents significant challenges. Currently, multimodal fusion learning has garnered extensive research interest across various fields, including multimodal sentiment analysis [1, 2], cross-modal retrieval [3, 4], visual questioning [5], and emotion recognition in conversation [6, 7]. Multimodal sentiment analysis (MSA) aims to analyze information in multiple modalities, including textual elements, acoustic attributes, and visual indicators. This approach enhances the prediction of both the polarity and intensity of human sentiments. Unlike traditional unimodal sentiment analysis, MSA effectively alleviates semantic and sentimental ambiguities by integrating and leveraging the rich, complementary information from multiple sources. Recently, MSA has been increasingly applied to various downstream fields, especially in areas such as healthcare [8], e-business [9], and recommended systems [10]. Multimodal sentiment analysis techniques rely on the comprehensive extraction of features from different data modalities. Much prior research has employed custom algorithms to extract fixed features, including acoustic attributes (such as pitch, bandwidth, and speech rate) and visual elements (like facial structure, head angle, and pose). Although these fixed features incorporate prior knowledge, they often lack sensitivity across different datasets. Recent studies, including MMML and UniMSE, highlight the significant potentialof pre-trained models during the feature extraction phase, providing valuable insights for our work. Multimodal sentiment analysis must account for the inherent heterogeneity in multimodal data, necessitating the accurate capture of inter-modality correlations. These correlations can be categorized into modality-consistent and modality-inconsistent information. Many current studies tend to prioritize modality-consistent information, which enhances the model's recognition capabilities but risks fostering an over-reliance on modal details that contribute significantly. Previous research [11] indicated that textual modality is generally more informative than acoustic and visual modalities. Consequently, when discrepancies arise in affective polarity between textual and other modalities, the model's performance can be substantially impacted. For instance, if a girl states, \"I really like the sun\", models will likely interpret her mood as positive. However, if her tone conveys sadness or she is depicted getting wet in the rain, the models might erroneously classify her sentiment as negative. Thus, leveraging the variability among modalities is crucial for improving model performance. Inspired by the observations above, we propose a novel Cross-Modality Dual-Attention Fusion Network (CMDAF). It conducts multi-level interactions to learn and understand inter-modality correlation, which improves the performance of MSA. In summary, our contribution can be summarised as follows:  
* We propose the Cross-Modality Dual-Attention Fusion Network. The network uses pre-trained models to extract unimodal raw data from the three modalities and convert them into a unified dimensional embedding representation. It facilitates inter-modal information interaction. * We achieve multi-level inter-modal information interaction by feeding the representations from three modalities into the cross-modality dual-attention fusion module. * We propose the Transformer-based Features Projector. It projects unimodal features into modality-consistent and modality-inconsistent representations.