## 1 Introduction  
With the recent surge in social media platforms, there has been an increasing focus on multimodal learning [1, 2]. Multimodal data, consisting of Acoustic (Ac), Visual (VI), and Language (La) modalities, have become prevalent in various applications. Multimodal sentiment analysis (MSA) [3, 4, 5, 6] is an approach that leverages information from multiple modalities to enhance the prediction accuracy of emotional states. Researchers in MSA have primarily concentrated on studying intricate fusion models [7, 8, 9], such as tensor-based fusion [10] and attention-based fusion. However, despite the advancements made in previous studies using benchmark datasets, the fusion mechanisms employed in MSA still encounter difficulties in effectively addressing the existing modality gaps between diverse modalities. These modality gaps refer to the variances that exist among the different modalities, posing a challenge in integrating their information seamlessly for accurate sentiment prediction. In the field of multimodal sentiment analysis, researchers have proposed various approaches to address the challenges associated with heterogeneous modalities. One such approach is the Multimodal Transformer (MuIT) provided by [PERSON] et al. [11]. The model leverages a crossmodal attention mechanism to adapt information between different modalities. However, MuIT ignores the interaction between modalities, and MuIT can not obtain comprehensive joint representations. Another important aspect emphasized by [PERSON] et al. [12] is the consideration of information exchange among all modalities for effective multimodal fusion. According to their argument, the capture of joint representations is a difficult task because of inherent differences in heterogeneity across modalities. Simply concatenatingmodality features to create a common representation does not fully explore the interrelationships between different modalities. On the other hand, projecting different modalities into the same representation space can achieve consistency but overlooks complementarity. In conclusion, the main challenges for multimodal sentiment analysis are capturing the joint representation and exploiting effective interactions across modalities. This involves the integration of consistency and complementarity to achieve enhanced accuracy in sentiment analysis. To address the aforementioned challenges, we propose a novel framework for multimodal sentiment analysis called Consistency and Difference using a Multitask Learning network (CDML). CDML can integrate unimodal diversity and bimodal consistency in a unified framework. For bimodality, we put forward a Joint Representation Translator (JRTran) to integrate joint representations of Vi-La, Ac-La, and Vi-Ac. The JRTran employs cycle translation. Generators and discriminators are designed in the cycle translation for adversarial training, which enables the translated features to better approximate the source features and further improve the performance of sentiment analysis. The process of translating between a source modality and a target modality yields an intermediate representation that acquires consistent information. Additionally, the attention mechanism is introduced to effectively assign the attention weights of each modality. Based on attention weights, the features of each modality can be efficiently fused. In the meantime, the generation of redundant information and interference of nonimportant modalities can be effectively reduced. At the end of CDML, a multitask learning framework is introduced to enhance the classification performance of the model. The model can pay more attention to the differentiated representations of each unimodal itself by jointly training multimodal tasks and unimodal subtasks. Multitask learning acts as a model regularizer with the help of unimodal auxiliary tasks. To sum up, the main novelties of this paper can be characterized as follows:  
1. A multimodal sentiment analysis model that operates on joint representations is introduced.