## 1 Introduction  
Emotion classification is a crucial subtask that has been extensively studied in domains such as human-computer interaction, healthcare, and medicine. Early research on emotion classification primarily utilized unimodal data, such as text [1, 2], audio [3], or visual cues [4], and achieved notable success. Recently, researchers have started incorporating two or more modalities to achieve multimodal emotion classification (MEC) [5, 6, 7]. MEC (explanations for all abbreviations in Appendix A Table A1) can furnish salient clues to more accurately discern the genuine emotional states of the opinion holder and enhance the precision of outcomes [8]. With the burgeoning of short video applications, MEC has become an active area of research [9]. The two most salient components in MEC tasks are unimodal emotion representation learning and multimodal fusion. In many scenarios, unimodal emotion expression differs from multimodal emotion perception, and multimodal annotation stems from the interaction between each modality. Taking a video clip from the CH-SIMS dataset as an exemplar, as illustrated in Figure 1, the text \"Isn\"t it about to go bankrupt\", with an emotional annotation of \(-0.8\), indicates a negative emotion, and the audio modality annotation is also \(-0.8\). The visual modality expression includes a smile and corresponds to a positive emotional annotation of \(1.0\). The combination of these three modalities results in a multimodal emotional annotation of \(0.6\), indicating a positive emotion for the video clip. The majority of existing research employs identical multimodal annotations (IMAs) for unimodal model pretraining and multimodal fusion [10; 11; 12]. The unimodal models trained using IMAs cannot learn satisfactory representations but rather those of forced alignment. The representation of forced alignment will contain certain discrepancies in the distribution of each modality and form an irregular semantic space, as depicted in Figure 2a. Multimodal fusion grounded on this will generate distorted and convoluted classification boundaries (see Figure 2c). If the unimodal models are trained using independent unimodal annotations (IUAs), the representations for each modality are roughly within a unified region and harbor certain differences (see Figure 2b). Leveraging unimodal representations with certain differences such as this for multimodal fusion will contribute to superior fusion outcomes [13]. Meanwhile, the multimodal fusion strategy is also imperative to achieve a comprehensive MEC model. A satisfactory fusion strategy can fully harness the information from each modality and can capture the differences and interactions between modalities. Building upon this, the distributions of each emotion category are uniform, and the classification boundaries are also relatively systematic (see Figure 2d). Most current studies on fusion pay less attention to the unimodal-multimodal emotions' inconsistency [14; 15; 16; 17] and struggle to learn the differences between different modalities [13]. Figure 1: An example of inconsistency between unimodal emotion expression and multimodal emotion perception. Stage One: unimodal emotion recognition, Stage Two: multimodal fusion for emotion recognition. _M_: Multimodal, _V_: Vision, _A_: Audio, _T_: Text, \(\oplus\): multimodal fusion. Figure 2: Hypothesized unimodal and multimodal emotion representation. In (**a**,**b**), the markers of different colors represent different modalities, and shapes with dotted curves of different colors represent semantic spaces of respective modalities. The black solid curves indicate the envelope of these colored lines. In (**c**,**d**), the markers of different colors represent different emotion categories, and the black solid curves represent classification boundaries. _V_: Vision, _A_: Audio, _T_: Text.