## 3 Methodology
### Model Structure
#### 3.2.2 Dual-Level Attention  
Attention mechanisms help the model better understand multimodal sentiment data and perceive emotional information. They enable the model to capture dynamics within a single modality or between different modalities during the multimodal sentiment processing. The Transformer [38] is a language model in the field of natural language processing; it is based on dot-product self-attention mechanisms. It employs self-attention to infuse global semantic information and consider long-range dependencies for every word in the sequence. Furthermore, the multihead mechanism allows the model to learn different subspaces of semantics. In simple terms, the Transformer processes the input sequence \(H\in\mathbb{R}^{T\times d}\) with positional encoding; it defines Query as \(Q=HW_{Q}\), Key as \(K=HW_{K}\), and Value as \(V=HW_{V}\), where \(W\) represents the weight matrices during the feature sequence mapping process. Therefore, self-attention can be represented by Equation (2):  
\[Self\text{-}Attention(H)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V \tag{2}\]  
In MulT [30], the Query and K-V pair in the self-attention computation process come from different modalities. Thus, MulT captures the interaction between the two modalities. MulT combines three modality pairs and calculates bidirectional modality interactions for each pair. As shown in Equation (3), for two modality feature sequences \(H_{1}\) and \(H_{2}\), MulT defines Query as \(Q_{1}=H_{1}W_{Q}\), Key as \(K_{2}=H_{2}W_{K}\), and Value as \(V_{2}=H_{2}W_{V}\). It calculates cross-modal attention in two directions between a pair of modalities:  
\[ \begin{split} Cross\text{-}Attention(H_{1}\to H_{2})& =softmax(\frac{Q_{1}K_{2}^{T}}{\sqrt{d_{k}}})V_{2}\\ Cross\text{-}Attention(H_{2}\to H_{1})& =softmax(\frac{Q_{2}K_{1}^{T}}{\sqrt{d_{k}}})V_{1}\end{split}  \tag{3}\]  
EMT [33] concatenates three unimodal utterance representations into a multimodal global representation. Inspired by EMT [33], we concatenate the utterance representations from each modality \(u_{m}\) as the global representation \(G=Concat(u_{t},u_{a},u_{v})\) during the cross-attention stage, where \(m\in(t,a,v)\). Subsequently, we utilize a Transformer to calculate intermodality information between the modality feature sequences \(F_{m}\in\mathbb{R}^{len\times d}\) and the global representation \(G\in\mathbb{R}^{3\times d}\), as shown in Figure 2 and Equation (4). \[ \begin{split} Attention(G\to F_{m})& =Cross\text{-}Attention(G\to F_{m})\\ Attention(F_{m}\to G)&=Cross\text{-}Attention(F_{m} \to G)\end{split}  \tag{4}\]  
On the other hand, we utilize modality-specific Transformer encoder layers, denoted as \(L_{s}\), to capture intramodality information for each modality individually (using Equation (2)). After encoding each modality, we use the self-attention mechanism in Transformer to process the unimodal feature sequences separately, in which the embedding at each position is able to learn the semantic and emotional information contained in the sequences. [PERSON] [30] used directional encoders for bimodal interactions separately, and subsequently augmented these dynamics with self-attention mechanisms. EMT [33] achieved cross-modal interactions by making global representations and unimodal sequences learn  
Figure 2: The structure of cross-attention. Cross-attention is used to capture dynamics between the global representation \(G\) and unimodal representations \(F_{m}\). from each other, while ignoring modality-specific information present in the self-attention unit. CCDA used both cross-modal attention and self-attention; first the two attention mechanisms were isolated, and then it used the cross-correlation loss to make them to interact after sufficiently learning the relevant intra- and intermodal information, respectively. This preserves the specificity information of the different attention mechanisms and optimizes the global representation by backpropagating the cross-modal feature sequences during the training progresses.