# M2 ER: Multimodal Emotion Recognition Based on Multi-Party Dialogue Scenarios
## 2 Detailed Design
### Multimodal Emotion Recognition Based on Attention Mechanism
#### 2.2.1 Cross-Modal Attention Interaction  
Because multimodal emotion recognition often involves a large number of features, determining the importance of these features and capturing the relationships among multimodal emotional features are key issues. In our fusion model, we incorporate the emotional features extracted from the text, audio, and video modalities. This is achieved by utilizing cross-modal attention to facilitate the interaction among different modalities. The input to the Cross-Modal Attention Interaction--_Part 1_ is the emotional features of the text, audio, and video modalities, represented as \(M^{T}\), \(M^{A}\), and \(M^{V}\), respectively. The specific architecture of _Part 1_ is shown in Figure 10.  
Figure 9: Video feature extraction model. Taking the example of inputting the text and audio modality into the Cross-Modal Attention Interaction module, \(M^{T^{*}}\) represents the feature representation with interaction obtained from the text through this module. The calculation formula for \(M^{T^{*}}\) is shown in Equations (1)-(3). Similarly, \(M^{A^{*}}\) represents the feature representation with interaction obtained from the audio modality through the T-A attention module. The calculation formula for \(M^{A^{*}}\) is shown in Equations (4)-(6). \[H^{TA}=M^{T}{M^{A}}^{T}, \tag{1}\]  
\[\alpha^{TA}=softmax(H^{TA}, \tag{2}\]  
\[M^{T^{*}}=(\alpha^{TA}M^{T})*M^{T}, \tag{3}\]  
\[H^{AT}=M^{A}{M^{T}}^{T}, \tag{4}\]  
\[\alpha^{AT}=softmax(H^{AT}, \tag{5}\]  
\[M^{A^{*}}=(\alpha^{AT}M^{A})*M^{A}, \tag{6}\]  
where \(H^{TA}\) and \(H^{AT}\) represent the cross-modal interaction information between the text and audio modalities. \(\alpha^{TA}\) and \(\alpha^{AT}\) represent the scores obtained for the text and audio modalities in the cross-modal attention interaction. By applying the soft attention mechanism to the emotional features of the input text and audio modalities in _Part 1_ and multiplying \(M^{T}\), \(M^{A}\) with the corresponding elements of their respective matrices, we obtain the feature representation with interaction for the text and audio modalities \(M^{T^{*}}\), \(M^{A^{*}}\). _Part 1_ is divided into three main parts: text-audio attention interaction (\(T\)-\(A\)), text-video attention interaction (\(T\)-\(V\)), and audio-video attention interaction (A-V). 1. _T-A_: The emotional features \(M^{T}\), \(M^{A}\) of the input text and audio modalities in _Part 1_ are used to obtain the interaction representation between text and audio \(M^{T^{*}}\), \(M^{A^{*}}\) through cross-modal attention;
2. _T-V_: The emotional features \(M^{T}\), \(M^{V}\) of the input text and video modalities in _Part 1_ are used to obtain the interaction representation between text and video, \(M^{T^{1}}\), \(M^{V^{*}}\) through cross-modal attention;
3. _A-V_: The emotional features \(M^{A}\), \(M^{V}\) of the input audio and video modalities in _Part 1_ are used to obtain the interaction representation between audio and video \(M^{A1^{*}}\), \(M^{V1^{*}}\) through cross-modal attention. Finally, we obtained the interaction feature representations of the text modality: \(M^{T^{*}}\), \(M^{T^{1}}\); the interaction feature representations of the audio modality: \(M^{A^{*}}\), \(M^{A1^{*}}\); and the interaction feature representations of the video modality: \(M^{V^{*}}\), \(M^{V1^{*}}\). These representations are concatenated with the respective emotional features of each modality using  
Figure 10: Cross-modal attention interaction architecture. fully connected layers to obtain the complete representation of the text, audio, and video modalities' emotional features. The calculation formulas for this process are shown in Equations (7)-(9). \[F^{T}=\tanh(W^{T}[M^{T}\oplus M^{T^{*}}\oplus M^{T1^{*}}]+b^{T}), \tag{7}\]  
Figure 9:0  
Figure 9:1  
\(W^{T}\), \(W^{A}\), \(W^{V}\), \(b^{T}\), \(b^{A}\), \(b^{V}\) are the parameters to be learned, \(\oplus\) denotes the concatenation operation.