## 4 Model Design and Interpretation
### Cross-Modal Attention Module  
This paper employs cross-modal attention to focus on the interaction between different modal data, learning the semantic interaction weights for the speech and text modalities and readjusting the feature representations. Figure 7: Multi-modal fusion framework. Firstly, both speech and text representations are projected into the same space using \(1\)D-CNN, and the representations are as follows:  
\[\overline{H}\{S,T\}=Conv1D\left(H_{\{S,T\}},k_{\{S,T\}}\right)\in R^{d} \tag{1}\]  
where \(S\) represents the speech modality, \(T\) represents the text modality, \(H_{\{S,T\}}\) represents the final emotional feature representations for speech and text obtained from the feature extraction module, \(k_{\{S,T\}}\) represents the convolution kernel size for modality \(\{S,T\}\), and \(d\) denotes the dimension of the projected features for speech and text. The speech embeddings mapped using 1D-CNN are denoted as \(\overline{H}_{S}\) and \(\overline{H}_{T}\). We denote the process of transferring information from the speech modality to the text modality as \(S\to T\), and correspondingly, \(T\to S\) is used to represent the information transfer from the text modality to the speech modality. To learn the relationship between speech and text, linear projection is initially employed to transform each feature sequence into a query matrix \(Q\), a key matrix \(K\), and a value matrix \(V\). The calculation formula is as follows:  
\[Q_{l} =W_{l}^{Q}\overline{H}_{l} \tag{2}\] \[K_{l} =W_{l}^{K}\overline{H}_{l}\] \[V_{l} =W_{l}^{V}\overline{H}_{l}\]  
where \(Q_{l},K_{l},\mathrm{V}_{l}\in R^{d\times d}\) represent the query matrix \(Q\), key matrix \(K\), and value matrix \(V\) for the feature sequence of the modality, and \(W_{l}^{Q},W_{l}^{K},W_{l}^{V}\in R^{d\times d}\) represent the corresponding weight matrices. Next, the dot product operation is performed on the query matrix and the key matrix for both speech and text. The softmax function is then applied to scale and normalize the results row-wise to obtain the attention weights. Finally, the feature sequences are aggregated using the corresponding weights to obtain the interactive information transferred between the two modalities. 1. Cross-Modal Transfer \(S\to T\)  
The information from the speech modality is transferred to the text modality using a cross-modal attention mechanism with \(h\) heads. Unlike the original multi-head self-attention mechanism where \(Q=K=V\), with the cross-modal attention mechanism, the query matrix is \(Q_{S}\), and the key matrix and value matrix are \(K_{T}\) and \(V_{T}\), respectively. This mechanism facilitates the transfer of speech information to the text modality, enabling learning the text feature representations as guided by the speech information. The similarity is computed by taking the dot product of the query matrix \(Q_{S}\) from the speech and the key matrix \(K_{T}\) from the text. The \(Softmax\) function is applied to scale and normalize the results, followed by multiplying them with the value matrix \(V_{T}\) to obtain the attention weights. The specific formula is as follows:  
\[Att_{S\to T}\big{(}\overline{H}_{S},\overline{H}_{T}\big{)}=\text{softmax} \Bigg{(}\frac{Q_{S}K_{T}^{T}}{\sqrt{d_{k}}}\Bigg{)}V_{T} \tag{3}\]  
Then, the results from \(h\) heads are concatenated and mapped. The specific process is as follows:  
\[M_{S\to T}\big{(}\overline{H}_{S},\overline{H}_{T}\big{)}=Concat(Att_{S\to T}( 1),\dots,Att_{S\to T}(i),\dots,Att_{S\to T}(h))W \tag{4}\]  
where \(Att_{S\to T}(i)\) represents the \(i\)-th (where \(i\in[1,h]\)) cross-modal attention weight. Finally, residual connection and layer normalization are applied to the single-modal speech features \(\overline{H}_{S}\) mapped using 1D-CNN and the interactive multi-modal features \(M_{S\to T}\).