# HyFusER: Hybrid Multimodal Transformer for Emotion Recognition Using Dual Cross Modal Attention
## 4 Hybrid Multimodal Transformer for Emotion Recognition
### Hybrid Multimodal Transformer for Emotion Recognition Model
#### 4.3.1 Intermediate Layer Fusion  
The embedded text and speech data are each passed through a transformer encoder, which is used to learn the contextual and structural features of the input data. The transformer encoder used in this study consists of three layers, with each layer comprising Multi-Head Attention and a Position-Wise Feed Forward Network. Multi-Head Attention performs multiple attention heads in parallel, learning the input data from various perspectives. This reduces dependency on specific parts of the data and enables the learning of more comprehensive features. The Position-Wise Feed Forward Network applies a non-linear transformation to the attention output to learn more complex patterns, which is applied independently to each sequence element. Through this process, the text and speech  
Figure 4: Each modality Standard Scaling process.  
data maintain their unique characteristics while being effectively learned. The structure of the transformer encoder is shown in Figure 5.  
Cross Modal Attention is utilized to learn the interactions between text and speech, extracting the most significant information from both modalities simultaneously. However, the learning direction in Cross Modal Attention depends on the configuration of Query (_Q_), Key (_K_), and Value (_V_). For example, if text is set as the Query and speech as the Key and Value, the model learns to interpret speech based on text information. This unidirectional learning approach may limit the complementary learning between text and speech. To address this limitation, a Dual Cross Modal Attention learning strategy is introduced, dividing the learning process into Step 1 and Step 2 to enable bidirectional and more effective cross modal learning.  
1. Step 1: In this step, the text embeddings are set as the Query, and the speech embeddings are set as the Key and Value. The learning process, as described in Equation (7), complements the features of speech based on the contextual information from the text. This approach enhances emotion recognition performance by focusing on text-centric features. \[\text{CrossModalAttention}(Q_{T},K_{S},V_{S})=\text{softmax}(\frac{Q_{T}K_{S}^{T }}{\sqrt{d_{k}}})V_{S}\] (7)
2. Step 2: In this step, the speech embeddings are set as the Query, and the text embeddings are set as the Key and Value. The learning process, as described in Equation (8), leverages the temporal and acoustic characteristics of speech to learn the semantic information from text. This approach improves emotion recognition performance by focusing on speech-centric features. \[\text{CrossModalAttention}(Q_{S},K_{T},V_{T})=\text{softmax}(\frac{Q_{S}K_{T} ^{T}}{\sqrt{d_{k}}})V_{T}\] (8)  
The results learned through Step 1 and Step 2 are reflected as two distinct probability distributions: a text-centric probability distribution that primarily captures the nuances of emotions perceived through text, and a speech-centric probability distribution that embodies the unique characteristics of speech, such as tone, pitch, and intensity. Each of these probability distributions represents the unique perspectives of text and speech, acting complementarily to enhance emotion recognition performance. These distributions serve  
Figure 5: Transformer encoder structure.  
as key inputs for the subsequent stages, leveraging the strengths of both modalities to achieve a more comprehensive understanding of emotions.