# HyFusER: Hybrid Multimodal Transformer for Emotion Recognition Using Dual Cross Modal Attention
## 4 Hybrid Multimodal Transformer for Emotion Recognition
### Hybrid Multimodal Transformer for Emotion Recognition Model
#### 4.3.2 Last Fusion  
In the Last Fusion stage, the two outputs derived from the Intermediate Layer Fusion are combined to predict the final emotion. This stage aggregates the characteristic information from the text-centric probability distribution and the speech-centric probability distribution using a GlobalAveragePooling1D Layer and a Dense Layer, subsequently predicting emotions as probabilities. The final emotion is determined through an average ensemble of these probabilities. The GlobalAveragePooling1D Layer compresses the sequence information of each modality, extracting global features. The Dense Layer uses the softmax function to calculate probabilities for each emotion. This structure preserves the unique characteristics of each data type while providing suitable vector representations for emotion classification. The detailed architecture of the Last Fusion stage can be seen in Figure 6.  
The average ensemble is used to merge the emotion predictions, calculated as probabilities, to produce more refined emotion predictions. This process integrates the learning results of text and speech data with equal weights, reflecting the strengths of both data modalities while minimizing biases from individual learning. The average ensemble is defined in Equation (9), which represents the process of generating the final prediction by taking the simple average of the emotion probability distributions derived from each modality. This approach ensures balanced contributions from both text and speech, leading to a more robust and accurate emotion recognition outcome. \[Y=\text{argmax}(\frac{Y_{Y}+Y_{S}}{2}) \tag{9}\]  
This ensemble strategy is particularly effective in addressing discrepancies that may arise from the inherent characteristics of text and speech data. For example, text data often excel in capturing the semantic meaning of emotions, while speech data provide valuable paralinguistic cues such as tone and pitch. By equally weighting these two modalities, the model avoids over-reliance on either modality, ensuring a more holistic understanding of  
Figure 6: Last Fusion structure. emotional expressions. Furthermore, this method enhances the generalization capability of the model, as it leverages complementary information from both modalities rather than treating them independently. This approach enhances the prediction consistency across various emotion classes and improves the model's stability. For instance, if the prediction probabilities for a specific emotion are 0.85 and 0.80 from text and speech, respectively, the average ensemble calculates the final prediction probability as \((0.85+0.80)/2=0.825\). Among the calculated probabilities for each emotion, the highest probability determines the final emotion. This simple yet effective technique not only smoothens prediction fluctuations across modalities but also ensures that less confident predictions from one modality are supplemented by the strengths of the other. Table 2 provides an example of the prediction results for a sentence. This method more accurately reflects the interaction between the two modalities, thereby improving the reliability and accuracy of emotion recognition. As such, the Last Fusion stage plays a pivotal role in synthesizing the outcomes of multimodal learning, maximizing the complementary learning between text and speech. For example, in cases where a speech signal may lack clarity due to noise, the text-based analysis can compensate by providing accurate semantic context, and vice versa. This cross modal interaction highlights the model's ability to handle real-world data variability and ensures that the final emotion prediction is both consistent and robust. Algorithm 1 explains the entire process of the proposed model.