# HyFusER: Hybrid Multimodal Transformer for Emotion Recognition Using Dual Cross Modal Attention
## 5 Experimental Evaluation
### Experimental Results  
In this section, the proposed HyFusER model was evaluated using 10,381 text and speech data samples from the KEMDy dataset. The dataset was split into an 8:2 ratio for training and validation, with a primary focus on assessing the model's performance using text data. The performance evaluation was conducted using metrics such as Accuracy, Recall, Precision, and F1-Score. To address the imbalance in the number of samples across classes, this study utilized a weighted average approach. This method calculates the average performance metric by weighting the metric of each class by its sample size and dividing by the total number of samples. The formula is presented in Equation (10), where \(w_{i}\) denotes the sample size of the i-th class, and \(m_{i}\) represents the performance metric of the \begin{table}
\begin{tabular}{c c c} \hline \hline
**Emotion** & **Collected Data** & **Final Data Used** \\ \hline Angry & 1352 & 1352 \\ Disgust & 446 & 446 \\ Fear & 840 & 840 \\ Happy & 2643 & 2643 \\ Neutral & 15,082 & 2643 \\ Sad & 1028 & 1028 \\ Surprise & 1429 & 1429 \\
**Total** & **22,820** & **10,381** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: KEMDy data number by emotion. \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Loss & Sparse Categorical Crossentropy \\ Model Checkpoint Monitor & Val Sparse Categorical Accuracy \\ Batch Size & 32 \\ Epochs & 30 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: _Cont._i-th class. This approach ensures that the evaluation of model performance fairly accounts for the class imbalance within the dataset, allowing each class to contribute equitably to the overall assessment. \[\text{WeightedAverage}=\frac{\sum_{i=1}^{n}(w_{i}\times m_{i})}{\sum_{i=1}^{n}w_{i}} \tag{10}\]  
To evaluate the performance of the proposed model, comparative experiments were conducted with existing emotion recognition methodologies. The comparative models included an LSTM model for text data; a CNN model for speech data; and several multimodal approaches: an average ensemble model that predicts the final emotion by averaging the results of single-modality models, a model utilizing Bidirectional Cross Modal Attention, and the KoHMT [26] model. The experimental results are summarized in Table 5. The results show that single-modality models demonstrated relatively lower performance. In contrast, multimodal models, particularly the proposed HyFusER model, achieved superior performance by leveraging the complementary learning between text and speech data. These results validate that the hybrid approach, combining Intermediate Layer Fusion and Last Fusion, is highly effective for emotion recognition. This approach provides critical insights and contributes to advancing the field of emotion recognition. To verify the reliability and generalization capability of the proposed HyFusER model, the Korean Emotion Recognition Dataset provided by AI-HUB, composed of text and speech, was additionally utilized. This dataset enabled the evaluation of the generalizability of the results obtained from the KEMDy dataset and allowed the measurement of model performance when applied to data from a different source. The experimental setup for the AI-HUB dataset was consistent with that used for the KEMDy dataset. The data were split into training and validation sets in an 8:2 ratio, and all comparative models, including the proposed HyFusER model, were trained under identical conditions. This ensured a fair performance comparison across the two datasets. Performance evaluation was conducted using metrics such as Accuracy, Recall, Precision, and F1-Score, with results presented in Table 6. Each metric was calculated as a weighted average to account for class imbalance. Notably, by comparing the performance of the proposed model with that of single-modality models and other multimodal models, the superiority and robustness of the HyFusER model were demonstrated.