# Multi-Label Emotion Recognition of Korean Speech Data Using Deep Fusion Models
## 5 Results and Performance Analysis
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Emotion**} & \multicolumn{2}{c}{**Binary Accuracy**} & \multicolumn{2}{c}{**Avg. Binary Accuracy**} & \multirow{2}{*}{**Recall**} & \multirow{2}{*}{**Precision**} & \multirow{2}{*}{**F1-Score**} & \multicolumn{2}{c}{**Avg. F1-Score**} \\ \cline{3-4} \cline{6-8}  & Happiness & 0.762 & & 0.423 & 0.582 & 0.490 & \\ \cline{2-8}  & Sadness & 0.617 & & 0.657 & 0.512 & 0.575 & \\ \cline{2-8}  & Anger & 0.663 & 0.678 & 0.448 & 0.454 & 0.451 & \\ \cline{2-8}  & Neutral & 0.658 & & 0.611 & 0.532 & 0.569 & \\ \cline{2-8}  & Disgust & 0.693 & & 0.518 & 0.437 & 0.474 & \\ \hline \multirow{4}{*}{\
\begin{tabular}{c} Vision- \\ Transformer \\ \end{tabular} } & Happiness & 0.773 & & 0.476 & 0.600 & 0.531 & \\ \cline{2-8}  & Sadness & 0.692 & & 0.387 & 0.701 & 0.499 & \\ \cline{2-8}  & Anger & 0.693 & & 0.304 & 0.507 & 0.380 & \\ \cline{2-8}  & Neutral & 0.647 & & 0.672 & 0.517 & 0.584 & \\ \cline{2-8}  & Disgust & 0.728 & & 0.506 & 0.493 & 0.499 & \\ \hline \multirow{4}{*}{
\begin{tabular}{c} DNN \\ DNN \\ \end{tabular} } & Happiness & 0.542 & & 0.586 & 0.311 & 0.406 & \\ \cline{2-8}  & Sadness & 0.521 & & 0.827 & 0.447 & 0.580 & \\ \cline{2-8}  & Anger & 0.671 & & 0.093 & 0.368 & 0.149 & \\ \cline{2-8}  & Neutral & 0.606 & & 0.413 & 0.453 & 0.432 & \\ \cline{2-8}  & Disgust & 0.518 & & 0.542 & 0.290 & 0.378 & \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Performance comparison of the ViT, 1D CNN-LSTM, and DNN models. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Emotion**} & \multicolumn{2}{c}{**Binary Accuracy**} & \multicolumn{2}{c}{**Avg. Binary Accuracy**} & \multirow{2}{*}{**Recall**} & \multirow{2}{*}{**Precision**} & \multirow{2}{*}{**F1-Score**} & \multicolumn{2}{c}{**Avg. F1-Score**} \\ \cline{3-4} \cline{6-8}  & Happiness & 0.776 & & 0.471 & 0.610 & 0.531 & \\ \cline{2-8}  & Sadness & 0.684 & & 0.481 & 0.631 & 0.546 & \\ \cline{2-8}  & Anger & 0.654 & 0.712 & & 0.621 & 0.456 & 0.526 & \\ \cline{2-8}  & Neutral & 0.686 & & 0.528 & 0.582 & 0.554 & \\ \cline{2-8}  & Disgust & 0.761 & & 0.368 & 0.584 & 0.451 & \\ \hline \multirow{4}{*}{\begin{tabular}{c} Vision- \\ Transformer and DNN \\ \end{tabular} } & Happiness & 0.774 & & 0.297 & 0.692 & 0.415 & \\ \cline{2-8}  & Sadness & 0.612 & & 0.633 & 0.508 & 0.563 & \\ \cline{2-8}  & Anger & 0.657 & 0.690 & 0.323 & 0.428 & 0.368 & \\ \cline{2-8}  & Neutral & 0.674 & & 0.471 & 0.571 & 0.516 & \\ \cline{2-8}  & Disgust & 0.732 & & 0.433 & 0.499 & 0.464 & \\ \hline \multirow{4}{*}{\
\begin{tabular}{c} Vision- \\ Transformer, 1D CNN-LSTM, \\ and DNN \\ \end{tabular} } & Happiness & 0.779 & & 0.378 & 0.657 & 0.480 & \\ \cline{2-8}  & Sadness & 0.663 & & 0.616 & 0.568 & 0.591 & \\ \cline{2-8}  & Anger & 0.669 & 0.701 & 0.287 & 0.444 & 0.349 & \\ \cline{2-8}  & Neutral & 0.658 & & 0.561 & 0.535 & 0.547 & \\ \cline{2-8}  & Disgust & 0.739 & & 0.356 & 0.518 & 0.422 & \\ \hline \multirow{4}{*}{
\begin{tabular}{c} Vision- \\ Transformer, 1D CNN-LSTM, \\ and DNN \\ \end{tabular} } & Happiness & 0.774 & & 0.514 & 0.593 & 0.551 & \\ \cline{2-8}  & Sadness & 0.674 & & 0.617 & 0.583 & 0.600 & \\ \cline{2-8}  & Anger & 0.665 & & 0.466 & 0.459 & 0.463 & \\ \cline{2-8}  & Neutral & 0.657 & & 0.693 & 0.527 & 0.599 & \\ \cline{2-8}  & Disgust & 0.762 & & 0.221 & 0.664 & 0.332 & \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Performance comparison of fusion models. Next, as shown in Table 9, the ViT and 1D CNN-LSTM fusion model achieved the highest performance, with an average binary accuracy of 71.2% and an average f1-score of 52.2%. By integrating the Log-mel spectrogram and MFCC features, the model learned diverse characteristics, such as frequency, amplitude, and frequency coefficients over time, which can facilitate the identification of complex emotions. Based on these results, the ViT and 1D CNN-LSTM fusion model performed the best.