# Small Language Models for Speech Emotion Recognition in Text and Audio Modalities
## 4 Results
### Ensemble Models
Considering that Whisper and OPT showed the best performance in both unimodal and multimodal ensemble architectures, these models were selected to build a multimodal ensemble architecture that supports contextual information according to the scheme shown in Figure 1. The ensemble model is composed of four SLMs, namely two Whisper models to process the audio with contextual information, and two OPT models to process the audio transcripts, also with contextual information. The proposed ensemble model showed a WA of 82.12% and a UA of 83.34%. Figure 3 shows the confusion matrix combined from the 5 folds, where the lowest accuracy corresponds to the neutral class (74.06%) and the highest to the angry class (89.66%). The model showed rapid convergence, reaching its maximum validation accuracy after an average of only 2.2 epochs. In addition, it achieved a WA of 80.11% and a UA of 81.68% after only one epoch. Furthermore, it is important to highlight the stability of the results, WA remained above 80% and UA above 81.5% during the first 4 epochs. Due to the early stopping criterion used, results are only available for the first 4 epochs in the 5 folds. However, in cases where training extended beyond this range, such as the fold that used session 1 as the validation set, where training stopped at epoch 7, both WA and UA metrics remained stable over the 7 epochs, with standard deviations of 1.4% for both metrics. To better understand the contribution of each component in the final model, we conducted an ablation study by progressively removing individual elements from the full ensemble, see Table 5. The results showed that removing the audio context led to a slight decrease in performance (UA dropped from 83.34% to 82.76%), while removing the text context caused a more pronounced decline (UA decreased to 81.11%). Excluding the entire text modality resulted in a significant drop in accuracy (UA fell to 76.31%). Similarly, eliminating both audio context and text context reduced performance considerably. These findings demonstrated that both modalities and their contextual information contributed meaningfully to the model's effectiveness. Figure 4 shows the learning curve of the proposed model during the first epoch, using session 5 as the validation set. The model achieved an accuracy of more than 80% after training with less than 150 batches, which represents 50% of the total dataset. Although some instability is observed at the beginning, both accuracy and loss tend to stabilize towards the end of the epoch. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Audio Model** & **Text Model** & **WA (\%)** & **UA (\%)** \\ \hline Wav2 Vec2 & BERT & 76.93 & 76.92 \\ Wav2 Vec2 & BART & 75.36 & 75.99 \\ Wav2 Vec2 & OPT & 76.11 & 75.79 \\ Whisper & BERT & 76.33 & 75.48 \\ Whisper & BART & 76.42 & 74.62 \\ Whisper & OPT & **79.06** & **78.05** \\ \hline \hline \end{tabular} Note: Bold indicates the highest value.  
\end{table} 
Table 4: Cross-validation results for multimodal ensemble model without contextual information. Figure 4: Learning curve of the proposed ensemble model (Whisper + OPT with context) during the first epoch. Figure 3: Normalized confusion matrix for the final proposed ensemble model (Whisper + OPT with context) from 5-fold cross-validation. \begin{table}
\begin{tabular}{c c c c c c} \hline \multicolumn{2}{c}{**Audio**} & \multicolumn{2}{c}{**Text**} & \multicolumn{2}{c}{**WA (\%)**} & \multicolumn{1}{c}{**UA (\%)**} \\ \hline
**Utterance** & **Context** & **Utterance** & **Context** & & \\ \hline ✓ & ✓ & ✓ & ✓ & **82.12** & **83.34** \\ ✓ & — & ✓ & ✓ & 81.84 & 82.76 \\ ✓ & ✓ & ✓ & — & 80.66 & 81.11 \\ ✓ & ✓ & — & — & 76.86 & 76.31 \\ — & — & ✓ & ✓ & 77.70 & 79.15 \\ ✓ & — & ✓ & – & 79.06 & 78.05 \\ \hline \end{tabular} Note: Bold indicates the highest value.  
\end{table} 
Table 5: Ablation study of the proposed model. WA (%) and UA (%) after removing each component.