# Small Language Models for Speech Emotion Recognition in Text and Audio Modalities
## 4 Results
### Ensemble Models  
Based on these results, the best model in each category was selected to investigate the effect of adding contextual information on SER accuracy. For each model, its performance was evaluated using a copy of the same model to process the contextual information, as well as using the other models of the same modality to process the contextual information (see Figure 2). Table 3 summarizes the results of the different combinations studied. In all combinations analyzed, a significant improvement in accuracy was observed over the individual models. However, the combination of different models did not lead to an improvement compared to using the same architecture twice. In the text modality, accuracy improvements of more than 9 points were achieved in both WA and UA. Conversely, in the audio modality, the improvements were more discrete. The use of two Wav2 Vec2 models with contextual information only resulted in an increase in UA of about 2 points, while WA \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **Architecture** & **Modality** & **WA (\%)** & **UA (\%)** \\ \hline ALBERT & encoder-only & text & 64.65 & 64.79 \\ BERT & encoder-only & text & 69.61 & **70.64** \\ DeBERTa & encoder-only & text & **69.80** & 70.60 \\ ERNIE & encoder-only & text & 68.11 & 69.67 \\ RoBERTa & encoder-only & text & 68.62 & 70.55 \\ BART & encoder-decoder & text & **69.16** & **70.23** \\ T5 & encoder-decoder & text & 66.20 & 67.52 \\ GPT-2 & decoder-only & text & 66.43 & 67.38 \\ GPT-Neo & decoder-only & text & 63.73 & 64.27 \\ OPT & decoder-only & text & **68.49** & **69.73** \\ Pythia & decoder-only & text & 64.31 & 65.97 \\ XLNet & decoder-only & text & 67.34 & 68.17 \\ HuBERT & encoder-only & audio & 68.85 & 66.99 \\ SEW & encoder-only & audio & 67.81 & 64.35 \\ WavLM & encoder-only & audio & 69.13 & 66.81 \\ Wav2 Vec2 & encoder-only & audio & **71.56** & **68.73** \\ S2T & encoder-decoder & audio & 65.30 & 63.34 \\ SpeechT5 & encoder-decoder & audio & 67.83 & 68.07 \\ Whisper & encoder-decoder & audio & **72.95** & **72.82** \\ \hline \hline \end{tabular} Note: Bold indicates the highest value per architecture type and modality.  
\end{table} 
Table 2: Cross-validation results of the baseline models, showing the average accuracy over 5 folds. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Utterance** & **Context** & **Modality** & **WA (\%)** & **UA (\%)** \\ \hline BERT & BERT & text & 75.83 & 78.38 \\ BERT & BART & text & 77.25 & 78.42 \\ BERT & OPT & text & 75.48 & 78.13 \\ BART & BART & text & 76.84 & 79.07 \\ BART & BERT & text & 75.47 & 78.32 \\ BART & OPT & text & 75.29 & 77.38 \\ OPT & OPT & text & 77.70 & **79.15** \\ OPT & BERT & text & 76.22 & 77.85 \\ OPT & BART & text & 76.92 & 78.68 \\ Wav2 Vec2 & Wav2 Vec2 & audio & 71.72 & 70.84 \\ Wav2 Vec2 & Whisper & audio & 72.28 & 71.60 \\ Whisper & Whisper & audio & **76.86** & **76.31** \\ Whisper & Wav2 Vec2 & audio & 75.73 & 75.17 \\ \hline \hline \end{tabular} Note: Bold indicates the highest value per modality.  
\end{table} 
Table 3: Cross-validation results for unimodal ensemble models using contextual information. remained almost the same. On the other hand, the use of two Whisper models resulted in an increase of almost 4 points in both WA and UA. In addition, the impact of combining text and audio models was evaluated to determine how the integration of different modalities affects overall performance, as shown in Table 4. The best multimodal classification results were obtained with the combination of Whisper and OPT. However, all the combinations studied achieved accuracies higher than those reported by the baseline models (see Table 2).