# Small Language Models for Speech Emotion Recognition in Text and Audio Modalities
## 3 Materials and Methods
### Experimental Setup  
Training and validation of the different models was done by 5-fold cross-validation, where 4 sessions in each fold were used for training and the remaining session was used for validation. All SLMs were initialized with the pre-trained weights available from Hugging Face ([[https://huggingface.co/](https://huggingface.co/)]([https://huggingface.co/](https://huggingface.co/)), accessed on 12 January 2025). Given that these models are pre-trained, a low learning rate (\(10^{-5}\)) was chosen. In addition, a small batch size of 12 was chosen and an early stopping mechanism was applied to prevent overfitting. This mechanism consisted of stopping the training process if there was no improvement in the validation unweighted accuracy (UA) after 3 epochs. Furthermore, a maximum of 20 epochs was set for the training duration. In addition, the loss function was weighted by class weights (inversely proportional to class size) to compensate for the effect of class imbalance of the IEMOCAP dataset. The audio models were trained using 10-s sequences, with padding or truncation applied as necessary to adjust them to that duration. In the case of Whisper, which is configured to support only 30-s sequences by default, the positional embedding layer was truncated to work with 10-s sequences. This change was made to allow a fair comparison with the other architectures. However, this truncation was not strictly necessary, as Whisper could have been trained by padding the audio inputs with 20 s of silence. An attention mask is applied to prevent the model from attending to these silent sections, which would hypothetically yield similar results to those obtained here, but with a significantly higher computational cost. All individual and text ensemble models were trained on a workstation with an NVIDIA RTX 2070 Super GPU with 8 GB of VRAM, while the audio ensemble and multimodal models were trained on a workstation with an NVIDIA P100 GPU with 16 GB of VRAM. The evaluation of the models was performed on the basis of weighted accuracy (WA) and UA, with the latter metric being used to select the best model. Different definitions of WA and UA exist in the literature, making it difficult to compare results. In this paper, one of the most extended definitions is used, where WA is considered as the percentage of correct predictions, while UA is defined as the average of the recall metric for each class:  
\[WA=\frac{\sum_{i=1}^{K}TP_{i}}{\sum_{i=1}^{K}(TP_{i}+FN_{i})} \tag{1}\] \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model Name** & **Version** & **Modality** & **Parameters** & **GFLOPs** \\ \hline T5 & base (encoder-only) & text & 109.63 & 17.72 \\ GPT-2 & 124m & text & 124.44 & 8.50 \\ GPT-Neo & 125m & text & 125.20 & 8.69 \\ OPT & 125m & text & 125.24 & 8.69 \\ Pythia & 160m & text & 123.69 & 8.50 \\ XLNet & base & text & 117.31 & 10.29 \\ HuBERT & base & audio & 94.37 & 69.61 \\ SEV & small & audio & 89.62 & 25.26 \\ WavLM & base & audio & 94.39 & 69.64 \\ Wav2 Vec2 & base & audio & 94.37 & 69.61 \\ S2T & ASR-large (encoder-only) & audio & 156.81 & 40.83 \\ SpeechT5 & ASR (encoder-only) & audio & 94.40 & 76.49 \\ Whisper & small (encoder-only) & audio & 87.01 & 43.58 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: _Cont._\[UA=\frac{1}{K}\sum_{i=1}^{K}\frac{TP_{i}}{TP_{i}+FN_{i}} \tag{2}\]  
where \(K\) is the number of classes, \(TP_{i}\) represents the true positives for class \(i\), and \(FN_{i}\) represents the false negatives for class \(i\). After training the baseline models, the model with the best validation AU in the text modality was selected for each category. In the audio modality, where no decoder-only model was used, the best encoder-only model and the best encoder-decoder model (using only the encoder part in this experiment) were selected. With the selected models, the effect of adding contextual information was investigated by combining different models or by using two copies of the same model, one for the utterance and one for the contextual information.