# Small Language Models for Speech Emotion Recognition in Text and Audio Modalities
## 3 Materials and Methods
### Language Models  
A selection of architectures with a similar number of trainable parameters and computational complexity was made. Table 1 lists all the baseline architectures evaluated, along with the number of trainable parameters for each model and the number of giga-floating point operations (GFLOPs) required to process one utterance. Parameter count is a common proxy for model complexity, but it does not always correlate with actual computational cost. GFLOPs provide a more direct measure of the operations required during inference. For instance, ALBERT reduces parameter count through techniques such as cross-layer parameter sharing, yet maintains similar GFLOPs to BERT. This highlights the relevance of GFLOPs when comparing the efficiency of different architectures. A fixed length of 100 tokens for text and 10 s for audio was used for each utterance as approximately 99% of the text sequences and over 90% of the audio sequences fell within these limits. The difference in coverage percentages is justified by the heavier computational demands of audio processing and the unique characteristics of prosody in audio data, which enable the prediction of emotional content even in incomplete dialogue segments. The value of GFLOPs was estimated using the fvcore [73] library developed by FacebookAI Research ([[https://ai.meta.com/research/](https://ai.meta.com/research/)]([https://ai.meta.com/research/](https://ai.meta.com/research/)), accessed on 2 February 2025). Models listed with (encoder-only) next to their name are encoder-decoder models where only the encoder part is used. In such a case, the values in Table 1 refer to the part of the model used, not the entire architecture.  
The baseline models were used as feature extractors. A 0.1 dropout layer was added to the output of the models to prevent overfitting, followed by a linear layer used as a classifier. The combination of different SLMs was also investigated. In this case, the output of the SLM was taken to a linear layer to equalize the output dimensionality of the different models before merging their features. The output features were then fused by concatenation. Figure 1 shows the proposed multimodal ensemble architecture, which represents the combination of four models. The same structure was followed when only two models were combined.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model Name** & **Version** & **Modality** & **Parameters** & **GFLOPs** \\ \hline ALBERT & v2-base & text & 11.69 & 8.70 \\ BERT & base & text & 109.49 & 8.50 \\ DeBERTa & v3-base & text & 184.43 & 16.88 \\ ERNIE & v2-base & text & 109.49 & 8.69 \\ RoBERTa & base & text & 124.65 & 8.69 \\ BART & base & text & 140.01 & 9.92 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Number of trainable parameters (millions) and GFLOPs for the baseline architectures used in the study.  
Figure 1: Multimodal ensemble architecture. Feature vectors are projected to a common dimension using a fully connected layer (FC 256), followed by a dropout layer with probability 0.1 (Dropout 0.1), and a final classification layer with four output units (FC 4).