# Small Language Models for Speech Emotion Recognition in Text and Audio Modalities
## 5 Discussion  
Speech emotion recognition is a complex task that is usually addressed by implementing specific models for this purpose, usually involving a combination of different feature extraction methods or the use of complex attention mechanisms. The most common approach, as evidenced by the works reviewed in Section 2.2, often employs one or more SLMs as a backbone and their subsequent optimization by means of various strategies such as exhaustive hyperparameter tuning [52], self-labeling [76], use of advanced attention mechanisms [75], or customized loss functions [85]. In this paper, we have proposed a different approach. We fine-tuned 19 different general-purpose SLMs on a limited number of epochs, adding only a dropout layer and a linear layer to perform classification. We deliberately refrained from data augmentation, exhaustive hyperparameter tuning, or any other technique that could improve the accuracy of the analyzed models. Given that the architectures analyzed are typically used as the backbone of more complex models, our experiments aimed to establish a baseline that can serve as a reference for future research, providing valuable information about the architectures and their combinations that yield the best performance with minimal modifications. The classification results of the baseline models highlight the importance of the choice of architecture for SER, as differences of more than 5 points were obtained between different architectures, despite having a similar number of trainable parameters. Encoder-only models are usually considered the most suitable for classification tasks and are the most widely used in the literature. In our study, these models were the most accurate for text classification. However, BART (encoder-decoder) and OPT (decoder-only) reported accuracies similar to those of the encoder-only architectures. Furthermore, when used in ensemble architectures with contextual information, BART and OPT slightly outperformed BERT. The \"Neutral\" class was the most frequently misclassified across both text and audio models, likely because neutral utterances often lack strong prosodic variations or distinctive lexical markers that facilitate the recognition of other emotions. Additionally, in a subject-independent evaluation, the neutral tone of one speaker may resemble the slightly emotive tone of another, increasing classification difficulty. This pattern aligns with prior studies on the same dataset, where neutral is also the worst-classified class [82; 83; 91; 92; 95; 96; 97]. In the case of audio models, although encoder-only models are the most commonly used in the literature, the experiments conducted showed that Whisper, an encoder-decoder model (only the encoder pair was used in this work), produced the best classification results. This is particularly remarkable considering that Whisper is the architecture with the lowest number of trainable parameters and the lowest computational complexity. These results challenge the widely accepted notion that encoder-only models are the best performers in classification tasks, and invite reflection on the potential of other types of architectures. The observed performance differences across models highlight the significant influence of pre-training data and objectives alongside architectural design. For instance, Whisper outperforms other audio models like Wav2 Vec2 despite having fewer parameters and lower complexity, which likely reflects its extensive pre-training on a much larger and more diverse dataset. Similarly, among text models sharing similar architectures, variations in accuracy (e.g., OPT vs. GPT-2) suggest that pre-training corpus size and diversity, as well as the specific tasks used during pre-training, critically affect the models' capacity to capture relevant emotional cues. Given that large-scale pre-training requires vast computational and data resources often unavailable to most researchers, our analysis considers architecture and pre-training jointly.