# Small Language Models for Speech Emotion Recognition in Text and Audio Modalities  
[PERSON]  
1 Instituto de Investigacion en Informatica de Albacete, Universidad de Castilla-La Mancha, 20271 Albacete, Spain; [EMAIL_ADDRESS] (J.L.G.-S.); francisco.looperzos@uclms (E.L.L.R. ); daniel.sanchez@uclms (D.S.-R.); robertson.sanchez@uclms (R.S.-R.)
2 Departamento de Sistemas Informaticos, Universidad de Castilla-La Mancha, 20271 Albacete, Spain
Cepstral Coefficients (MFCCs) combined with traditional machine-learning (ML) classifiers such as support vector machines (SVMs) have been successfully used for SER. However, in recent years, deep-learning (DL) methods have been increasingly adopted and gradually outperform traditional methods [8]. Deep-learning, and especially transformer-based architectures, achieve state-of-the-art accuracy on most popular SER datasets [9]. Part of the success of these approaches lies in their general-purpose design. Unlike traditional methods, whose knowledge capability is usually problem-specific, knowledge from a deep model can be easily transferred to perform a different task. This capability allows DL models to learn not only from small, manually labeled SER datasets, but also from much larger datasets focused on a different task, or even from unlabeled data [10]. The year 2018 marked a turning point in NLP, with the presentation of bidirectional encoder representations from transformers (BERT) [11] and generative pre-training (GPT) [12]. These transformer-based models demonstrated a great ability to capture longer-term dependencies in language in various NLP tasks, such as text generation, translation, summarization, question answering, or sentiment analysis. With 110 and 117 million parameters, respectively, the BERT and GPT architectures were originally considered large language models (LLMs). Since then, there has been significant progress in the development of much larger models, with architectures of billions or even hundreds of billions of parameters becoming standard [13]. As a result, models with just over 100 million parameters, previously considered large, are now referred to in the scientific community as small language models (SLMs) [14; 15; 16]. In general, LLMs report higher performance than SLMs in a wide range of NLP tasks [17]. However, their large size limits the usability of this type of model due to the large amount of computational resources required for both training and inference. On the other hand, SLMs can be fine-tuned relatively quickly on consumer-grade computers. And using techniques such as quantization [18] or low-rank adaptation (LoRA) [19], SLMs can be implemented in low-end devices, greatly increasing the number of potential applications for this type of model. The relatively light weight of SLMs facilitates the assembly of multiple models, which can significantly increase system performance in SER, especially when combining models of different modalities such as audio and text [20]. Several review articles published in recent years highlight the growing number of approaches using language models for SER [9; 21; 22], mostly models based on BERT for text processing or Wav2 Vec2 [23] for audio processing. However, these models are often used as the backbone within more complex architectures, and research often focuses on the proposed architecture, sometimes neglecting the importance of the choice of language model. In this paper, the performance of some of the most popular SLMs for text and audio modalities on the SER task is evaluated using a public benchmark dataset. All models are evaluated based on their accuracy and computational efficiency, with the aim of determining the most suitable architectures for the proposed task for each modality. Subsequently, the selected models are combined into an ensemble architecture to investigate the effect of combining models of the same modality but from different families on system performance, as well as the effect of combining text and audio modality models.