# Small Language Models for Speech Emotion Recognition in Text and Audio Modalities
## 2 Related Work
### SLMs for SER  
Since the presentation of the BERT model in 2018, a large number of research papers have been published using this architecture or its variants as the backbone for SER from audio transcription, often combined with acoustic processing models to create multimodal systems. For example, a trimodal model was developed by merging text features extracted with RoBERTa, visual features extracted with DenseNet, and audio features extracted with OpenSmile through a long short-term memory (LSTM) and a multimodal attention module [47]. Another proposal fused MFCC features together with BERT embeddings in a recurrent neural network (RNN) with different attention mechanisms [48]. In the same vein, the location of the merging layer effect in a hybrid model combining BERT together with MFCC and other audio features has been explored [49]. The results of this work reflect an increase in the efficiency of the model. However, they point out a small influence of the location of the merging layer on the results. Similarly, a BERT model was combined with an attention-based gated recurrent unit (GRU) for processing audio features from MFCC and different types of spectrograms [50]. The same approach was later used by replacing the BERT encoder with ALBERT and performing data augmentation using a generative adversarial network (GAN), which significantly improved the accuracy of the model [51]. On the other hand, a multihead attention module was used to fuse MFCC and other audio features with features extracted using a RoBERTa-based model, along with electroencephalogram (EEG) features, to create a multimodal brain-computer fusion network for human intention recognition [52]. In addition, a considerable amount of research has been done on architectures that combine BERT-type encoders for text feature extraction with CNNs for audio feature extraction from spectrograms or MFCCs. A visual geometry group (VGG)-type architecture with MEL spectrograms as input has been used in some works [53; 54]. A modified AlexNet was used for feature extraction from audio spectrograms [55], and a similar approach but using three-dimensional spectrograms was also adopted [56]. A customized CNN incorporating RNN layers has also been proposed [57]. Finally, another customized CNN, but followed with a channel-wise global head-pooling, has been introduced [58]. On the other hand, although auto-regressive models are not commonly used in classification tasks, several proposals using this approach for SER have been presented recently. A model based on XLNet with bidirectional GRU and an attention mechanism has been proposed [59]. This model showed superior performance compared to BERT and the standard version of XLNet. Also, a generative framework for sentiment analysis designed to learn generic sentiment knowledge among different sentiment analysis subtasks was introduced [60]. The approach was evaluated on three different backbone architectures: BART, T5, and GPT-2. The results of the study showed that BART showed superior performance compared to the other two models. In addition, a trimodal approach using GPT for text, WaveRNN for audio, and FaceNet for images was used [61]. The results of this work indicate a superior accuracy of the text modality over the other two. However, the highest performance was achieved with the combination of the three modalities. A variational autoencoder combining RoBERTa as an encoder and the BART decoder for emotion recognition in conversation was developed [62]. In this approach, the decoder was used to train the utterance reconstruction model, which was shown to improve emotion recognition performance. On the other hand, several studies using audio language models as the backbone of different architectures for SER have been published in recent years. For example, the output of Wav2 Vec2 has been processed with CNNs, achieving improved accuracy over the base model in emotion recognition [63; 64].