# Small Language Models for Speech Emotion Recognition in Text and Audio Modalities
## 2 Related Work
### Architectures of SLMs
#### 2.1.1 Encoder-Only Models  
These models generate contextual representations for each token and are capable of understanding the interconnections between all input tokens. They are particularly well suited for natural language understanding tasks, where language features are extracted for subsequent applications such as text classification. However, several experiments show that the accuracy of this type of model is comparable to other models of similar size with encoder-decoder or decoder-only architectures in various text classification tasks such as financial sentiment analysis [24], fake news detection [25], or empathy detection in text [26].  
In the text modality, BERT [11] is one of the first encoder-only models, and probably the most representative. BERT is a bidirectional language model designed to take into account the context of the surrounding tokens, both upstream and downstream of a given token. BERT's architecture consists of multiple encoders arranged sequentially in a feed-forward structure, where the output of each encoder is used as the input of the next encoder. As a result of the great popularity of BERT since its release, several models based on its architecture have been introduced that have demonstrated better performance than the original model in several NLP tasks. The main improvements over the base model are briefly described below:  
* RoBERTa [27]: Uses an optimized BERT training procedure by adjusting the hyperparameters, eliminating the next sentence prediction task, and using mini-batches and higher learning rates during training.
* ERNIE 2.0 [28]: Improves language understanding by integrating external knowledge sources during training and using knowledge masking to improve semantic understanding.
* ALBERT v2 [29]: Splits the embedding matrix into two smaller matrices and uses repeated layers split into groups to reduce the number of trainable parameters and increase training speed.
* DeBERTaV3 [30]: Uses disentangled attention, an improved mask decoder, and ELECTA-style pre-training with gradient disentangled embedding sharing for improved performance over BERT and RoBERTa.  
On the other hand, the BERT architecture has also served as a basis for the creation of audio models that apply similar principles of learning contextual representations to acoustic data. For example, Wav2 Vec2 masks the speech input in latent space using a convolutional neural network (CNN) and solves a contrastive task defined over a quantization of the latent representations jointly learned in a BERT-like transformer. Similarly, HuBERT [31] uses the same blocks as Wav2 Vec2, but introduces an offline clustering operation to provide aligned target labels for a BERT-like prediction. Similarly, WavLM [32] is based on the HuBERT framework with an emphasis on modeling spoken content and preserving speaker identity. This model extends the transformer architecture by incorporating a gated relative position bias, which improves its performance in recognition tasks. Additionally, squeezed and efficient Wav2 Vec (SEW) [33] has been proposed as a lightweight alternative to this type of architecture, which modifies the CCN feature extractor and squeezes the input to the transformer for improved efficiency.