## 2 Emotion Recognition and Related Works
### Related Works in SER  
Speech is crucial for understanding human emotions, as it provides significant clues about emotional states. In addition to word choice and linguistic content, there are non-verbal components, such as vocal tone, pitch, and pace, that help listeners interpret the feelings and intentions behind spoken words. Prosodic features, such as intonation and tone, can be analyzed based on energy and pitch (F0), offering valuable insights. In the early stages of SER, the focus was on selecting speech characteristics or features that might reveal more about the emotional level of each utterance. Formants, pitch, duration, and energy contours, as well as statistical characteristics, are commonly accepted descriptors of emotional speech. The next step, following the feature extraction, entails applying pattern recognition techniques or machine learning algorithms to classify the diverse array of emotional expressions embedded in speech. In the study of [PERSON] et al. [19], the combination of k-Nearest Neighbours with the selection of the most prominent features outperformed other parametric and non-parametric approaches. In [20], decision trees effectively handled a variety of non-essential aspects of data. The Gaussian Mixture Model (GMM), Hidden Markov Model (HMM), and Support Vector Machines (SVMs) are also frequently used methods. The GMM utilizes Gaussian distributions to characterize clusters of points for distinct categories. The Ergodic HMM is suitable for the acoustic modelling, capturing states like phonemes, pauses, and variations in F0 during speech [21]. The SVM allows for the classification of non-linearly separable data and is often used for regression tasks [22; 23; 24]. While these traditional methods are interpretable and require less data for training, they depend on handcrafted features and have a limited capacity to learn intricate hierarchical representations automatically. Over the years, the utilization of neural networks for emotion recognition has seen continuous improvement. Multilayer Perceptron (MLP), a type of neural network consisting of multiple layers of interconnected nodes or neurons, has been employed in [25]. However, the shallow nature of MLPs lacks the capacity to effectively capture complex hierarchical patterns and handle sequential dependencies present in speech data. The emergence of deep learning signalled a notable change in the field of SER. CNNs, RNNs, and self-attention architectures, such as Transformer-based models, have demonstrated high performance in capturing intricate patterns and dependencies in speech data. In their study, [PERSON] and [PERSON] [26] employed a Bidirectional Long Short-Term Memory (BLSTM) to generate high-level feature representations of audio signals, which were represented by 32 acoustic features. Their model demonstrated an accuracy of 63.89% on four categorical emotions from the IEMOCAP dataset. [PERSON] et al. [27] and [PERSON] et al. [28] both used two-dimensional spectrogram features as input for a deep learning network. [PERSON] et al. [27] proposed the application of a Deep Retinal CNN approach, which consists of two phases. The first phase involves a data augmentation, where retina and convex lens imaging principles are applied to spectrogram features. The second phase uses a Deep CNN to extract high-level representations from the spectrograms. This method achieved an overall accuracy of 48.8% across six categorical emotions from the IEMOCAP dataset. [PERSON] et al. [28] first trained a CNN on the extracted spectrogram data, experimenting with various network topologies. The subsequent addition of temporal modelling using the BLSTM layer further improved overall accuracy, achieving 68.8%, compared to 66.1% with CNN layers alone, when classifying four emotions from the IEMOCAP dataset. Integrating SER into multi-modal systems enhances the assessment of emotional states in complex environments, such as industrial, aviation, or healthcare scenarios.