# Review of Automatic Estimation of Emotions in Speech
## 7 Neural (Machine Learning) Techniques for Emotion Classification of Speech
### Supervised and Self-Supervised Methods  
Whether to use specific relevant data training for estimation tasks is an important factor in performance. Recognition models that rely entirely on supervised training often have lower accuracy, as access is limited only to existing data. Current public emotional datasets are insufficient to train a robust supervised learning model, as their diversity of speakers and conditions is too limited. In the related field of ASR, many classification approaches use supervised training, where dynamics of spectra can be estimated from large training databases by examining short sequences of speech, e.g., context-dependent phones or senones (e.g., three phones in a row, or triphones). However, as emotional speech data as sparse, SER often uses general methods that require fewer labeled data. One can generalize models by use of data augmentation, which deforms existing data for training in a random fashion, using reverberation and noise, or other distortion, which is artificially added [109; 116; 117]. However, most speech augmentation techniques are those derived for ASR, which may not be appropriate for SER, e.g., altering speaking rate clearly affects emotion. In addition, little research has been performed on SER using noisy speech [118]. A related domain adaptation approach to deal with lack of suitable SER data is generative adversarial networks (GANs) [117; 119; 120]. This technique contrasts actual (true) and synthetic (false) samples of data using two networks: generator and discriminator. The generator formulates data in a low-dimension latent space; then, the discriminator learns to discriminate between \"real\" training data and \"false\" generator outputs. Generators maximize the error rate of the discriminator, while discriminators reduce their error rate. GANs can represent phonetically or phonologically meaningful information. Diffusion models can provide an alternative to GANs [121]. Recently, large language models (LLMs), e.g., ChatGPT and Claude, have been applied to SER [122]. LLMs are pre-trained on vast text corpora, instead of speech, and have found vast applications. Pre-trained models use massive amounts of unlabeled data to learn general speech representations, which are then fine-tuned for specific tasks, which reducing needs for extensive labeled datasets. For many machine learning tasks (including SER), a common way to address current database limitations is to use SSL models. They may encode a diversity of phonetic, prosodic, and speaker information [115; 123]. The popular wav2 vec2 model [124], WavLM [125], and HuBERT [27] are self-supervised, i.e., trained to determine representations of speech automatically from data that are unlabeled. Which features these models abstract from data depend greatly on their choice of loss functions and their application as encoder/decoder. They compress input data to (a reduced set of) latent representations, and a measure of success is the quality of their reconstructed output data. Their action is thus quite different from feature-based coders (e.g., LPC). Such SSL models suffer from the general lack of interpretability of most neural models, which leads to great difficulty in disentangling emotional cues from speaker- or context-specific information in speech. These all remain very active areas of research. Wav2 vec2 has a CNN encoder, randomly masks (i.e., omits) portions of input speech in a latent space, and then tries recovering via a contrastive loss criterion between the learnable quantizations and reconstructed versions. A quantization module converts latent speech representations to discrete copies, which are then used as targets. The entire model trains to solve a contrastive proxy task; this needs identification of the true quantized latent speech representations for a masked time unit within a set of distractors. The HuBERT method employs a similar approach: a 1-D CNN, and then a Transformer encoder.