# Review of Automatic Estimation of Emotions in Speech
## 6 Datasets for SER  
Many SER systems need training datasets that are labeled and also use ASR (to estimate the text of an emotional utterance), while recent neural approaches may not need these. The diverse set of applications and ways to evaluate SER leads to use of many acoustic datasets. To compare SER processes, typical datasets are used. Because neural network classifiers are based fully on training examples, the distribution of such data is crucial. For scientific control, most SER databases have speech by actors simulating different emotions, including some official \"challenges\" [37; 69]. Some \"in-the-wild\" databases, often from YouTube, use unprompted emotional speech [70]. Most databases ignore inter-rater ambiguity [71], and assign distinct labels to speech (Table 1). Perhaps the largest English SER corpus, MSP-PODCAST has speech from 62,140 speaking turns in podcasts. In comparison to SER, note that many cloud-based ASR models use thousands of hours of training speech; as a result, ASR, in general, has higher accuracy than SER. Popular for SER experiments, the IEMOCAP database has five sessions, each with two actor (man and woman) speakers: 10,039 total turns with nine emotion labels (anger, happiness, excitement, sadness, frustration, fear, surprise, other, neutral). It has both scripted and improvised conversations, with imbalanced class distribution [16]. MSP-IMPROV has 8438 improvised speaking turns with four emotions (happy, sadness, anger, neutral). Table 1 lists other databases with actors simulating various emotions in text-independent, text-dependent, and improvised scenarios. Many SER systems train and then test using the same database. Instead, to increase robustness, training and testing on different sets is called cross-corpus [86; 87]. In general, SER datasets are much smaller than those used for ASR, as general ASR has access to huge numbers of normal utterances, whereas SER sets are often limited to actor speech. \begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Name** & **Language** & **Categories** & **Comments** & **Citation** \\ \hline MSP-PODCAST & English & 8 & 100 h & [72] \\ \hline IEMOCAP & English & 9 & 12 h & [16; 73] \\ \hline MSP-IMPROV & English & 4 & 9 h & [74] \\ \hline CREMA-D & English & 6 & 7442 clips & [75] \\ \hline EMOVO & Italian & 6 & 6 actors, 14 sentences & [76] \\ \hline EmoDB & German & 7 & 10 actors, 10 sentences & [77] \\ \hline eNTERFACE & English & 6 & 42 subjects & [78] \\ \hline RAVDESS & English & 7 & 24 actors & [79] \\ \hline Vera am Mittag & German & Large range & 12 h; spontaneous & [80] \\ \hline MASC & Mandarin & 5 & 68 actors & [81] \\ \hline — & Japanese/English & 5 & 90 sentences & [82] \\ \hline FAU Aibo Emotion Corpus & German & 12 & spontaneous from children; 9 h & [83] \\ \hline EmoMatchSpanishDB & Spanish & 6 & 50 subjects & [84] \\ \hline RUSLANA & Russian & 6 & 61 subjects & [85] \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Common datasets for SER. As a result, many SER systems instead start from a general ASR platform, and fine tune with SER datasets [88]. In particular, SER has seen much use of HuBERT and Wav2 Vec2 models [89; 90], i.e., use of SSL speech models. An alternative is Whisper, a Transformer model trained with weakly supervised learning [91]. In the available databases of emotional speech, most have an insufficient number of utterances and speakers to allow effective single-corpus training of large neural networks. Thus, cross- and multi-corpus SER is common. However, databases differ greatly in recording settings (e.g., acted, forced, or natural) and language. Strategies such as domain adaptation or adapter transfer learning can improve SER performance [53; 92]. Best practices for evaluation include the importance of cross-corpus testing, robustness to spontaneous versus acted emotion, and generalization across speaker identities and recording conditions.