# Review of Automatic Estimation of Emotions in Speech
###### Abstract  
Identification of emotions exhibited in utterances is useful for many applications, e.g., assisting with handling telephone calls or psychological diagnoses. This paper reviews methods to identify emotions from speech signals. We examine the information in speech that helps to estimate emotion, from points of view involving both production and perception. As machine approaches to recognize emotion in speech often have much in common with other speech tasks, such as automatic speaker verification and speech recognition, we compare such processes. Many methods of emotion recognition have been found in research on pattern recognition in other areas, e.g., image and text recognition, especially in recent methods for machine learning. We show that speech is very different compared to most other signals that can be  
and in complex ways. In speech, the effects of emotion vary temporally, are expressed and perceived over multiple modalities, are often inherently ambiguous, and may vary within a dialog [12]. As evidence of the difficulty of SER, both humans and machines have SER accuracy rates that are below performance levels in most other speech applications. Differences in the ways people express emotion may require \"personalized\" SER, i.e., techniques adapted to individuals [13].  
We define emotional concepts in Section 2, first noting various theories and categories for emotion, then describing physical measures of emotion in speech; thereafter, we have sub-sections on the effects of emotion on two major aspects of speech: intonation and voice quality. Vocal tract (VT) shape is a prime determiner of phonetic information in speech, and the focus of automatic speech recognition (ASR), but is likely not a major component of emotion; thus we focus on intonation and voice quality, which can be heavily influenced by emotion. We examine how human listeners perceive emotion from speech, as SER can emulate this behavior.  
As VT shape affects speech spectra, which are heavily used in SER, we examine the major methods for speech analysis in Section 3. This longer section describes how computers can process speech to extract acoustic features that can be useful for speech applications. We start with simple energy and spectra, and examine the traditional methods found in most of speech processing: linear predictive coding (LPC), mel frequency cepstral coefficients (MFCCs), filter banks, and time windows. We also delve more into the components of intonation analysis: durations, pitch, and amplitude.  
Section 4 examines various mathematical models that have been applied to SER: hidden Markov models (HMMs), embeddings, Support Vector Machines (SVMs), and Low-Level-Descriptors (LLDs). Section 5 notes the great variability among SER tasks, and how to evaluate SER performance. Section 6 describes many of the datasets that are commonly used for SER.  
The extensive Section 7 describes the operation of artificial neural network (ANN) techniques, which are now in standard use for most SER. The review here of ANNs is not intended to be comprehensive, but to introduce basic ideas about ANNs and their use for SER. The first sub-section describes the fundamental units and architecture of ANNs. Then, we examine the most common variants: convolutional neural networks (CNNs), recurrent neural networks (RNNs), attention, and Transformer. Examples of state-of-the-art SER are given. We end this section with the important aspect of supervised and unsupervised training.  
A discussion ensues in Section 8, and we conclude the paper with suggestions to improve SER, focusing on the better use of acoustic features.