# Multimodal Emotion Recognition Using Visual, Vocal and Physiological Signals: A Review
## 4 Human Emotional Expression Recognition
### Single Modality
#### 4.1.2 Speech Modality  
Speech/voice emotion recognition is preferred for applications in e-learning and telemarketing, call centers, phone banking and many other applications [162]. Psychological studies of emotion prove that vocal parameters, especially pitch, intensity, speaking rate and voice quality, play an essential role in emotion recognition and sentiment analysis [163]. Vocal emotion cues, which play a crucial role in emotion recognition, include prosodic variables/features (such as pitch, pause duration (pause), tone of voice, intonation, rate of speech, and voice probabilities) and energy-related features. Other emotion recognition features used by some researchers are the zero-crossing rate, signal energy, entropy of energy, spectral centroid, spectral spread, spectral entropy, spectral flux, spectral roll-off, vocal formants (especially their frequencies), Teager energy operator (TEO)-based features (or Teager-Kaiser energy operator), beat histogram (beat sum, strongest beat), perceptual linear predictive coefficients (PLPs) (such as linear prediction cepstral coefficients (LPCC)), log frequency power coefficients (LFPC), Mel-frequency cepstral coefficients (MFCCs) and bark-frequency cepstral coefficients (BFCCs) [164; 165]. Some features, such as prosodic features, are insignificant for involuntary emotional expression. Various works have been  
Figure 4: Multistream convolutional neural network-based methods. Figure 5: Three-dimensional convolutional neural network-based methods. Figure 6: Recurrent convolutional network-based methods. carried out based on the types of features needed for better analysis [166; 167]. Speech data are difficult to find, and datasets are made of a few samples. Public databases for speech data are solely acted data with no prosody control. Table 8 details the public datasets commonly used for speech emotion recognition. Recent speech emotion recognition methods are detailed in Tables 9 and 10. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Ref** & **Year** & **Database** & **Features** & **Classifier** & **Average Accuracy** \\ \hline
[174] & 2008 & Crema-D & CNN & CNN & 60–70\% \\
[175] & 2018 & AESDD & LSTM & LSTM & F1 score: 0.6523 \\
[176] & 2021 & LSSED & GRU & GRU & Acc:53.45 \\
[177] & 2020 & ESD & CNN & SVM & Acc:63.78 \\
[178] & 2022 & AESDD & CNN & CNN & 77\% \\  & & TESS & & WAAcc: 99.46\% \\  & & EMO-DB & Ensemble of & & WAAcc: 95.42\% \\
[179] & 2023 & RAVDESS & 1D CNN, & & WAAcc: 95.62\% \\  & & SAVEE & LSTM & FCN & WAAcc: 93.22\% \\  & & CREMA-D & & & WAAcc: 90.47\% \\
[180] & 2024 & IEMOCAP & & & 66.2\% \\  & & NoiseA92 & Wave-U-Net & Wave-U-Net & 62.4\% (at 0 dB SNR) \\  & & EMO-DB & & & 99.86\% \\  & & RAVDESS & Ensemble of & & 96.3\% \\
[181] & 2024 & SAVEE & Transformer, & & 96.5\% \\  & & IEMOCAP & CNN & & 85.3\% \\  & & SHEIE & and LSTM & & 83\% \\  & & RAVDESS & & & Acc: 87.08 \\
[182] & 2023 & EMOVO & Statistical mean of MFCCs & 3-1D CNN & Acc:83.90 (speaker-dependent) \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Speech emotion recognition methods using deep learning features. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Dataset** & **No. of** & **No. of** & **Persons** & **Comments** & **Difficulty** \\ \hline EMO- & & & & & \\ DB [168] & 7 & 535 & 10 & Acted & Medium + text \\ EVD [169] & 5 & 9750 & 5 & Acted & Medium \\ CHAD [170] & 7 & 6228 & 42 & Acted & Medium \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Speech emotion expression datasets.