# Multimodal Emotion Recognition Using Visual, Vocal and Physiological Signals: A Review
## 4 Human Emotional Expression Recognition
### Single Modality
#### 4.1.3 Physiological Modality  
Emotion recognition using physiological data is one of the applications of brain-computer interfaces (BCIs), in which computers decode people's emotional states from their brain signals. It has wide applications in healthcare. Motor expressions (physical expressions) of emotion, such as visual (e.g., facial expression and body gesture) and verbal expressions, can be deceptive to a certain degree. For example, people might reign an emotion they do not feel or express an unfelt emotion simply because it is socially desirable. Physiological data, such as electroencephalographs (EEGs), body temperature, electrocardiograms (ECGs) and electromyograms (EMGs), are less imitable, and their analyses are not easily affected by human subjectivity. Therefore, they are more reliable indicators of genuine emotional states [189] and are always a predictable indicator of the mental state of the person involved. EEG physiological signals are the most commonly used physiological data in emotion research. They can indicate people's immediate responses to emotional stimuli with a high temporal resolution and cannot be misled by human expressions. EEG signals are distributed across multiple channels, and using more channels can provide more precise data for emotion recognition. However, the disadvantage/ inadequacy is that it is more expensive for data collection and more intrusive for the participants/holders/subjects. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Ref** & **Year** & **Database** & **Features** & **Classifier** & **Average Accuracy** \\ \hline \multirow{183}{*}{[183]} & \multirow{5}{*}{2019} & \multirow{5}{*}{EMO-DB} & Feature fusion by DBNs: prosody features (fundamental frequency, power), voice quality features (the first, second and third formants with their bandwidths), spectral features (WPCC and W-WPCC) AM-FM modulation features (MS, MFF), cepstral features (ECC, EFCC, SMFCC) from THT FCN: fully connected network; Acc: accuracy; WAAcc: weighted average accuracy. & \\ \cline{1-1} \cline{3-6}  & & & Feature fusion by DBNs: prosody features (fundamental frequency, power), voice quality features (the first, second and third formants with their bandwidths), spectral features (WPCC and W-WPCC) AM-FM modulation features (MS, MFF), cepstral features (ECC, EFCC, SMFCC) from THT FCN: fully connected network; Acc: accuracy; WAAcc: weighted average accuracy. & \\ \cline{1-1} \cline{3-6}  & & & & & \\ \cline{1-1} \cline{3-6}  & & & & & \\ \cline{1-1} \cline{3-6}  & & & & & \\ \cline{1-1} \cline{3-6}  & & & & & \\ \cline{1-1} \cline{3-6}  & & & & & \\ \cline{1-1} \cline{3-6}  & & & & & \\ \cline{1-1} \cline{3-6}  & & & & & \\ \cline{1-1} \cline{3-6}  & & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 10: _Cont._EEG-based emotion recognition using handcrafted features exploits linear or nonlinear dynamic features extracted from EEG signals. Linear features include power spectrum and wavelet features. In contrast, nonlinear features include different types of entropies (approximate entropy, sample entropy), Hurst exponents, fractal dimensions (FDs), correlation dimensions (CDs), the largest Lyapunov exponent (LLE) and higher-order statistics [190; 191]. Nonlinear characteristics can reveal emotions better than linear characteristics. The empirical mode decomposition (EMD) method was used to decompose EEG signals and derive the sample entropy of the derived intrinsic mode functions (IMFs) [192]. In addition, some studies have employed a divide-and-conquer strategy for feature extraction. For example, EEG signals were divided into segments according to the time windows in ref. [193]. Support vector machines (SVMs) are the classification method commonly used for EEG-based emotion recognition. Apart from SVMs, other classical classification methods that have been used for EEG-based emotion recognition are K-nearest neighbor (KNN) and linear discriminant analysis (LDA) [194].