# Multimodal Emotion Recognition Using Visual, Vocal and Physiological Signals: A Review
## 4 Human Emotional Expression Recognition
### Multimodal Emotion Recognition
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline
**Ref** & **Year** & **Database** & **Modality** & **Features** & **Classifier** &
\begin{tabular}{c} **Average** \\ **Accuracy** \\ \end{tabular} & **Fusion Method** \\ \hline
[164] & 2015 & eNTERFACE & \
\begin{tabular}{c} Audio/facial \\ (visual)/text \\ combined \\ \end{tabular} & - & SVM & Acc: 87.95\% &
\begin{tabular}{c} Feature-level \\ fusion (concatenation) \\ \end{tabular} \\
[237] & 2012 & \begin{tabular}{c} RML \\ eNTERFACE \\ \end{tabular} & \begin{tabular}{c} Visual/video \\ and audio \\ \end{tabular} & - & HMM & \
\begin{tabular}{c} Acc: 80 – 85\% \\ \(70-75\%\) \\ \end{tabular} &
\begin{tabular}{c} Feature-level \\ fusion (concatenation) \\ \end{tabular} \\
[238] & 2013 & \begin{tabular}{c} RML \\ eNTERFACE \\ \end{tabular} & \begin{tabular}{c} Facial visual \\ and audio \\ \end{tabular} & - & HMM & \
\begin{tabular}{c} Acc: 70 – 80\% \\ \(70-85\%\) \\ \end{tabular} &
\begin{tabular}{c} Combination of \\ feature-level \\ (KECA) and \\ decision-level \\ \end{tabular} \\
[239] & 2020 & \begin{tabular}{c} ITMDER \\ WESAD \\ \end{tabular} & \begin{tabular}{c} ECG, EDA, \\ RESP and BVP \\ \end{tabular} & - & \begin{tabular}{c} SVM (ar), RF \\ (val) \\ \end{tabular} & \
\begin{tabular}{c} ar: 87.6\%, val: \\ 89.26\% \\ \(87.6\%\), val: \\ 92.9\% \\ \end{tabular} &
\begin{tabular}{c} Feature-level \\ fusion (concatenation) \\ \end{tabular} \\
[219] & 2011 & \begin{tabular}{c} In-house dataset \\ (induced) \\ \end{tabular} & \begin{tabular}{c} Eye gaze and \\ EEG \\ \end{tabular} & - & SVM & \
\begin{tabular}{c} (FF) ar: 66.4\%, \\ val: 58.4\% \\ (DF) ar: 76.4\%, \\ val: 68.5\% \\ \end{tabular} &
\begin{tabular}{c} Feature-level \\ (FF) \\ and decision- \\ level (DF) \\ \end{tabular} \\
[217] & 2019 & \
\begin{tabular}{c} MAHNOB-HCI \\ DEAP \\ \end{tabular} & Facial and EEG & CNN/PSD & CNN/SVM & ar: 74.17\%, val:
\begin{tabular}{c} AdaBoost for \\ decision-level \\ fusion \\ \end{tabular} \\
[240] & 2022 & \begin{tabular}{c} DEAP \\ \end{tabular} & \begin{tabular}{c} EOG and EMG \\ complexity \\ \end{tabular} & \
\begin{tabular}{c} PSD, Hjorth \\ activity and \\ complexity \\ \end{tabular} & Logit boost & 59\% &
\begin{tabular}{c} Data- \\ level fusion \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table} 
Table 13: _Cont._ \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Ref** & **Year** & **Multimodal** & **Elicitation** & **Features** & **Classifier** &
\begin{tabular}{c} **Average** \\ **Accuracy** \\ \end{tabular} & **Fusion Method** \\ \hline
[235] & 2024 & \
\begin{tabular}{c} In-house \\ dataset \\ \end{tabular} & Induced & CNN & - & Acc: 91.21\% &
\begin{tabular}{c} Decision- \\ level fusion \\ \end{tabular} & Facial/EEG \\  & & & & & & \\  & & & & & & \\  & & & & & & \\  & & & & & & \\  & & & & & \\  & & & & & & \\  & & & & & & \\  & & & & & & \\  & & & & & & \\  & & & & & & \\  & & & & & & \\ \end{tabular}
\end{table} 
Table 13: _Cont._Table 15 lists the widely used datasets for multimodal emotion recognition/analysis. To our knowledge, no publicly available datasets for multimodal emotion recognition focus solely on subtle expressions; however, some include a mix of macro and micro-expressions. Different methods can be used for dimensionality reduction [238], and multiple kernel learning algorithms are employed to analyze the data [164]. HMM was used as a classifier to understand emotion and measure statistical dependence across successive time segments [237]. Preprocessing methods have been proposed to reduce the noise in modality data. CNNs perform well in modeling the spatiotemporal information from the data of the three modalities for emotion recognition. Spectral powers of the pupil diameter data and the blinking rate of the eyes were extracted from the eye data. In contrast, the power spectral density (PSD) was extracted from different frequency bands. These features were then used as inputs for an SVM classifier with a radial basis function (RBF) kernel [219].