# Multimodal Emotion Recognition Using Visual, Vocal and Physiological Signals: A Review
## 4 Human Emotional Expression Recognition
### Single Modality
#### 4.1.1 Visual Modality
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Ref** & **Database** & **Features/Classifier** & **Best Performance** \\ \hline \multirow{4}{*}{[117]} & \multicolumn{3}{c}{2D CNN} \\ \hline
[117] & \begin{tabular}{c} SMIC, CASME II, \\ and SAMM \\ \end{tabular} & EMR & \begin{tabular}{c} UFI:0.7885 \\ and UAR:0.7824 \\ \end{tabular} \\  & \
\begin{tabular}{c} SMIC, CASME II, \\ and SAMM \\ \end{tabular} & Dual-Inception & UFI:0.7322 and UAR:0.7278 \\  &
\begin{tabular}{c} SMIC \\ \end{tabular} & ResNet, Micro-Attention & Acc:49.4\% \\
[133] & \
\begin{tabular}{c} CASME II \\ SAMM \\ \end{tabular} & ResNet, Micro-Attention & Acc:65.9\% \\  &
\begin{tabular}{c} SMIC \\ \end{tabular} & & Acc: 48.5\% \\
[134] & \begin{tabular}{c} CASME II \\ SAMM \\ \end{tabular} & OFF-ApexNet & Acc:67.6\% \\  & \
\begin{tabular}{c} OFF-ApexNet \\ \end{tabular} & Acc:88.28\% \\  &
\begin{tabular}{c} RCNN \\ \end{tabular} & Acc:69.18\% \\ \hline \multirow{4}{*}{[135]} & \multicolumn{3}{c}{RCNN} \\ \hline
[135] & \begin{tabular}{c} SMIC \\ CASME \\ CASME II \\ \end{tabular} & MER-RCNN & Acc:57.1\% \\  & \
\begin{tabular}{c} MER-RCNN \\ \end{tabular} & Acc:63.2\% \\  &
\begin{tabular}{c} CASHE \\ \end{tabular} & Acc:65.8\% \\ \hline \multirow{4}{*}{[137]} & \multicolumn{3}{c}{3D-CNN} \\ \hline
[114] & \begin{tabular}{c} CASME I/II \\ \end{tabular} & DTSCNN & Acc: 66.67\% \\  & \
\begin{tabular}{c} SMIC, CASME II, \\ and SAMM \\ \end{tabular} & STSTNet &
\begin{tabular}{c} UFI:0.7353 \\ and UAR:0.7605 \\ Acc:54.49\% \\ Acc:54.49\% \\ Acc:54.44.28\% \\ Acc:59.11\% \\ \end{tabular} \\ \hline \multirow{4}{*}{[138]} & \multicolumn{3}{c}{Combined 2D CNN and 3D CNN} \\ \hline
[138] & \begin{tabular}{c} SMIC \\ CASME II \\ SAMM + CASME II \\ \end{tabular} & \
\begin{tabular}{c} TSNN-IF \\ TSNN-LF \\ \end{tabular} &
\begin{tabular}{c} F1:0.6631, UAR:0.6566, \\ WAR:0.7547 \\ F1: 0.6921, UAR:0.6833, \\ WAR:0.7632 \\ \end{tabular} \\ \hline \multirow{4}{*}{[139]} & \multicolumn{3}{c}{Combined 2D-CNN and LSTM/GRU/RNN} \\ \hline
[129] & \
\begin{tabular}{c} SMIC \\ CASME II \\ SAMM \\ \end{tabular} & Apex–time network & UFI:0.497 and UAR:0.489 \\  &
\begin{tabular}{c} UFI:0.523\% and UAR:0.501 \\ UF1:0.429 and UAR:0.427 \\ \end{tabular} \\ \hline \hline \multirow{4}{*}{[140]} & \multicolumn{3}{c}{2D-CNN and then LSTM/GRU/RNN} \\ \hline
[141] & \
\begin{tabular}{c} CASME II \\ SAMM \\ \end{tabular} & CNN-LSTM & Acc: 60.98\% \\  &
\begin{tabular}{c} F1 score:0.5 \\ F1 score:0.409 \\ \end{tabular} \\ \hline \hline \multirow{4}{*}{[140]} & \multicolumn{3}{c}{Spatial contextual} \\ \hline
[140] & \
\begin{tabular}{c} SMIC, CASME II, \\ and SAMM \\ \end{tabular} & CapsuleNet & UFI:0.6520\% and UAR:0.6506\% \\  &
\begin{tabular}{c} CASME II \\ SAMM \\ \end{tabular} & AU-GACN & Acc:49.2\% \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Microexpression recognition for spontaneous datasets using deep learning features. can be applied to a sequence of frames to increase and balance data classes [114; 129]. Additionally, a temporal interpolation model can be employed to fit the sample sequence into a recurrent model that expects a fixed temporal length [130]. The performance of new automatic facial microexpression recognition methods is commonly compared against the baseline LBP-TOP method (presented results or reproduced LBP-TOP) [130]. The recognition performance measures are the F1 score, weighted average recall (WAR), or accuracy. The unweighted average recall (UAR) is beneficial for balanced accuracy (averaging the accuracy scores of each individual class without consideration of class size), and the macro-averaged F1 score provides a balanced metric when considering highly imbalanced data [111]. The micro-expression recognition methods are evaluated using spontaneous micro-expressions databases. Publicly available datasets for visual features are used as the benchmark databases for visual emotion recognition. With the advent of spontaneous micro-expressions databases, many methods of microexpression recognition have been proposed.