# Multimodal Emotion Recognition Using Visual, Vocal and Physiological Signals: A Review
## 4 Human Emotional Expression Recognition
### Single Modality
#### 4.1.1 Visual Modality
\begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Ref** & **Year** & **Database** & **Features/Classifier** & **Best Performance** \\ \hline
[107] & 2016 & \
\begin{tabular}{l} SMIC \\ CASME II \\ \end{tabular} & OSF + OSW/SVM & Acc: 52.44\% \\  & 2016 &
\begin{tabular}{l} CASME \\ CASME II \\ \end{tabular} & LBP-TOP/RK-SVD & Acc: 63.16\% \\
[101] & 2017 &
\begin{tabular}{l} CASME II \\ \end{tabular} & LBP-TOP with EVM/SVM & Acc: 69.04\% \\
[101] & 2017 &
\begin{tabular}{l} CASME II \\ \end{tabular} & LBP-TOP with EVM/SVM & Acc: 63.25\% \\
[21] & 2017 &
\begin{tabular}{l} SMIC \\ CASME II \\ \end{tabular} & LBP-TOP with a sparse sampling/SVM & Acc: 75.30\% \\
[109] & 2017 &
\begin{tabular}{l} SMIC \\ CASME II \\ \end{tabular} & LBP-TOP with a sparse sampling/SVM & Acc: 58.00\% \\
[108] & 2017 & \begin{tabular}{l} SMIC-HS \\ CASME \\ \end{tabular} & FDM/SVM & F1: 0.2401, Acc: 42.02\% \\  & 2018 & \begin{tabular}{l} CASME II \\ \end{tabular} & & F1: 0.2972, Acc: 41.96\% \\  & 2018 & \
\begin{tabular}{l} CASME II \\ \end{tabular} & Bi-WOOF/SVM & F1: 0.61 \\  & 2018 &
\begin{tabular}{l} SMIC \\ \end{tabular} & & F1: 0.62 \\
[102] & 2018 &
\begin{tabular}{l} SMIC-HS \\ CASME II \\ \end{tabular} & HIGO + Magnification/SVM & Acc: 75.00\% \\
[113] & 2019 & \
\begin{tabular}{l} SMIC \\ CASME \\ \end{tabular} & DISTLBP-RIP/SVM & Acc: 64.33\% \\  & 2018 &
\begin{tabular}{l} CASME II \\ \end{tabular} & & Acc: 64.78\% \\ \hline \hline \end{tabular}
\end{table} 
Table 3: _Cont._presented in [116]). If the networks are retrained, they can be initialized using large facial datasets such as ImageNet2012 (e.g., in [117]). Optical flow provides robust features to the diversity of facial textures. Consequently, to ensure a reduced computational complexity in terms of time, the optical flows are extracted between the onset and apex frames of micro-expression video clips--it is between these two moments that the motion is most likely to be the strongest. Commonly, one optical field (either vertical, horizontal, or magnitude) or a pair of horizontal and vertical fields are used as inputs to the 2D CNN. For facial micro-expression, the vertical optical field that encodes the vertical motion has been proven to offer better results than the horizontal optical field. Furthermore, to reduce the computational complexity over time, the vertical optical field alone was used with a customer 2D CNN shallow convolutional network [116] for fast training. [PERSON] et al. [114] proposed a two-stream 3D CNN model (see Figure 4). In contrast to a standard one-stream 3D CNN model (see Figure 5) that also captures the temporal dimension of facial expressions, the two-stream 3D CNN model used two datasets (CASME and CASME II) to increase the amount of data available for meaningful training of a deep learning network. The two datasets have different frame rates, each dataset was fed into a stream. The model uses a shallow network to avoid overfitting. The input of each stream receives the optical flow data to enrich the input data with motion information and to ensure that the shallow network acquires high-level features. After feature extraction using the 3D CNN, classification was implemented using a support vector machine (SVM) on four categories: negative, positive, surprise and others [114]. Using expression states (i.e., stages of a single expression) instead of an expression as the input increases input data and improves the expression class separability of the learned micro-expression features [129]. The method proposed by [PERSON] et al. [129] uses a 2D CNN to learn the spatio-temporal feature representation in the first stage and consequent LSTM in the second stage (see Figure 6). The spatial model is learned, and the expression-state features learned from the