# Multimodal Emotion Recognition Using Visual, Vocal and Physiological Signals: A Review
## 4 Human Emotional Expression Recognition
### Single Modality
#### 4.1.1 Visual Modality
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Ref** & **Year** & **Database** & **Features/Classifier** & **Best Performance** \\ \hline
[96] & 2011 & SMIC & LBP-TOP + TIM/MKL & Acc: 71.4\% \\
[100] & 2013 & SMIC-HS & LBP-TOP + TIM/SVM & Acc: 48.78\% \\
[111] & 2014 & SMIC & STM/ AdaBoost & F1: 0.4731, Acc: 44.34\% \\  & & CASME II & & F1: 0.3337, Acc: 43.78\% \\
[97] & 2015 & SMIC & LBP-SIP/SVM & Acc: 62.80\% \\  & & CASME II & & Acc: 66.40\% \\
[99] & 2015 & SMIC & LBP-MOP/SVM & Acc: 50.61\% \\  & & CASME II & & Acc: 45.75\% \\
[98] & 2015 & SMIC & STLBP-IP & Acc: 57.93\% \\  & & CASME II & & Acc: 59.51\% \\
[112] & 2015 & CASME II & Monogenic Riesz Wav./SVM & F1: 0.4307 \\
[104] & 2015 & CASME II & LBP-TOP with adapt. magnification/SVM & Acc: 69.63\% \\
[106] & 2015 & SMIC & OSW-LBP-TOP/SVM & Acc: 57.54\% \\  & & CASME II & & Acc: 66.40\% \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Facial micro-expression recognition using handcrafted features. Handcrafted feature extraction methods demand highly specialized knowledge of data, and the extraction can be highly computationally involved. The existing handcrafted features have limitations due to their reliance on prior knowledge and heuristics. Deep learning models have been used to extract emotion features and distinguish emotional states. Unlike the handcrafted feature extraction, deep learning models require less specialized data knowledge and minimal reliance on prior knowledge. Handcrafted feature extraction requires feature detection during the preprocessing stage, which affects the overall feature extraction and classification. The handcrafted features also fail to perform well on a dataset with more image variation and partial faces. Spontaneous micro-expression recognition using handcrafted features has a sophisticated design and limitations regarding the recognition rate for practical applications [114]. Deep learning overcomes the imbalanced dataset problem by effectively learning features from imbalanced data using data augmentation. Data augmentation is also used to avoid overfitting. Deep learning also has the property of translation invariance. With the success of the convolutional neural network (CNN) in visual object recognition and detection, visual emotion recognition methods have relied mainly on CNN. Although deep learning is still developing, it has already achieved good emotion recognition rates. However, further exploration of various deep learning frameworks is necessary to enhance these rates and improve the recognition of subtle movements in facial expressions. Table 4 provides an overview of different deep learning approaches used specifically for macro-expression extraction, illustrating the advancements and options available in this area. The first deep learning-based method for facial micro-expression analysis was proposed in the year 2016. The basic micro-expression recognition algorithms use the standard 2D CNNs (see Figure 3) commonly used for static display on the apex frame of the original video clips and extract spatial information (e.g., use of ResNet [115]). However, these algorithms do not encode the muscle motions. Other methods first encode the muscle motion (e.g., subtle facial emotion) from the original videos. The motion-encoded data, such as the optical flow data, are directly supplied to the off-the-shelf 2D CNN (e.g., ResNet and VGG). This 2D CNN accepts 2D input RGB color images using transfer learning (e.g., use of ResNet [116]) or via retraining (e.g., use of ResNet [117]) or modifying the off-the-shelf/backbone networks (e.g., modification of the ResNet for the custom 2D CNN