# Multimodal Emotion Recognition Using Visual, Vocal and Physiological Signals: A Review
## 4 Human Emotional Expression Recognition
### Single Modality
#### 4.1.1 Visual Modality
The LBP-TOP is a spatiotemporal extension of the classic local binary pattern (LBP) descriptor. LBP-TOP extracts the histograms from the three planes, XY, XT and YT, and concatenates them into a single-feature histogram. [PERSON] et al. [97] proposed a method to reduce the redundancies in the LBP-TOP by utilizing only six intersection points in the 3D plane to construct the feature descriptor. An integral projection technique was \begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Ref** & **Year** & **Database** & **Features/Classifier** & **Best Performance** \\ \hline
[90] & 2011 & BU3 DFE & LGBP+LBP/SVM & Acc: 71.10\% \\  & & CK & & Acc: 96.60\% \\
[89] & 2013 & JAFFE & LDN/SVM & Acc: 91.60\% \\  & & MMI & & Acc: 95.80\% \\  & & CMU-PIE & & Acc: 94.40\% \\
[86] & 2013 & CK+ & Geometric/SVM & Acc: 97.35\% \\
[91] & 2014 & CK+ & AAM/NN & Acc: 85.73\% \\
[94] & 2020 & CK+ & AUs/RF & Acc: 75.61\% \\
[87] & 2021 & in-house & ICAT/RF & Acc: 98.17\% \\
[88] & 2021 & CK+ & AUs/Ensemble & Acc: 72.55\% \\  & & MUG & & Acc: 88.37\% \\
[95] & 2023 & MUG & AUs and geometric features/ANFIS & Acc: 99.6\% \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Facial macro-expression recognition using handcrafted features. also proposed to preserve the property of micro-expressions and enhance the discrimination of micro-expressions [98]. Spatio-temporal LBP with integral projection (STLBP-IP) applies the LBP operator to horizontal and vertical projections based on difference images. The method is shape-preserving and robust against white noise and image transformations. Examples of other handcrafted features for micro-expression recognition are LBP-MOP [99] (with an improved integral projection). The LBP-TOP, with preprocessing using the temporal interpolation model (TIM) [100], uniformly samples a fixed number of image frames from the constructed data manifold. LBP-TOP (sparsity-promoting dynamic mode decomposition [DMDSP]) acts to select only the significant temporal dynamics when synthesizing a dynamically condensed sequence [21]. LBP-TOP (EVM + HIGO) magnifies the video (Eulerian video magnification) in an attempt to accentuate the subtle changes before feature extraction [101; 102]. [PERSON] et al. [103] used the dynamic features to recognize macro-expressions; specifically, LBP-TOP. The case of adaptive motion magnification combined with the LBP-TOP method was proposed in ref. [104]. Adaptive motion magnification techniques are used to emphasize micro-expression motion. Although motion magnification improves expression class separability, magnification parameters, such as frequency bands and magnification factors, are sensitive enough to achieve a good performance. Motion information is used to portray the subtle changes exhibited by micro-expressions. For example, the extraction of a derivative of optical flow called an optical strain was used originally for micro-expression spotting [105] but later adopted as a feature descriptor for micro-expression recognition [106; 107]. Leveraging the discriminativeness of optical flow, other approaches that exploit the optical flow have been proposed, such as bi-weighted oriented optical flow (Bi-WOOF) and FDM [108; 109]. LBP-TOP-HOOF [110], a handcrafted approach that uses K-SVD, uses optical strain and aims to preserve the temporal dimension as its dynamics [107]. LBP-TOP + STM [111] also uses TIM, the same as in ref. [100]. Table 3 presents a comprehensive overview of micro-expression recognition methods that utilize handcrafted features.