# Multimodal Emotion Recognition Using Visual, Vocal and Physiological Signals: A Review
## 4 Human Emotional Expression Recognition
### Single Modality
#### 4.1.1 Visual Modality
While some algorithms extract emotion from static displays (e.g., static image or apex frame in a visual expression video clip), other algorithms also exploit the temporal information embedded in the video (i.e., the use of the dynamic expression display). Recognition methods that add a time dimension (i.e., dynamic expression), especially micro-expression recognition, have been shown to recognize spontaneous subtle expressions. The exploitation of dynamic displays has been the core of facial micro-expression and dynamic emotional body expression recognition. The dynamic emotional body movements involve both changes in positions and displacements of body joints. Dynamic expression recognition depends on factor loading, the number of significant video co-loadings and categories. Facial Action Coding System (FACS) encodes facial muscle changes in response to emotion states [83,84]. The system establishes the ground truth of each action unit's exact beginning and end time. According to FACS, each facial emotional expression is identified based on a combination of action units (AUs), also referred to as facial action units. [PERSON] et al. [73] argued that using facial action units during emotion recognition instead of emotion labels can define micro-expressions more precisely, since the training process can learn based on specific facial muscle movement patterns. They further proved that this leads to higher classification accuracies. \"objective classes\" based on FACS action units have been used as categories for micro-expression recognition. Traditionally, facial macro-expression recognition has been achieved through the extraction of handcrafted features, including Scale-Invariant Feature Transform (SIFT), Histogram of Oriented Gradients (HOG), and Local Binary Patterns (LBP). A classifier subsequently utilizes these features to perform the recognition task (see Table 2 for examples). LBP, with its simplicity in computation, is commonly used because of its robustness towards illumination changes and image transformations [85]. Classical LBP characterizes the local textural information by encoding a vector of binary code into histograms. Handcrafted feature extraction methods for identifying macro-expressions are categorized as either geometric-based or appearance-based. The action units or the Euclidean distance between the action units for facial expression recognition are the basis of geometric features; one of the features most used in facial expression recognition (e.g., [86; 87; 88]). Geometric features lack information on some parts of the face. They are sensitive to noise, and the accumulated errors during tracking produce inaccurate features. Other types of features commonly used in facial expression recognition are appearance-based features. These features do not require detecting part of a face; instead, they consider the entire face structure [89; 90]. They can encode fine patterns in the facial image and are less sensitive to noise. However, a misalignment of the face reduces the recognition performance due to the extraction of features from unintended locations. Methods that combine geometric-based and appearance-based features for facial expression recognition (e.g., [91]) and deep learning-based methods (e.g., [92; 93]) overcome the shortcomings of the geometric feature-based methods and appearance-based methods. However, none of the current classical methods consider the shape deformations related to each facial emotional expression. Facial micro-expressions are characterized by small-scale facial motions that result in feature vectors with a low discriminative power. Micro-expression recognition aims to identify small-scale facial motions. The first successful recognition method for spontaneous facial micro-expressions was presented in ref. [96]. The method extended from texture features to spatial-temporal features. Researchers have mainly chosen the local binary pattern with three orthogonal planes (LBP-TOP) as the primary baseline feature extractor.