# Multimodal Emotion Recognition Using Visual, Vocal and Physiological Signals: A Review
## 4 Human Emotional Expression Recognition
### Single Modality  
#### 4.1.1 Visual Modality  
Visual emotion modalities such as facial expression, body posture, hand or head gestures and body movement play a significant role and are essential for maintaining human relationships. During face-to-face human interaction, facial expressions significantly affect the message received by the listener [74]. Poor recognition of visual emotion expression in humans is usually associated with an inability to interact effectively in social situations [75]. The visual emotion cues that clearly show a person's positive or negativeemotions are gestures, body movements, changes in body posture, and movement of a part of the body that includes the head, limbs or hands (e.g., scratching of the head). Other visual emotion cues, which form part of a facial expression but can be considered independently to assess the emotion conveyed, are changes in mouth length, pupil size or eye movements. In the case of face-to-face communication, spatial distance from the addressee can also be an emotional cue. In human-to-human interactions, facial expression contributes significantly to emotion perception, while other visual cues, such as posture, have a more subtle effect on emotion perception. Visual emotion recognition uses video footage or images of visual expressions and exploits computer vision and machine learning algorithms to achieve automated expression recognition. Computer vision approaches are used to extract features from facial data and classify these data as one of the categories of emotional states. The performance of visual emotion recognition depends on both feature extraction and classification. Visual data, such as face data, are less sensitive to noise. Though there have been substantial advances in visual emotion recognition, the current methods do not achieve sufficient performance, given the high intraclass variation. Facial emotional expression is commonly divided into macro-expressions and micro-expressions. Macro-expressions are more applicable to entertainment applications. The recognition of human emotion based on macro-expressions may be misleading in other applications, given that some people may hide their true emotions. Micro-expressions apply to healthcare, marketing, education, security and other applications. Facial micro-expressions are spontaneous and involve brief facial muscle movements that are not subject to people's consciousness. They reveal the genuine emotion of the subject [76]. Their intensity is very subtle and occurs in only specific parts of the face [77,78]. They occur when people try to hide their true emotions, either via suppression (deliberate concealment) or repression (non-conscious concealment) [79]. These states occur in a number of video frames without significant recognizable facial motions. The subtlety of micro-expressions can result in spatial features that are insufficient for recognizing the expressions, even in the apex frames. Full facial macro-expression lasts between 1/2 and 4 seconds [80] and is easily identifiable by humans. Micro-expressions are fleeting and imperceptible, typically lasting less than 1/5 seconds [80]. Some micro-expressions are as short as a 1/25 of a second [77,79], and some may last as long as 1/2 to 1 second [76,81]. Typically, a micro-expression lasts between 1/25 and 1/5 of a second. It is difficult for a human to notice or recognize micro-expressions; for example, [PERSON] et al. [82] found that only highly trained individuals can distinguish between various micro-expressions, and the recognition accuracy is just 47%. There is a need to design effective methods to automatically recognize micro-expressions, given that human performance on micro-expression recognition remains considerably low. Macro-expression data are in the form of still images (static displays) or videos, and micro-expression data are in the form of videos.