# Multimodal Emotion Recognition Using Visual, Vocal and Physiological Signals: A Review
## 4 Human Emotional Expression Recognition
### Single Modality
#### 4.1.1 Visual Modality
The databases comprise micro-expression video sequences. Existing public facial expression datasets are given in Table 6. Some micro-expression databases (e.g., USF-HD [105] and [PERSON] et al. 's [142] databases) contained micro-expressions that are actually either posed or acted out instead of naturally spontaneous ones. The posed expressions are captured by requesting that the participants form a facial expression for a certain emotion (e.g., facial image data in the entire CK dataset and a significant portion of the CK+ dataset). The acted expressions are collected when the participants are instructed to perform certain reactions and make expressions as responses to certain stimuli (e.g., eNTERFACE dataset). The posed and acted expressions are induced based on different numbers of basic emotions. They are hidden expressions--it is quite common for the real emotion of a participant not to match the facial expression made. The disadvantages of posed and acted datasets are that the occurrence duration of their micro-expressions might not be the same as the duration of the naturally spontaneous ones (e.g., USF-HD and [PERSON] et al. 's are longer (2/3 s) than [PERSON]'s definition (1/3 s)). Spontaneous natural expressions with expression classes are labeled by trained coders based on the presence of FACS (e.g., microexpression databases SMIC, CASME II and SAMM). There are also limitations on the size of the datasets, which can be insufficient for proper experimentation and analysis (YorkDDT [143] contained only 18 micro-expressions). CASME II and SAMM micro-expression datasets have objective classes, as provided in ref. [143]. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Dataset** & **No. Samples** & **No. Characters** & **Comments** \\ \hline FER2013 & 35 K (small) [35; 887] & 35,887 & Hard: Web/Montreal \\ JAFFE [144] & 213 & 1 actor & Medium \\ BU-3 DFE [145] & 2500 & 100 & Video \\ RAF-DB [146] & 30 K & 100 & Medium \\ Oulu-CASIA & 30 K & 80 subjects & Medium \\ Ferg & 55 K & 6 3D characters & Easy \\ KDEF [147] & 837 & 36 & Image \\ SFER & 30 K & - & Subtle \\ CK+ [148] & 327 (593) & 123 subjects & video \\ MUG [149] & 75 K & 476 & Image \\ TFEID [150] & 368 & 4 & Image \\ RaFD [151] & 676 & 18 & Video \\ CASME II [152] & 247 & 26 subjects & Spontaneous subtle: \\  & & & video \\  & & & Spontaneous subtle \\ SAMM [153] & 159 & 32 subjects & Spontaneous subtle: \\  & & & video \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Facial emotion expression datasets. Emotions can also be observed from images and videos of hands/heads and body gestures. Gesture-based emotion recognition has applications in realistic gaming environments and allows neuro-atypical individuals to better integrate into society [154]. Emotion recognition from full body movement (i.e., a set of body posture/gesture features) and other emotional cues can be used in rehabilitation [155]. [PERSON] et al. [156] proposed a method to assess human emotions from body gestures with near-human (perception) accuracy. The method uses static expressions extracted from the head's position and the body's inclination. By using a neural network with 50 actors across body types and ethnicities in different poses, the method provided more than 87% accuracy, despite the varied input datasets. Note that while the absolute accuracy is 87%, the accuracy compared to human emotion recognition was close to 95% in this case. [PERSON] et al. [16] proposed using CNNs and achieved their objective with an accuracy of over 95% from videos. There are recent reviews available on emotional body gesture recognition [156] and emotion recognition based on body movement [157]. Table 7 provides an overview of body gesture-based recognition methods. Gesture-based emotion recognition is challenged by the degree of subjectivity involved in establishing benchmarks. Despite the fact that visual cues have higher informational contributions in human-to-human interactions, the main disadvantage of visual emotion is that the emotion can be feigned/suppressed.