# Multimodal Emotion Recognition Using Visual, Vocal and Physiological Signals: A Review
## 4 Human Emotional Expression Recognition  
The performance of an emotion recognition system greatly depends on the dataset's characteristics. Publicly available datasets for emotion recognition are used to identify specific emotions and serve as a benchmark to establish a standard comparison of emotion recognition algorithms. Each database has its advantages and disadvantages. Based on current methodologies for dataset collection for emotion recognition, data can be categorized as natural/naturalistic, induced and acted. Acted data are recordings of subjects acting based on pre-decided scripts--emotion data are collected while an individual feigns an emotion. Induced data are data collected when an individual observes a scene that is expected to induce a specific emotion--emotion data are collected while an individual experiences multimedia stimuli. Natural data, such as natural videos, are gathered from recordings taken without the expressers knowing that the emotional data are being extracted from them. Acted datasets can suffer from inaccurate actions by subjects, leading to corrupted samples or erroneous information for the training dataset (figned expression). For the induced data, an individual might express an emotion we do not expect. In such a case, data validation will be the next step; for example, using facial action units (FAUs) in the case of facial expressions. The main obstacle is obtaining accurate data to train the recognition. Class imbalances exist in almost all the databases. Therefore, to address this problem, recognition methods implement different data augmentation mechanisms, and the evaluation of the recognition methods uses performance measures that are less sensitive to imbalance.  
Emotion data and emotion recognition methods focus on a single modality (emotion state assessed either from vocal expression, visual expression or physiological cues) or multimodality (emotion state assessed, for example, from audio-visual expressions). The emotion recognition model based on single-modality data has some limitations that can be solved using effective emotional multimodality. However, since the multimodal methods take advantage of a particular unimodal method for each modality, the unimodal methods that are part of the multimodal method must also perform well on their own. Some studies have shown that multimodal methods can have lower performances, which happens when the necessary steps are not taken to ensure that the multimodal methods improve the unimodal methods [72]. Some unimodal methods can recognize subtle emotional expressions, and by combining subtle expression recognition methods from multiple modalities, multimodal methods for subtle expression recognition can be expected to perform better.  
Emotion recognition methods can directly classify emotion states. However, they can also classify these states based on non-emotional categories, such as the objective states made of combinations of action units (AUs) for facial recognition [73]. However, the classification must be followed by emotion interpretation to derive the emotions from the objective states. Since emotion is dynamic, emotion has a start time and an end time. The period of emotion can be divided into stages: onset (start time of an emotion experience), apex (time of the maximum emotion intensity that can be inferred; for example, from the maximum muscle movement of the facial expression starting from the onset), and offset (end of an emotional experience). There are also two transition periods: onset to apex and apex to offset. Therefore, the recognition results will depend on the emotion stage of the input data frame (or image) to the recognizer for the case of static expression.