# Multimodal Emotion Recognition Using Visual, Vocal and Physiological Signals: A Review
## 5 Deep Learning Challenges and Solutions for High-Quality Emotion Recognition
In addition, the problem of class imbalance for supervised learning algorithms that tends to bias the classification towards the majority classes leads to poor classification--classes with more data in the training set are distinguished better and have better accuracies [130]. For example, the temporal interpolation model (TIM) method can be used as a data augmentation approach to increase data but can provide less information than real data [130]. Likewise, combining different data sources can degrade classification instead of improving classification. For example, the model trained on a single domain (CASME II) had more salient locations (action units) than that on a cross-domain (CASME II and SAMM), which can hinder facial action unit-based classifiers. The combination of datasets gathered from different sources may also mask important information [255]. To properly handle class imbalances, one well-accepted balanced performance metric of micro-expression recognition is the F1 score [111]. Though multimodal emotion analysis models should be trained on big data from diverse contexts and diverse people to build generalized models, robust techniques must be designed to account for unseen contexts. For example, given speakers in speech emotion recognition, it has been shown that languages affect the emotional recognition of the speakers. However, a dataset used for the design of speech emotion recognition is limited in the number of languages used by the speakers in the dataset, and this limitation affects the problem of the designed model when it is deployed. Effective modeling of temporal information in big data can also be devised. For the problem of small training sets, especially for micro-expression recognition tasks, and for the problem of class imbalance, the recognition task can use different deep-learning strategies. One of the strategies is the transfer of domain knowledge from macro-expression recognition tasks, namely the source, such as CK+ [148] or BU-3 DFE [145]) to micro-expression recognition (namely, the target) using a transfer learning technique for micro-expression tasks. Different domain adaptation techniques, also referred to as style-aggregated methods, have been used and help to enrich the available training samples. These methods include adversarial training [256; 257] such as cycleGAN used in ref. [115; 117] to obtain domain-invariant features. These methods can be implemented together with the attention transfer mechanism from a teacher model (for the macro-expression) to a student model (for the microexpression) (e.g., [132]) to improve recognition accuracy. Better transfer learning techniques can be explored for further works targeting micro-expression recognition. Another domain adaptation technique that is different from transfer learning methods is the expression magnification and reduction (EMR) technique [117]. Data augmentation can also be implemented by increasing optical flow data for each micro-expression video clip. For example, in ref. [116], for each micro-expression video clip used for training, in addition to the optical flow between the onset and apex frame, a second flow computed between the onset and apex+1 frame was also included. The accuracy of emotion recognition is not satisfactory, given the high disagreement regarding emotion state when different methods are assessed and approximate accuracies of less than 80%. However, considering the context/situation as secondary data to enhance the recognition of the user's emotional state can improve the recognition. The representation of input also data contributes to the performance of recognition. For example, the representation of a face image as a 3D object on a manifold is useful in capturing the shape deformation and fine details of an emotional expression. Therefore, the recognition system can first extract the emotional features from the manifold through a deep learning-based approach.