# Multimodal Emotion Recognition Using Visual, Vocal and Physiological Signals: A Review
## 5 Deep Learning Challenges and Solutions for High-Quality Emotion Recognition  
In the literature, different methods have been proposed to improve the performance of facial micro-expression recognition tasks and, in particular, to obtain discriminative features. One of the methods is expression magnification, also referred to as motion magnification, which is a data processing method that magnifies the motion features of the original micro-expression video clips (examples of magnifications for micro-expression recognition can be found in references [250; 251; 252]). In ref. [253], instead of feeding the features extracted from the whole single face into the classifier, the authors adopted the use of part-informed features, where the motion features extracted from different parts of the face (e.g., eyes, \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset** & **No. Emotions** & **No. Utterances** & **No. Persons** & **Comments** \\ \hline \hline \multirow{2}{*}{Belfast [241]} & \multirow{2}{*}{8} & \multirow{2}{*}{1440} & \multirow{2}{*}{60} & dimensional + \\  & & & & categorical \\ eNTERFACE [242] & & & & - \\ \multirow{2}{*}{HUMAINE [243]} & \multirow{2}{*}{7} & \multirow{2}{*}{50} & \multirow{2}{*}{10} & naturalistic and \\  & & & & induced data + \\  & & & & & text \\  & & & & & Actors \\  & 10 & 5K-10K & 10 & conversation + \\  & & & & text \\ SEMAINE [244] & & 959 & 150 & - \\ RAVDESS [245] & & 1440 & 60 & - \\ CREMA-D [246] & & 7442 & 91 & - \\ GEMEP [247] & 15 & 1260 & 10 & - \\ MSP- & & - & 12 & - \\ IMPROV [248] & & & & Dimensional \\ DEAP & - & EEG and audio & 25 (useful) & emotion model \\  & & & & description \\ \hline \hline \end{tabular}
\end{table} 
Table 15: Multimodal emotion recognition datasets. eyebrow, nose and mouth) are treated separately and classified separately. The part-informed features are split for the single feature map obtained after the last convolutional layer to form a part-based classification. Part-informed features can also be fused at a later stage. This method injects structure priors into the classification network. For example, in ref. [117], the feature map after the last convolutional layer is split into two parts to represent the eye and the mouth areas separately. Part-informed features offer fine-grained information from the input source. In addition, the part-informed forces the encoder to learn representations focusing on local motions on the face, which is discriminative for expression reconnecting. To succeed in implementing the part-based deep neural network (or part-based classification in general), variations in the natural scene that are not related to facial expressions, such as head posture and background, must be considered. Image registration must be conducted as data preprocessing to have similar head posture, remove background effects [117], and reduce the impact of head posture variations. In subtle emotions, emotion recognition must consider and handle the dynamic emotion subtleties that depend on an individual's background or condition to better encode fine patterns present in the facial image. Deep learning-based methods (e.g., [92; 254]) overcome the shortcomings of the geometric feature-based methods and appearance-based methods. The extraction of deep learning features, the same as handcrafted features, requires face alignment during preprocessing and may require facial landmarks extracted to crop the region of interest in case the facial data are not pre-cropped. However, public datasets such as CASME II have pre-cropped video frames. Small sample sizes remain an impediment for deep learning-based approaches, given that the performance of deep learning models depends on the size of the dataset. Therefore, using larger training datasets can improve the performance of deep learning methods.