# Multimodal Emotion Recognition Using Visual, Vocal and Physiological Signals: A Review
## 4 Human Emotional Expression Recognition
### Single Modality
#### 4.1.1 Visual Modality
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Ref** & **Year** & **Database** & **Features/Classifier** & **Best Performance** \\ \hline
[118] & 1994 & in-house & Neural network & Acc: 68\% to 89\% \\
[119] & 2002 & JAFFE & Neural network & Acc: 73\% \\
[120] & 2005 & DFAT-504 & Gabor filters + AdaBoost/SVM & Acc: 93.3\% \\
[121] & 2016 & FER2013 & + & DNNRL & Acc: 71.33\% \\  & & SFEW 2.0 & & & \\
[122] & 2017 & CK+ & PHRNN + MSCNN & Acc: 98.50\% \\  & & Oulu-CASIA & & Acc: 86.25\% \\  & & MMI & & Acc: 81.18\% \\
[123] & 2018 & in-house & Wavelet entropy/ & Acc: 96.80\% \\  & & Neural network & & \\
[124] & 2018 & KDEF & CNN/SVM & Acc: 96.26\% \\  & & CK+ & & Acc: 95.87\% \\
[125] & 2020 & FACES & VGG-16/RF & Acc: 97.21\% \\  & & Lifespan & & Acc: 98.08\% \\  & & CIFE & & Acc: 84.00\% \\  & & FER2013 & & Acc: 71.50\% \\
[126] & 2023 & CK+ & CNN + DAISY/RF & Acc: 98.48\% \\  & & FER2013 & & Acc: 70.00\% \\
[127] & 2023 & JAFFE & VGG19 + GoogleNet + & Acc: 97.62\% \\  & & CK+ & ResNet101/SVM & Acc: 98.80\% \\  & & FER2013 & & Acc: 94.01\% \\  & & SFEW 2.0 & & Acc: 88.21\% \\
[128] & 2024 & Aff-Wild2 & Landmarks/GCN & Acc: 58.76\% \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Facial macro-expression recognition using deep learning. trained spatial model are fed into an LSTM network to learn the temporal dependencies of expression-state spatial features for a given emotion [129]. The spatial features of an emotion category are learned at the expression state level of a video emotion using a CNN. Three expression states (onset, apex, offset) and the transitions from onset to apex and apex to offset are counted as states. The aim is to cluster the expression states of each emotion category, and, by doing so, to cluster the emotion categories. In other words, the spatial features of emotion are learned, regardless of the expression state. The loss function ensures continuity in consecutive states of an emotion category. Several objective functions are optimized during spatial learning to improve expression class separability, and the expression states are adopted in the objective function during spatial feature learning. The temporal dimension represents the emotion dynamics over time, which is crucial for recognizing facial movements and discriminating between emotion classes. Spatial and temporal modules are needed to attain a good performance. [PERSON] et al. [130] propose a method that uses an LSTM temporal module to learn temporal dynamics, preserve the temporal dimension and characterize temporal dynamics. For the spatial learning stage, the input data are encoded by a CNN to a fixed-length vector, \(\phi(x_{t})\), that represents the spatial features at time \(t\). Subsequently, for the temporal learning stage, a 4096-fixed length \(\phi(x_{t})\) is then passed to a recurrent neural network (LSTM) to learn the temporal dynamics. This is similar to ref. [129], but with no consideration of expression state. Two variants are proposed, both having the following inputs: optical flow (3D flow image made of horizontal flow dimension, vertical flow dimension and optical flow magnitude), optical strain and grayscale images. The spatial enrichment variant uses a single VGG-16 CNN applied to a large stacked 3D optical flow image, a 2D optical strain image, and a grayscale image of the microexpression data. The VGG-16 was trained from scratch, and the trained model gives a 4096-fixed-length feature vector at the last fully connected layer to be fed into the LSTM. The temporal enrichment model uses three pre-trained VGG-Face models that were trained on large-scale Labeled Faces in the Wild (LFW) data for face recognition [131] (i.e., transfer learning that allowed for fast convergence because both microexpression data and LFW data involve faces and their components).