# Emotion Identification in Movies through Facial Expression Recognition
## 2 Related Work
### Facial Expression Recognition
#### 2.2.2 FER Methodologies  
Table 2 demonstrates State of the Art (SoA) approaches and results on the most widely evaluated categorical datasets. SoA approaches achieve over 90% of accuracy in CK+ [33] and JAFFE [34], which is justified since they are datasets with laboratory-controlled ideal conditions. However, datasets with subjects who perform spontaneous expressions under \"in-the-wild\" scenario conditions, such as FER2013 [26] and SFEW [41], have less satisfactory results. \begin{table}
\begin{tabular}{c c c c c} \hline
**Database** & **Sample (C.Res.)** & **Subjects** & **Source** & **Annotation** \\ \hline \multirow{2}{*}{AFEW [41]} & 1809 videos & \multirow{2}{*}{330} & \multirow{2}{*}{Movie I P \& S} & \multirow{2}{*}{6 BEs + Neutral} \\  & (RGB N/A) & & & \\ \hline \multirow{2}{*}{AFEW-VA [29]} & 600 videos & \multirow{2}{*}{240} & \multirow{2}{*}{Movie I P \& S} & \multirow{2}{*}{Valence, Arousal, FLs} \\  & (RGB N/A) & & & \\ \hline \multirow{2}{*}{AffectNet [30]} & 450,000 images & \multirow{2}{*}{N/A} & \multirow{2}{*}{Web I P \& S} & \multirow{2}{*}{6 BEs + Neutral} \\  & (RGB 425 \(\times\) 25) & & & \\ \hline \multirow{2}{*}{Aff-Wild2 [31]} & 558 videos & \multirow{2}{*}{458} & \multirow{2}{*}{Web I S} & \multirow{2}{*}{Valence, Arousal} \\  & (RGB 1454 \(\times\) 890) & & & \\ \hline \multirow{2}{*}{AM-FED+ [32]} & 1044 videos & \multirow{2}{*}{1044} & \multirow{2}{*}{Web I S} & \multirow{2}{*}{11 AUs, FLs, Liking} \\  & (RGB 320 \(\times\) 240) & & & \\ \hline \multirow{2}{*}{CK+ [33]} & 593 videos & \multirow{2}{*}{123} & \multirow{2}{*}{Lab I P} & \multirow{2}{*}{7 BEs + contempt, AUs, FLs} \\  & (RGB 640 \(\times\) 480) & & & \\ \hline \multirow{2}{*}{EmotioNet [8]} & 950,000 images & \multirow{2}{*}{N/A} & \multirow{2}{*}{Web I P \& S} & \multirow{2}{*}{12 AUs, 23 BE and CEs} \\  & (RGB N/A) & & & \\ \hline \multirow{2}{*}{FER2013 [26]} & 35,887 images & \multirow{2}{*}{N/A} & \multirow{2}{*}{Web I P \& S} & \multirow{2}{*}{6 BEs + Neutral} \\  & (G 48 \(\times\) 48) & & & \\ \hline \multirow{2}{*}{KDEF [35]} & 4900 images & \multirow{2}{*}{70} & \multirow{2}{*}{Lab I P} & \multirow{2}{*}{6 BEs + Neutral} \\  & (RGB 562 \(\times\) 762) & & & \\ \hline \multirow{2}{*}{JAFFE [34]} & 213 images & \multirow{2}{*}{10} & \multirow{2}{*}{Lab I P} & \multirow{2}{*}{6 BEs + Neutral} \\  & (G 256 \(\times\) 256) & & & \\ \hline \multirow{2}{*}{MMI [36; 37]} & 2900 videos & \multirow{2}{*}{75} & \multirow{2}{*}{Lab I P} & \multirow{2}{*}{6 BEs + Neutral, AUs} \\  & (RGB 720 \(\times\) 576) & & & \\ \hline \multirow{2}{*}{OULU-CASIA [38]} & 2880 videos & \multirow{2}{*}{80} & \multirow{2}{*}{Lab I P} & \multirow{2}{*}{6 BEs} \\  & (RGB 320 \(\times\) 240) & & & \\ \hline \multirow{2}{*}{RAF-DB [39; 40]} & 26,672 images & \multirow{2}{*}{N/A} & \multirow{2}{*}{Web I P \& S} & \multirow{2}{*}{6 BEs + Neutral,} \\  & (RGB N/A) & & & & \\ \hline \multirow{2}{*}{SFEW [41]} & 1766 images & \multirow{2}{*}{95} & \multirow{2}{*}{Movie I P \& S} & \multirow{2}{*}{6 BEs + Neutral} \\  & (RGB N/A) & & & & \\ \hline \end{tabular}
\end{table} 
Table 1: Principal databases used in FER systems (C. = color; Res. = resolution; G = grayscale; RGB = RGB-colored; P = posed (expression); S = spontaneous (expression); BE = basic emotion; AU = action unit; CE = compound emotion; FL = facial landmark). \begin{table}
\begin{tabular}{c c c} \hline
**Study (Year)** & **Approach** & **Acc (\%) (No. Classes)** \\ \hline \multicolumn{3}{c}{CK+ [33]} \\ \hline
[43] (2019) & FAN & 99.69 (7) \\
[44] (2016) & CNN & 98.9 (6) \\
[45] (2017) & CNN & 98.62 (6) \\ \hline \end{tabular}
\end{table} 
Table 2: FER approaches and results on widely evaluated datasets. As shown in Table 2, CNN-based approaches are the foundation of SoA results and can be applied to FER tasks to achieve consistent performances. These SoA methods/models are derived from traditional DL architectures, which use well-known backbones for feature extraction (e.g., VGG, ResNet).