# Emotion Identification in Movies through Facial Expression Recognition
## 3 Proposed Methodology  
Based on the evidence discussed in Section 2.2, it becomes clear that there are no sufficient large-scale movie datasets with face-derived emotion annotations. As a direct consequence, there are not many studies that validate the use of FER deep learning models specifically for the movie domain. Therefore, the problem we investigate can be defined through the following research questions: Can current FER datasets and Deep Learning (DL) models for image classification lead to meaningful results? What are the main \begin{table}
\begin{tabular}{c c c} \hline \hline
**Study (Year)** & **Approach** & **Acc (\%) (No. Classes)** \\ \hline \multicolumn{3}{c}{FER2013 [26]} \\ \hline
[46] (2016) & CNN-VGG & 72.7 (7) \\
[45] (2017) & CNN & 72.1 (7) \\
[46] (2016) & CNN-Inception & 71.6 (7) \\ \hline \multicolumn{3}{c}{JAFFE [34]} \\ \hline
[47] (2019) & SVM & 97.10 (7) \\
[48] (2019) & 2 channel-CNN & 95.8 (7) \\
[49] (2019) & ATN & 92.8 (7) \\ \hline \multicolumn{3}{c}{SFEW [41]} \\ \hline
[50] (2015) & CNN & 55.96 (7) \\
[51] (2015) & CNN & 53.9 (7) \\
[52] (2019) & CNN (ACNN) & 51.72 (7) \\ \hline \multicolumn{3}{c}{OULU-CASIA [38]} \\ \hline
[53] (2018) & GAN + CNN & 88.92 (6) \\
[52] (2019) & CNN (ACNN) & 58.18 (6) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: \(Cont.\)challenges and limitations of FER in the movie domain? How can current results on affective/emotional analysis with other media be translated to FER in the cinema domain? Are the current emotional models adequate to the cinema domain where expressions are more complex and rehearsed? Based on these research questions, we defined the following steps as the experimental design:  
1. From the list of available datasets provided in Section 2.2.1, we analyzed and selected a dataset for training the DL models and evaluated them in the movie domain;
2. We pre-processed the selected datasets through a facial detector to extract more refined (tightly cropped) facial regions;
3. We tested and benchmarked CNN architectures using accuracy as a performance metric. Furthermore, this first evaluation will also tackle the unbalance of the training dataset;
4. Following the findings reported in Section 2.1 to study an approach for dimensionality reduction which allows to compare our findings with other domains (e.g., audio and text). This final step is divided into two approaches: 1. Using only the top-N performing classes; 2. Clustering the classes using the emotion clusters found in other studies from the SoA. Within the datasets introduced in Section 2.2, none fit perfectly the requirements since there is no large-scale FER database in the film domain. Thus, we propose using a cross-database scenario involving two in-the-wild settings that can unite the benefits of a large database with the benefits of a film-based database. For that purpose, FER2013 [26] was selected based on its size and in the fact that it includes both posed and spontaneous samples. This dataset was created using the Google image search API with 184 different keywords related to emotions, collecting 1000 images for each search query. Images were then cropped in the face region and a face-alignment post-processing phase was conducted. Prior to the experiments, images were grouped by their corresponding emotions. Each image, represented in the 48x48 vector in pixels, is labeled with an encoded emotion. The number of samples per class of the dataset is presented in Table 3. The imbalance of the dataset is fairly evident, especially between disgust (with only 547 samples) and happy (with 8989 samples) classes. This imbalance is justifiable as it is relatively easy to classify a smile as happiness, while perceiving anger, fear or sadness is a more complicated task for the annotator. SFEW [41] was also chosen for this analysis since the images were directly collected through film frames. Furthermore, the labels of SFEW are consistent with FER2013 dataset, making the aforementioned cross-database study possible.