# Emotion Identification in Movies through Facial Expression Recognition
## 4 Results  
Following the experimental design referred in Section 3, to set a baseline for our work, we benchmarked several SoA CNN architectures that were initialized with pre-trained weights from ImageNet. The selected backbones were MobileNetV2, Xception, VGG16, VGG19, ResnetV2, InceptionV3 and DenseNet. These models were selected based on their solid performance in other image challenges, with the premise that they could also be applied to FER tasks. FER2013 was separated into training and testing sets. The baseline models were optimized using cross-entropy and accuracy, for validation purposes, during 25 epochs with a mini batch size of 128. The initial learning rate was set to 0.1, being decreased by a factor of 10% if the validation accuracy did not improve for three epochs. Moreover, the dataset was also extended by applying data augmentation with a probability of 50% on every instance. The selected augmentation methods were horizontal flip and width/height shift (min 10%). Table 5 presents these results for each baseline architecture, while Figures 1-6 illustrate their corresponding confusion matrices. From the results, it is clear that none of the vanilla models achieved SoA results. In contrast, Xception performed well in inference time, with the second fastest training time of our tests, and the best accuracy result. Taking into account this preliminary analysis, Xception was selected as the baseline model for the purpose of the study to be conducted next. Since SFEW has few samples, FER2013 was used to train the selected model using a large in-the-wild database of facial expressions. The trained model was tested with SFEW since it contains faces of actors directly extracted from film frames. This enables understanding whether the developed model is robust enough to adapt to a new context. Results are shown in Table 6 and Figures 7 and 8. From the presented numbers, we can conclude that Xception was able to achieve an overall accuracy of 68% in FER2013, which is within state-of-the-art values. Additionally, since FER2013 is a dataset built in a lab environment with a controlled image capturing conditions, these experiments will allow us to analyze whether a network trained in these conditions will have the ability to generalize to the film domain, by testing it with SFEW. Having achieved the first objective, the next step was to simulate a real testing scenario of the network by submitting it to images taken from films. From a pool of 891 images, results were not satisfactory, reaching an overall accuracy of only 38%. Given this result, the next step was to address an already identified problem: the imbalance of FER2013. \begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Train** & **Validation** & **Test** \\ \hline Angry & 178 & 77 & \\ Disgust & 49 & 23 & \\ Fear & 78 & 46 & \\ Happy & 184 & 72 & \\ Sad & 161 & 73 & \\ Surprise & 94 & 56 & \\ Neutral & 144 & 84 & \\ \hline Total & 888 & 431 & 372 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: SFEW aligned face samples per class. The test set contains 372 unlabeled images. Figure 1: MobileNetV2—training set (FER2013). Figure 2: Xception—training set (FER2013). \begin{table}
\begin{tabular}{c c c} \hline \hline  & **Inference Time (ms)** & **Accuracy (\%)** \\ \hline MobileNetV2 & 85.32 & 52.57 \\ Xception & 245.85 & 64.71 \\ VGG16 & 1364.90 & 61.43 \\ ResnetV2 & 4024.74 & 62.57 \\ InceptionV3 & 1127.68 & 53.14 \\ DenseNet & 1131.98 & 58.86 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Benchmark of CNN architectures. Figure 4: ResNetV2—training set (FER2013). Figure 5: Inceptionv3—training set (FER2013). Figure 3: VGG16—training set (FER2013). Figure 6: DenseNet—training set (FER2013).