## 4 Results and Discussion
A sharp difference in validation and training accuracy can be noticed in both the models revealing a case of overfitting, as the training accuracy increases while the validation accuracy stays approximately the same with oscillations. This discrepancy in accuracy could not be alleviated by increasing dropout probability, and has been described previously in [21] by [PERSON] et al. as a general problem of AFER. They attribute this phenomenon to deep neural networks requiring a large amount of training data to avoid overfitting, and the existing facial expression databases not being sufficiently large for the task. Additionally, they point out that high inter-subject variations due to different personal attributes and levels of expressiveness, variations in pose, illumination and occlusions, all common in spontaneous interactions, contribute to the overfitting issue. To evaluate in more detail how the accuracy is distributed over classes, we can examine the confusion matrices of the models evaluated on the test dataset. Figures 7-10 depict these for the best performing frame, CNN lip and RNN lip models. For the CNN lip and RNN lip models, respectively, the face model trained on the same number of frames was also included. In addition to the confusion matrices, heatmaps of the differences in the confusion matrices between the lip models and the corresponding face models are included. Obtained by subtracting the face model matrix from the lip model matrix, positive values signify increases in accuracy for the lip model, and vice versa for negative values. The horizontal axis represents the predicted classes, while the vertical classes represent ground truth. The rows of the confusion matrices are individually normalized to a range from 0 to 1. Figure 5: Differences in accuracy between lip and facial only models on test data. Figure 6: Training and validation accuracy for the best performing RNN models. Patterns common to all of the models can be observed: _neutral_ and _sad_ appear to be somewhat ambiguous for the models; _happiness_ has a very high accuracy; and _fear_ and _surprise_ have very low accuracy. Figure 8: (**Left**) Confusion matrix of the CNN model with the same number of frames as the best performing CNN lip model. (**Right**) Confusion matrix of the RNN model with the same number of frames as the best performing RNN lip model. Figure 10: Confusion matrix of the best performing Frame model. Figure 7: (**Left**) Confusion matrix of the best performing CNN lip model. (**Right**) Confusion matrix of the best performing RNN lip model. Figure 9: (**Left**) Differences between the confusion matrices of the CNN lip and the CNN model. (**Right**) Differences between the confusion matrices of the RNN lip and the RNN model. accuracy. _Surprise_ is in part confused for _happiness_, and _fear_ for _surprise_. The frame model displays some deviations from these patterns and generally has lower accuracy, but fares well. Its biggest weak spot is the poor performance on _surprise_. If we confront CNN models with the RNN models we can also notice certain patterns, such as the RNN models being better at classifying _surprise_. We found these curious dissimilarities between the CNN and RNN models consistently across models with differing numbers of frames, which might suggest a hybrid model would achieve superior performance. The improvements obtained by the lip models over the corresponding face models are visible in Figure 9 for the RNN and CNN models, respectively, depicted as heatmaps of values obtained by subtracting the confusion matrix of the face model from the confusion matrix of the lip model. Recall that a desirable outcome for such a representation is to have positive values on the diagonal, meaning that classification accuracy has increased, and having negative values elsewhere, meaning that misclassification has decreased. We observe in Figure 9 that nearly all values on the diagonals are positive, meaning that nearly all classes improved in accuracy.