# Guided Spatial Transformers for Facial Expression Recognition
## Appendix C Analysis of Errors
### Most Common Errors across Emotions  
To apprehend the limitations of our models, we accomplish an investigation to distinguish the most distanced errors in terms of their valence values. These errors are those labeled as positive but predicted as negative, and those annotated as negative but predictive as positive. As both datasets have annotations of the emotions, we do this analysis across emotions. Instead of analyzing a specific strategy, we consider the most failed images by the following STN models: Baseline STN, STN with landmarks (binary, soft, and dilatation), and saliency. These erroneous images should inspire future versions of the models and the set-up to reduce the number of failed samples. Starting with AffectNet, we can see in Table 13 that most of the misclassified images correspond to the contempt category. Checking some of the contempt images in Figure 14 give an intuition about what may be happening. It seems that our models are confusing 'Happiness' with 'Contempt'. This mistake is understandable since both emotions share common patterns, but in AffectNet 'Contempt' images are assigned to a strong negative valence, in average this class has a valence of \(-0.51\) in the annotations. The second emotion most mistaken corresponds with 'Sad'. We can see in Figure 14 that some of the pictures show people crying because of the emotion but not because they feel sad. These images could also confound the model since they contain information from two different emotions. From Figure 14, we can see that in most samples people are smiling or showing their teeth. This common characteristic makes us think that the systems are learning more information from the mouth than other parts of the face, which could be reasonable because of the resolution of the images and the importance that the mouth has. Figure 13: Samples of miss-detected faces on FER-2013 by the STN with landmarks—soft masks model. TL version. Figure A4: Samples predicted as positive but with negative label. Examples of the most negative valence images incorrectly classified. \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & **Neutral** & **Sad** & **Surprise** & **Fear** & **Disgust** & **Anger** & **Contempt** & **None** \\ \hline
**fold 1** & 35 & 136 & 3 & 9 & 57 & 102 & 316 & 71 \\ \hline
**fold 2** & 48 & 141 & 3 & 2 & 70 & 99 & 342 & 57 \\ \hline
**fold 3** & 38 & 125 & 2 & 4 & 74 & 82 & 306 & 57 \\ \hline
**fold 4** & 37 & 142 & 2 & 7 & 75 & 87 & 317 & 58 \\ \hline
**fold 5** & 33 & 127 & 2 & 5 & 67 & 109 & 364 & 61 \\ \hline
**Total** & 191 & 671 & 12 & 27 & 343 & 479 & 1.645 & 304 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Samples predicted as positive but with negative label. Common errors on AffectNet of models: Baseline-STN, STN with landmarks (binary and soft masks) and STN with saliency. Regarding the negative predictions for real positive valence, we can see in Table 14 and Figure 15 some of the common wrongs recognized pictures by our models. If we focus on the most numerous group, that correspond with the 'Happy' category, there are several reasons that may explain the errors committed. One cause may be the occlusions. A second reason could be the age difference since ancient people and babies express their emotions differently because of the morphology of their faces. Although both datasets do not contain annotations about age ranges, the under-representation of some of them could be introducing a bias in our models. Concerning 'None' images, they represent different emotions what makes this group especially challenging. In fact, the standard deviation of valence annotations of this group is 0.40 against the other emotional categories whose deviation varies between 0.07 and 0.22. Introducing this group in the training and evaluation may have altered the learning process since annotations are not as clear as for other categories. Still, it is interesting to have it because it is closer to 'in the wild' environments. From this analysis, we draw several conclusions.