# Guided Spatial Transformers for Facial Expression Recognition
## 2 Related Work  
### Spatial Transformer Networks  
Due to the adequacy of STNs [3] to solve visual tasks, they have been employed in many domains. We have classified the latest publications in the literature into three groups: task-based, framework-based, and model-based.  
The first group that we called task-based publications contains all the studies that apply STNs to previous areas. Until the appearance of the STNs, these problems were solved with other architectures such as CNNs. However, with the development of these models, the community reached state-of-the-art results applying STNs on previously studied tasks. Some examples of this success occurred in the areas of lip movement detection [4], emotion recognition [5], or saliency prediction [13].  
The second trend of publications, which are framework-based, includes research studies that investigate how to connect or incorporate this model in other frameworks. As in [14], where the authors combine an STN with a Generative Adversarial Network (GAN) to create more realistic images, or in [15], with the aggregation of an STN before a CNN network that recognizes people, making use of triplet loss as the cost function.  
The third relevant line of STN papers, those that are model-based, aims to improve the original version of the STN by combining it with other ideas as in the work of [PERSON] et al. [16] that integrate Lucas and Kanade (LK) algorithm into a classic STN, or the proposed STN-RNN in [17] that uses recurrent neural networks into an STN pipeline.  
This third group also includes the work of [PERSON] et al. [18]. In their article, the authors argue that their \"Image-and-Spatial-Transformer Networks\" (ISTN) could improve the medical image registration problem, which consists of the alignment of several images. They propose to add an extra network on top of an STN. The top network (ITN) generates the segments that conform to the input image. Then, the STN predicts the transformation matrix to align the images from the generated segments of the ITN.  
However, for the training of the ITN, it is necessary to have the ground truth of the images with the landmarks of the segments correctly annotated to train the ITN network.  
Our proposal follows a similar idea in a more general way because we do not have access to the ground truth of the attention regions. Instead, we evaluate the images generated automatically from different general purpose pre-trained networks that emphasize the most relevant areas of an input image. With these generic masks, we conduct an ablation study to assess the impact of passing each of them to the localization network of the STN. Results reveal that their inclusion enhances the performance of conventional STNs for the emotion recognition task.  
One of the advantages of our proposal is that we do not need to re-train the models that extract the masks, as they are general purpose models that can be applied to several tasks. The use of these generalistic models reduces the manual annotation effort required to create supervised-tailored models.  
Apart from the similarities with the work in [18], there are also several differences to our work. The first one is that we apply our proposal to a different domain, thus we require other landmarks extractors that detect morphologically different regions. Another of the main differences is the architecture of the STN that we use, as we need to add some extra layers after predicting the transformation matrix parameters to address the Facial Emotion Recognition task. Finally, we extend the ablation study that they do by evaluating modified versions of the landmarks and the idea of using the saliency maps, everything extracted automatically without re-training these models.