# Guided Spatial Transformers for Facial Expression Recognition
## 1 Introduction  
For many years, computer vision has been an active area of research. The development of Convolutional Neural Networks implied a revolution in this field as they were demonstrated to be an effective framework to improve many image processing tasks, such as image classification [1], Facial Expression Recognition [2], etc. However, they lack an attention mechanism that can identify the most relevant parts of an image. As a consequence of this necessity, the Spatial Transformer Networks (STN) appeared [3]. These models aim to detect the main regions that appear on an image and correct spatial variations by transforming the input data. Using these modified images, the following layers of the network increase their recognition rates [4; 5; 6].  
Unlike conventional STNs, where the localization network exclusively receives the original input image, we propose replacing the original input image feeds into the localization network with our generated masks. These representations prove to be adequate to improve the attention on relevant local regions of the face, as they are the eyes, mouth, nose, etc. by increasing the final recognition rate.  
To evaluate the viability of this idea, we test our proposal on a Facial Emotion Recognition task, given its interest in different fields. Recognizing emotions lets us efficiently interact with others. By analyzing user reactions, it is also possible to detect a loss of trust or changes in emotions in Embodied Conversational Agents (ECAs), letting one react to this event and adapt machine behaviors to improve interactions or modify the dialogue content, tone, or facial expression (if it has them) to create a better socio-affective user experience [7]. Furthermore, systems able to recognize certain emotions or deficits of them could help to diagnose certain diseases like depressive disorders [8], Parkinson's [9], etc., and improvethe treatment of the patients. Another relevant application of Facial Expression Recognition is for automotive safety. Recognizing negative emotions like stress, anger, or fatigue is crucial to avoid traffic accidents and increase the security on the road [10] on intelligent vehicles, allowing them to act accordingly to the state of the driver.  
The selected datasets for our work are AffectNet [11] and FER-2013 [12]. In both datasets, the strategies that employ masks instead of the original images significantly improve the results reached by the conventional STN model.  
To summarize, the contributions of this paper are as follows:  
* We propose a new module, that we call \"Mask Generator\", to attach to the Spatial Transformer to improve its performance. This module will generate several masks that are fed into the STN together with the original images. These masks, which are directly crafted from the estimated facial landmarks or assimilated as the resulting visual saliency maps from the original image, are practical to enhance the attention on relevant local regions.
* We also solve the Facial Emotion Recognition task on two popular datasets, achieving statistically significant results with our strategies, which improves the conventional STNs. The results obtained in this task expose the efficacy of this idea and open the possibility of applying the same procedures to other computer vision tasks where the ground-truth of the interest regions is not available.  
To our knowledge, this is the first work that analyzes and compares strategies to study the effect of feeding different images with embedded domain knowledge of the task into the localization network of an STN, trained end-to-end with only class emotional labels. The rest of the paper is organized as follows. Section 2 describes the related works and preceding research studies. Section 3 summarizes the methodology. Throughout Section 4, we describe the experiments, the datasets used, and some implementation details. Section 5 presents the main experiments performed and results. Finally, in Section 6, we discuss the main conclusions of our study and indicate some future research lines.