# Guided Spatial Transformers for Facial Expression Recognition
## 3 Methodology
### Spatial Transformer Network  
We implement a Spatial Transformer Network. As defined in [3], the classic version contains three main modules: the localization network, the grid generator, and the sampler. The localization network receives the input image \(U\) and outputs the \(\theta\) parameters that form the transformation matrix, \(\tau_{\theta}\). These parameters vary depending on the input image and change during the training because they usually correspond to the weights extracted from a fully connected layer. This module is in charge of detecting the most relevant regions to focus on the image for identifying the class in later layers. The grid generator creates a regular grid of a certain size and receives the parameters \(\theta\) of the localization network. With these two inputs, it applies an affine transformation on the regular grid \(G\), resulting in the parametrized sampling grid _G_s, which is a warped  
Figure 1: Samples of the generated masks on the AffectNet dataset. Columns are organized as follows: Original image, Landmark-based binary mask, Landmark-based soft masks, Landmark-based dilated binary masks, Visual Saliency based masks, Landmark-based facial patches, and Landmark-based image weighing masks. The first two rows have images with positive valence, the middle row shows an image with neutral valence, and the 2 last rows are examples of negative valence. version of the regular grid centered on the most important pixels of the input image \(U\), as \(\theta\) is estimated from \(U\). This sampling grid is a mapping that lets us infer a relationship between the coordinates of the pixels in the input image, \(U\), and the expected position of the transformed pixels at the output image, \(V\). This step is represented by the following equation:  
\[ \begin{pmatrix}x_{i}^{s}\\ y_{i}^{s}\end{pmatrix} =\tau_{\theta}(G)= \begin{pmatrix}\theta_{11}&\theta_{12}& \theta_{13}\\ \theta_{21}&\theta_{22}&\theta_{23}\end{pmatrix} * \begin{pmatrix}x_{i}^{t}\\ y_{i}^{t}\\ 1\end{pmatrix}  \tag{2}\]  
where \((x_{i}^{s},y_{i}^{s})\) are the source coordinates of the input image \(U\) that define the sample points of \(G_{s}\); \((x_{i}^{t},y_{i}^{t})\) are the target coordinates of the regular grid \(G\) in the output image, \(V\); and \(\theta\) is the coefficient of the transformation matrix. Notice that the transformation matrix has an inverse. By taking the inverse and the input image, it is possible to obtain the values of the pixels in the output image. Finally, the sampler module applies the spatial transformation by taking the sampling grid \(G_{s}\) and the input image \(U\), to produce the transformed image \(V\). To obtain the value of each pixel of \(V\) when there is no direct correspondence with the input image pixels, it applies an interpolation based on these 2 inputs. In Figure 2, we can see an example of how the STN transforms an input image. The transformed image passes to the classification network, usually a CNN or an MLP. Both the spatial transformer and classification networks are complementary and their combination ensures the sufficient utilization of both global and local features to solve the recognition problem. In this work, we use a compacted STN architecture with few convolutional and fully connected layers, similar to the one reported in [5], as their results indicate that these small networks can solve Facial Emotion Recognition problems with relatively high accuracy. In Appendix B, we summarize our architecture with the input layers, the expected output size, and the filters and strides used for the convolutional layers and max-poolings. Unlike in classic STN which receives the same image at the input of the localization network and the sampler, we examine the effect of adding an extra module, the mask generator, that changes the input that receives the localization network by a mask. More specifically, we evaluate whether precomputed masks that contain information about the relevant areas to focus on may improve the performance of the classic STN.