## 3 Methodology
### Facial Emotion Recognizer
#### 3.3.2 Static vs. Sequential Models  
Once we obtained the AUs of each video frame, we needed to accomplish a pre-processing step to adapt the input format to the specific model, whether static or sequential. To evaluate the static models, we computed the average vector of the sequence of AUs extracted from each video, collapsing all the temporal steps into a single softened vector. In this stage, we tested two strategies, applying a normalization of the AUs in the range of 0-1 for each column, or not employing any normalization. After adapting the AUs to the static problem, we introduced the samples into different models: an SVC, a k-NN classifier, and an MLP. Figure 3: Plots of the 18 AUs generated by the OpenFace library [71] for the intensity version ranging from 0 to 5 of two videos of the RAVDESS dataset. (**a**) AUs’ intensities for an angry sample; (**b**) AUs’ intensities for a happy sample. This approach, adopted as our baseline, had two main benefits: the first one was its simplicity, and the second one was 'the average effect'. To illustrate this, suppose that a video has a deviation on several frames from a prototype emotion because the person on it has closed their eyes. Thanks to the average pooling, these frames would not severely influence the final recognition, as long as all the other frames encapsulated the correct emotion. Nevertheless, this method has also an evident drawback: sequential data may exhibit a natural temporal ordering. However, this heuristic aggregation of individual frame-level features ignores temporal order, which in turn may result in a sub-optimal discriminative capability. As an alternative to the static models, we have adopted a sequential model, which is usually effective, especially for sequential data, assuming that there is relevant information in the order of the frames. As a sequential model, we employed an RNN. This RNN used a long short-term memory (LSTM) network. This model can process its inputs sequentially, performing the same operation, \(h_{t}=f_{W}(\overrightarrow{AU_{t}},h_{t-1})\), on each of the distinct timesteps that conform to our input series, where \(h_{t}\) is the hidden state, \(t\) the time step, and \(W\) the weights of the network. As input to this model, we introduced the sequence of AUs produced per frame of the video \((\overrightarrow{AU_{1}},\overrightarrow{AU_{2}},\ldots,\overrightarrow{ AU_{N}})\), to exploit the temporal patterns enclosed in the AUs and make a final prediction at the video level. Regarding the architecture of the sequential model, it consisted of several bidirectional-LSTM layers with a deep self-attention mechanism, similar to the proposed in [72]. In Figure 4, we show a picture of the structure of the employed Bi-LSTM. The Bi-LSTM layer works in a bidirectional way, which allowed us to collect the sequential information in both directions of the hidden states \(h_{1},h_{2}\ldots,h_{N}\) of the LSTMs. In particular, our Bi-LSTM consisted of two LSTMs: _forward LSTM_, which permitted the analysis of the frames from \(\overrightarrow{AU_{1}}\) to \(\overrightarrow{AU_{N}}\), and an _inverse or backward \(\overleftrightarrow{LSTM}\)_, which permitted a similar analysis to be carried out but in the opposite direction, from \(\overrightarrow{AU_{N}}\) to \(\overrightarrow{AU_{1}}\). Figure 4: Bidirectional-LSTM with attention mechanism for emotion recognition using sequences of Action Units at the input. Modified version from source [73]. To obtain the emotional tag from the Bi-LSTM layers, we concatenated the embeddings of the outputs of each specific direction (see Equation (1) in which \(||\) corresponds to the concatenation operator and \(L\) to the size of each LSTM). \[h_{i}=\overrightarrow{h_{i}}||\overleftarrow{h_{i}}\,\text{ where }h_{i}\in R^{2L} \tag{1}\]  
The main task of the attention layer was to distinguish the most relevant AUs associated with a specific frame when determining the emotion portrayed in the whole video.