## 2 Related Work
### Facial Emotion Recognition  
Although voice is a crucial indicator of a subject's emotion, other modalities could enhance the SERs' performance, as demonstrated by [PERSON] et al. in [33], which incorporated textual features to supplement the speech emotion recognizer. In our scenario, we included the visual information of the facial expressions.  
Several libraries allow for the detection of facial variations and facial morphology. The dlib library [45], which estimates position of the landmarks on a face, is an example of these tools. According to [PERSON] et al. [46] and [PERSON] et al. [47], landmarks encapsulate meaningful information about a person's facial expression that helps to solve automatic emotion recognition.  
As an evolution of the facial landmarks, we have the facial action coding system (FACS) [48]. This system consists of a set of descriptions that refer to the movement of the facial muscles and are coded in what [PERSON] et. al. called Action Units (AUs). Due to their simplicity and power, they have been investigated and utilized in several publications, such as in the work of [PERSON] et al. [49], where they employed 12 AUs to classify emotions in the Cohn-Kanade (CK) database using a decision-tree; obtaining a recognition rate of 90%.  
In the work of [PERSON] et al. [50], they used a first classifier to predict the presence of 14 AUs. Then, they fed the output of this classifier into an SVM to recognize emotions, reaching an average recognition rate of 94.07% for females and 90.77% for males on the CK database. In [51], [PERSON] et al. compared two systems: the first one consisted of training a classifier for detecting the presence of AUs, followed by an emotion recognizer; and the second one involved using only an emotion recognizer classifier. Their results confirmed that the system based on AUs reached higher scores than the version without including the AUs on several datasets, when they both receive facial images coded by local gaborbinary pattern (LGBP) histogram differences. These results reinforced the reliability of AUs for performing emotion recognition.  
In the contribution of [PERSON] et al. [52], they extracted 15 pairs of AUs' descriptors with the OpenFace library [53]. These descriptors indicated the presence of the AU and its associated intensity. After generating these features, they trained a stacked auto-encoder to compact the information and create an intermediate and more compact representation from the mentioned AUs. Then, they appended a fully connected layer to classify seven emotions. They reported a 96.5% classification rate at frame level on the RAVDESS dataset. However, they did not specify certain hyper-parameters and training settings necessary to replicate their experiments and compare proposals, like the learning rate or the cost function, among others.  
Aside from using hand-crafted or automatically generated features for emotion recognition, several deep models that work directly with facial images can also be found in the literature. One of these models is EmotionalDAN [54], a CNN-based model that addresses emotion, valence, and landmark recognition all at once. Deep-emotion model [55] is another example of this. Deep-emotion model uses an STN architecture [56] with an attention mechanism to address emotion recognition. Finally, the work of [PERSON]. [PERSON] et al. [57] employs an Xception model to perform facial emotion recognition.  
Since our target was to estimate emotion in videos rather than in frames (or images), we evaluated two strategies to give a final prediction on the whole video:  
The first proposal collapsed the sequence of AUs generated in each timestep into a single vector that was the average of all the temporal steps. These new features fed three distinct static models: an SVM, a k-NN, and an MLP.  
The second proposal employed a sequential model, i.e., a bi-LSTM with an attention mechanism to extract the video verdict from the sequence of AUs retrieved from each frame of the video.