## 3 Method and Experimental Setting
### Models  
The choice of using a deep learning model lays in the fact that state of the art for AFER is achieved by this approach. Most of the deep models used in research adopt convolutional layers, recurrent layers, or both. In order to investigate both these architectures, we develop two models, a convolutional neural network (CNN) and a gated recurrent unit (GRU) recurrent neural network (RNN), to classify the emotion expressed by a speaking subject in a video sequence. Each of these models has two modalities: the first is trained only on data relating to facial features which we will call the _face model_, and the second is trained both on data relating to facial features and viseme-related features which we will call the _lip model_. Finally, each of the models and the modalities is trained \(n\) times, each time on a different number of consecutive video frames provided in input. This approach provides an opportunity to answer not only the main question, whether the inclusion of viseme-related features in input improves accuracy for a speaking subject, but also to see the relationship between the accuracy and the temporal context provided as input to the model.  
In addition to the above, we develop a simple multilayer perceptron (MLP), taking as input only facial features for a single video frame and classifying the expressed emotion, which we will call the _frame model_. When presented with a video sequence in input, we evaluate the expression with the model frame-by-frame, and then get the mode of the predictions as the predicted class for the whole sequence. With this approach we replicate what is done in [30] by [PERSON] et al. with the purpose of using it as a baseline for the evaluation of the other models. The only difference to [30] is that we do not treat the _neutral_ expression as noise by leaving it out of the mode calculation but instead for completeness consider it a first-class citizen.  
To extract emotion-related facial features from video sequences to feed into our models, we adopt a pretrained model based on the VGG19 architecture (available at [[https://github.com/WuJie1010/Facial-Expression-Recognition.Pytorch.git](https://github.com/WuJie1010/Facial-Expression-Recognition.Pytorch.git)]([https://github.com/WuJie1010/Facial-Expression-Recognition.Pytorch.git](https://github.com/WuJie1010/Facial-Expression-Recognition.Pytorch.git))). The model was trained to classify the seven basic emotions: _anger_ (ANG), _disgust_ (DIS), _fear_ (FEA), _happiness_ (HAP), _sadness_ (SAD), _surprise_ (SUR) and _neutral_ (NEU), achieving 73.11% on the FER2013 dataset, and 94.64% on the CK+ dataset. We discarded the last layer of the model and took the resulting 512-dimensional feature vector as facial features to input in our models. On the other hand, we took the articulation features from LipNet [38]. Visual-only lip reading as a task is intimately tied to mouth configurations and dynamics in language articulation, and a model trained for lip reading will have learned their representations. We support this claim with the high accuracy of the mapping of LipNet outputs to visemes in [38]. While it would be possible to consider a different set of features to represent speech-related mouth configurations, e.g., including phoneme information, it would deviate from our visual-only approach, and we retain it would be a more indirect way of representing them. We discarded the bidirectional GRU layers and took the output of the last spatio-temporal CNN (STCNN) + spatial pooling layers, yielding a 96-dimensional feature vector representing mouth movement features. In line with the emotions that the facial features model is trained to predict, and in line with the available emotions in the dataset we use, all of the models we train predict the same seven basic emotions.