## 2 Background and Related Works
### Problems with the Speaker  
As stated from the beginning, one of the hindrances in replicating accuracies of the acted datasets on spontaneous interactions lays in the _speaking effect_[3]. One approach in the literature to alleviate the speaking effect, i.e., to increase accuracy of AFER on speaking subjects, rests on the assumption that emotional states and the pertaining facial cues are temporally more persistent than facial movements related to speech articulation. This approach has been adopted both by traditional approaches [8; 9],and by deep learning models that take as input single frames and output either vectors of features, or discrete emotion classes. The speech articulation-related facial movements are considered noise, and a statistic such as the mean or the mode is calculated on the features or classes seeking to cancel, smooth or filter out the \"noise\". In [30] the authors fine-tuned on the FERPlus dataset a ResNet50 variant pretrained on the VGG-Face2 dataset. On the resulting classifications they calculated the mode of the predictions while ignoring the _neutral_ expression and achieved 49.3% accuracy on the AFEW 6.0 dataset, thereby outperforming the baseline. Similarly, as an example of this approach applied to features, in [10] the authors extracted frame-level features with three diverse networks--VGG13, VGG16 and ResNet; and normalized the features by calculating the signed square root and performing \(\ell_{2}\) normalization and afterwards concatenating them. Then, over all the feature dimensions, the mean, the variance, the minimum and the maximum were calculated and fed into a support vector machine (SVM) classifier obtaining 59.42% on the AFEW 6.0 dataset. While the above filtering approaches provide a certain gain in classification accuracy, they also filter out potentially useful information for emotion classification, thereby leaving room for further improvements in accuracy. On the other hand, a different approach would be to directly model the information relating to expressing emotion and that relating to speech articulation. However, works pursuing this approach are few and far between. In [31] [PERSON] et al. used asymmetric bilinear factorization to perform the decoupling of linguistic and affective information. They build on their previous work [32] that models facial expressions as having three fundamental components: speaker, lexical and emotional. In [32], by using factor analysis, they demonstrate the dominant influence of speech articulation on the orofacial features. However, their model depends on having the speech transcription to compensate for the lexical information. In [31] they rectify that dependency, achieving 60.25% on the IEMOCAP [33] dataset with four discrete emotions on a baseline of 53.8%. [PERSON] et al. [34] developed an approach for removing the speaking effect using eigenface conversion, whereby they convert speaking faces in non-speaking faces by compensating for the articulation effect in the mouth area. Finally, a somewhat related problem is lip reading, the connection being that a model that performs visual-only lip reading will model speech articulation itself by definition. Prior to the advent of deep learning, most lip reading techniques were based on hand crafted features and were modeled by hidden Markov models [35]. However, lip reading, as did many other problems, experienced a quantum leap in recent years thanks to both the emergence of deep learning techniques and the availability of large datasets [36]. Deep learning is employed in lip reading for either feature extraction, upon which a classifier is applied, or as an end-to-end system. The models also differ in whether they model only visual information, or both audio and visual, with the latter providing greater accuracy [37]. A particularly interesting model is LipNet [38] by [PERSON] et al., an end-to-end, visual information only model that maps a variable-length sequence of video frames to text, making use of spatio-temporal convolutions followed by recurrent layers.