## 5 Conclusions  
We explore the problem of automatic facial expression recognition in speaking subjects. As language articulation lessens facial expression recognition accuracy we propose adding articulation-related features, extracted from a neural network trained to lip read, in input to a deep learning model.  
We devise two models: a spatio-temporal CNN and a GRU cell RNN. They are first trained on facial features only, and afterwards both on facial features and articulation features on the RAVDESS dataset. We find that including information related to language articulation improves accuracy up to 12%, but that the increase in accuracy is highly dependent on the number of consecutive frames  
Figure 11: From left to right, top to bottom, frames sequentially extracted in seven frame increments from the RAVDESS dataset of Actor 18 expressing _fear_ and pronouncing _“The kids are talking by the door.”_  
Figure 12: Plots of the performance of the 60 frame models on the test video illustrated in Figure 11. The horizontal axis shows the initial frame provided in inputs to the models, while the vertical axis measures the probability assigned by the models to a particular emotion class. On the left are the probabilities of _fear_ of the CNN lip model and the probabilities of _fear_ and _sadness_ of the CNN model. On the right are the probabilities of _fear_ of the RNN lip model and the probabilities of _fear_ and _sadness_ of the RNN model.  
provided in input. Accuracy gains were positive and linearly rose for input segments from 1 to 2.6 s in length.  
While the results are promising, RAVDESS is an acted dataset and in future works we intend to evaluate our approach on benchmark in-the-wild datasets, such as the AFEW 6.0, to see the accuracy improvement when confronted with spontaneous interactions, occlusions, and other challenges non-acted scenarios bring. In general, the still low accuracy of AFER in speaking subjects offers ample opportunity for research and improvement. It would be interesting to see how the upper area of the face is used for communicating nonverbal cues unrelated to affect, which to the best of our knowledge hasn't been addressed before. Also, human communication being a multimodal phenomenon, we consider the greatest potential for improvement in emotion recognition in speaking subjects laying in modelling multiple modalities, in addition to visual. Thus, modelling prosody, semantic, or textual information in addition to visual might further improve performance.  
Conceptualization, G.B. and A.F.; data curation, S.B. and A.D.; formal analysis, S.B., G.B. and A.F.; investigation, S.B. and A.D.; methodology, G.B., A.F. and R.L.; project administration, G.B. and A.F.; resources, G.B.; software, S.B. and A.D.; supervision, G.B., A.F. and R.L.; validation, S.B., A.D. and R.L.; visualization, A.D.; writing--original draft, S.B., G.B. and A.F.; writing--review and editing, S.B., G.B., A.F., A.D. and R.L. All authors have read and agree to the published version of the manuscript. This research received no external funding.  
This work has been partially supported by Fondazione Cariplo, through the project \"Stairway to elders: bridging space, time and emotions in their social environment for wellbeing\", grant no. 2018-0858.  
The authors declare no conflict of interest.