# Detecting Deception from Gaze and Speech Using a Multimodal Attention LSTM-Based Framework
## 5 Conclusions  
This paper deals with the development of an automatic deception detection system based on gaze and speech features. We present two main contributions. In the first one, we explore the use of attention LSTM networks for single-modal ADD systems with either frame-level gaze features (fixations and pupil diameter) or frame-level speech features (log-mel spectrograms) as input. In the second contribution, we present a multimodal ADD system where two fusion strategies for the combination of the gaze and speech modalities into the attention LSTM architecture have been tested, namely Late Fusion and Attention-Pooling Fusion. The proposed systems have been assessed over the Bag-of-Lies dataset, a multimodal database for deception detection recorded in real conditions. First, it can be observed that attention LSTM-based systems significantly outperform the performance of the reference systems that are based on traditional machine learning methods (in particular, SVM) running over compact feature representations. These results show that attention LSTM networks are able to properly modeling the dynamics of raw gaze and speech feature sequences that contains relevant cues of deception. Second, both combination strategies,  
Figure 6: Confusion matrices at turn level [%] for the LSTM-based system and three different cases: (**a**) Gaze; (**b**) Speech; (**c**) Multimodal system with AP fusion. Late and AP Fusion, achieve better results in terms of accuracy and Area-Under-the-Curve than the individual systems. In particular, our best system (Attention LSTM + AP Fusion) achieves a relative increase in AUC at turn level of 19.81% with respect to Attention LSTM + Gaze, 38.13% with respect to Attention LSTM + Speech and 36.35% with respect to the best multimodal SVM-based system (SVM + Late Fusion). Conceptualization, A.G.-A. and J.M.M. ; methodology, A.G.-A. and J.M.M. ; software, A.G.-A. ; formal analysis, A.G.-A. and J.M.M. ; investigation, A.G.-A. and J.M.M. ; data curation, A.G.-A. ; writing--original draft preparation, A.G.-A. ; writing--review and editing, A.G.-A. and J.M.M. ; funding acquisition, A.G.-A. and J.M.M. All authors have read and agreed to the published version of the manuscript. This research was partly funded by the Spanish Government-MinECo under Projects TEC2017-84395-P and TEC2017-84593-C2-1-R and Comunidad de Madrid and Universidad Carlos III de Madrid under Project SHARON-CM-UC3M. Ethical review and approval were waived for this study, because the dataset used is accessible under request for research purposes and the authors of this work were adhered to the terms of the license agreement of this dataset. Subject consent was waived because the dataset used is accessible under request for research purposes and the authors of this study were adhered to the terms of the license agreement of this dataset. In addition, no sensitive personal information was handled in this work. The multimodal database used in this paper is the Bag-of-Lies dataset that is available under request from [[http://iab-rubric.org/resources/BagLies.html](http://iab-rubric.org/resources/BagLies.html)]([http://iab-rubric.org/resources/BagLies.html](http://iab-rubric.org/resources/BagLies.html)), accessed on 3 July 2021. This database is available only for research and educational purpose and not for any commercial use. No new data were created in this study. The authors wish to acknowledge the Image Analysis and Biometrics (IAB) Lab @ IIT Jodhpur, India, for making the Bag-Of-Lies database available. The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results. The following abbreviations are used in this manuscript: