# Detecting Deception from Gaze and Speech Using a Multimodal Attention LSTM-Based Framework
## 4 Discussion  
Results corroborate the two main hypotheses of this paper. The first hypothesis focuses on the importance of the modeling of the dynamics of gaze and speech feature sequences in the development of ADD systems, whereas the second one points out the complementary nature of the gaze and speech modalities for this task. Regarding the first hypothesis, the achieved results suggest that the temporal evolution of both, gaze and speech features, contain relevant information for the task of deception detection and for that, it is required to choose classification techniques able to properly model it, as is the case of LSTM networks. The success of the LSTM-based systems for this task also rely on the attention mechanism, that is able to automatically determine which frames of the gaze and speech sequences are most significant for the discrimination between truths and lies. As a consequence, it seems crucial to accentuate the contribution of the more relevant frames for the task regardless the modality used. The behavior of the attention mechanism is illustrated in the following examples. Figure 5a shows an instance of the gaze modality, where from top to bottom are depicted the FPOGX, FPOGY, MPD feature sequences and the corresponding gaze Attention-Pooling weights corresponding to a recording segment of 3 s labeled as deception. As a general observation, the gaze weights present a significant variation with time according to the importance that the attention procedure gives to each temporal frame. As can be observed, \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline  & & & \multicolumn{3}{c}{**Segment Level**} & \multicolumn{3}{c}{**Turn Level**} \\ \hline
**Modality** & **System** & **DACC** & **HACC** & **ACC** & **AUC** & **DACC** & **HACC** & **ACC** & **AUC** \\ \hline Late Fusion & SVM & 60.59 & 61.86 & 61.19 & 62.04 & 50.93 & 72.37 & 61.07 & 64.84 \\ Late Fusion & LSTM & 62.50 & 72.82 & 67.35 & 72.00 & 57.75 & 78.87 & 67.74 & 75.43 \\ \hline Early Fusion & SVM & 58.97 & 63.23 & 60.97 & 62.12 & 48.98 & 71.03 & 59.41 & 62.75 \\ AP Fusion & LSTM & 66.58 & 70.99 & 68.66 & 73.45 & 68.06 & 73.32 & 70.55 & 77.62 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Average deception accuracy (DACC) [%], honesty accuracy (HACC) [%], accuracy (ACC) [%] and Area-Under-the-Curve (AUC) at segment and turn level achieved by the SVM-based reference system and the LSTM-based system with the different fusion strategies. Figure 4: Mean and standard deviation of AUC [%] for both, SVM-based and LSTM-based systems, at (**a**) segment level and (**b**) turn level. large weights are assigned to the segment from 6.3 to 6.8 s where FPOGX and FPOGY present a large deviation from the mean value, which means that the user made a sharp eye movement. According to [5; 8; 9; 13], this kind of gaze dynamics are indicators of deception. In this slice, MPD shows a high degree of fluctuation, very likely due to the limitations of the eye-tracker for measuring the pupil size during an abrupt fixation change. For the same 3 s segment, Figure 5b depicts from top to bottom the waveform, log-mel spectrogram and Attention-Pooling weights for the speech modality. Again, the weight value of a specific frame is related to its relevance to the task according to the attention mechanism. In this case, as can be expected, larger weights are assigned to high energy speech frames. However, in silence regions, like the slice from 6.4 to 7.5 s, weights are smaller but greater than zero. This fact supports the hypothesis that pause characteristics are cues of lie detection, as pointed in [15]. Regarding the second hypothesis, results achieved by the fused systems show that gaze and speech carry complementary information about deceptive or honest behaviour. In order to highlight this fact, we have obtained the confusion matrices at turn level produced by the single-modality systems and the multimodal system with the AP Fusion strategy.