# Detecting Deception from Gaze and Speech Using a Multimodal Attention LSTM-Based Framework
## 2 Materials and Methods
### Deception Detection Systems
#### 2.3.2 Single-Modal Attention LSTM-Based Add System  
As the dynamics of gaze (FPOGX, FPOGY and MPD) and speech features (log-mel spectrograms) contain relevant information about the deception behaviour of a subject, our proposal is to develop a common framework for both modalities based on attention Long Short-Term Memory networks that are known to have the ability to learn long-term dependencies by storing information from the past in their memory blocks [23; 24]. LSTMs perform a sequence-to-sequence learning where an input sequence of length \(T\), \(X=\{x_{1},\ldots,x_{T}\}\) is transformed to an output sequence \(Y=\{y_{1},\ldots,y_{T}\}\) of the same length. ADD is a many-to-one problem in the sense that a single label (truth or lie) must be assigned to the whole input segment. For this reason, usually the LSTM sequence \(Y\) is summarized in a single value \(Z\), which can be seen as a segment level representation, prior to be fed to the classifier itself by means of a certain pooling mechanism [29; 42]. In this work, we adopted the attention pooling strategy as it has been successfully used in other classification problems involving the modeling of temporal sequences [26; 27; 28; 29; 30; 31; 32; 33; 34]. The hypothesis behind this approach was that certain LSTM frames contained more cues about the task under consideration than other ones. Therefore, frames conveying more useful information for the discrimination between truths and lies should be emphasized and their contribution to the \(Z\) computation should be higher, whereas non-relevant frames should be diminished or even ignored. This way, the segment level representation \(Z\) is computed as the weighted arithmetic mean of the output LSTM frames as follows,  
\[Z=\sum_{t=1}^{T}\alpha_{t}y_{t} \tag{1}\]  
where \(\alpha=\{\alpha_{1},\alpha_{2},\ldots,\alpha_{T}\}\) is the attention weight vector, and the value of each weight is related to the importance of the corresponding frame for deception detection. The weights were obtained following the technique proposed, among others, by [30], that was adequate for scenarios with scarce training data, as is our case. According to this approach, the un-normalized attention weights, \(\bar{\alpha}=\{\bar{\alpha_{1}},\bar{\alpha_{2}},\ldots,\bar{\alpha_{T}}\}\), are calculated through the following Equation (2). \[\bar{\alpha_{t}}=u^{\text{tr}}y_{t}, \tag{2}\]  
where \(u\) is the attention parameter vector and the superscript \(tr\) denotes a transpose operation. The attention parameters and the LSTM outputs were jointly learnt during the training of the network. Finally, these weights were normalized for guaranteeing that their sum across all the frames of the sequence is equal to one. For doing this, a softmax transformation was applied, yielding a set of normalized weights \(\alpha=\{\alpha_{1},\alpha_{2},\ldots,\alpha_{T}\}\), as indicated in Equation (3). \[\alpha_{t}=\frac{e^{\alpha_{t}}}{\sum_{k=1}^{T}e^{\alpha_{k}}} \tag{3}\]  
Figure 2 depicts the block diagram of the attention LSTM system proposed for ADD that used either gaze features or log-mel spectrograms as input. In both cases, mean and standard deviation normalization were applied at segment level. As can be observed, it was composed of two main stages: LSTM and Fully Connected modules. The LSTM module consisted of a Masking layer that makes sure the dummy values of the padded sequences were not used in further computations (see Sections 2.2.1 and 2.2.2) and an attention LSTM layer with 32 hidden units and a dropout of 0.25 in order to avoid overfitting in the training process. As well, the attention parameter vector \(u\) had a dimension of 32. The Fully Connected module was composed of a dense layer of 50 neurons, and final dense layer with 1 node that was activated by a sigmoid function for performing the binary classification. The only difference between the gaze and speech-based systems was the type of feature vectors used as input and the length of the LSTM input/output sequences \(T\).