# A Semi-Supervised Lie Detection Algorithm Based on Integrating Multiple Speech Emotional Features
## 2 Method
### Feature Fusion  
In order to effectively utilize the dynamic time-frequency information in the speech signal, this paper adopts LSTM to process the log Mel spectrogram features \(X_{s}\) to obtain more advanced depth frame-level feature \(X_{sd}\). and processes the acoustic statistical features \(X_{a}\) through the self-coding network AE to extract the deep statistical features \(X_{ad}\) with stronger representations. Acoustic statistical features and Mel spectrogram features have a good complementary relationship, so this paper combines acoustic statistical features with Mel spectrogram features to capture finer-grained features. First, the depth statistical features \(X_{ad}\) and depth frame-level features \(X_{sd}\) are simply spliced and fused to form the A-S joint feature vector \(J=[X_{ad};X_{sd}]\in R^{d_{j}\times L}\), where \(d_{j}=d_{ad}+d_{sd}\), denotes the dimension of the A-S joint feature vector \(J\). Next, in order to obtain the semantic correlation between features and fully utilize the connection between similar features and different features, the combination of deep statistical features \(X_{ad}\), deep frame-level features \(X_{sd}\) and A-S joint feature vector \(J\) is considered to compute the joint correlation matrix. The joint correlation matrix of depth statistical features \(X_{ad}\) and A-S joint feature vector \(J\) is shown in Equation (5):  
\[C_{a}=\tanh(\frac{X_{ad}^{T}W_{jad}J}{\sqrt{d_{j}}}) \tag{5}\]  
where \(W_{jad}\) denotes the learnable weight matrix mapping the relationship between the depth frame-level features \(X_{ad}\) and the joint A-S feature vector \(J\). Similarly, the joint correlation matrix of the depth frame level features \(X_{ad}\) and the joint feature vector \(J\) can be obtained from Equation (6):  
\[C_{s}=\tanh(\frac{X_{sd}^{T}W_{jad}J}{\sqrt{d_{j}}}) \tag{6}\]  
where \(W_{jad}\) denotes the learnable weight matrix mapping the relationship between the depth frame-level features \(X_{sd}\) and the joint A-S feature vector \(J\). Then, the attention weights of acoustic statistical features \(X_{a}\) and log Mel spectrogram features \(X_{s}\) are further calculated to help the feature fusion model focus on the important information and improve the efficiency of the model when learning the distribution of similar features. The attention weights of acoustic statistical features \(X_{a}\) can be obtained by combining the joint correlation matrix \(C_{a}\) and the deep statistical features \(X_{ad}\), and the detailed calculation is shown in Equation (7). \[H_{a}=\text{ReLU}(\text{W}_{\text{ad}}\text{X}_{\text{ad}}+\text{W}_{\text{ ca}}\text{C}_{\text{a}}) \tag{7}\]  
where \(H_{a}\) denotes the attention weights of the acoustic statistical features \(X_{a}\), \(W_{a}d\) and \(W_{ca}\) is the learnable weight matrix. The attention weights of the log Mel spectrogram features \(X_{s}\) can be obtained by combining the joint correlation matrix \(C_{s}\) and the depth frame-level features \(X_{sd}\), and the detailed calculation is shown in Equation (6). \[H_{s}=\text{ReLU}(\text{W}_{\text{sd}}\text{X}_{\text{sd}}+\text{W}_{\text{cs} }\text{C}_{s}) \tag{8}\]  
where \(H_{s}\) denotes the attention weights of the acoustic statistical features \(X_{s}\), \(W_{s}d\) and \(W_{cs}\) is the learnable weight matrix. The attentional correlation features of the two types of features are computed by means of the attentional weights of the acoustic statistical features \(X_{a}\) and the log-Mel spectrogram features \(X_{s}\), with the formulas shown in (9) and (10) as follows. \[X_{att,a}=H_{a}W_{ha}+X_{ad} \tag{9}\]  
\[X_{att,s}=H_{s}W_{hs}+X_{sd} \tag{10}\]  
where \(W_{ha}\) and \(W_{hs}\) denote the learnable weight matrices for acoustic statistical features \(X_{a}\) and log Mel spectrogram features \(X_{s}\), respectively. Finally, the two types of attention-related features are combined to obtain the joint attention feature \(X_{att}=[X_{att,a}X_{att,s}]\).