# Imaginary Speech Recognition Using a Convolutional Network with Long-Short Memory
## 4 Results
### CNNLSTM vs CNN  
This study focused on highlighting the advantages of the 2D CNNLSTM neural network in recognizing imaginary phonemes and words using the inter-channel cross-covariance method in the feature extraction stage. The results achieved were compared with the ones recorded in study [17], where the same feature extraction method was used with a CNN neural network as classifier. The results comparison can be observed in Table 3. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Accuracy** &
\begin{tabular}{c} **Balanced** \\ **Accuracy** \\ \end{tabular} & **Kappa** & **Recall** \\ \hline CNN [17] & 0.3758 \(\pm\) 0.004 & 0.3749 \(\pm\) 0.004 & 0.3132 \(\pm\) 0.004 & 0.3750 \(\pm\) 0.004 \\ \hline CNNLSTM [This study] & 0.4398 \(\pm\) 0.004 & 0.4392 \(\pm\) 0.004 & 0.3837 \(\pm\) 0.004 & 0.4398 \(\pm\) 0.004 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: The CNNLSTM neural network performances presented in comparison to CNN neural network, described in [17]. The CNNLSTM neural network managed to increase the accuracy from 0.3758 to 0.4398, according to Table 3, when the same preprocessing and feature extraction processing chain was followed. This difference in accuracy is due to the ability of the LSTM neural network to learn the long-term dependencies of time series in combination with the 2D CNN neural network, which encodes the spatial correlation between the electrodes too. The mean confusion matrix of the k-folds is presented in Figure 5 and reveals that there is hardly any confusion between the phonemes and words, as expected after the visualization of the LDA dimension reduction results in Figure 2.  
An analysis of the distances between the mean of each class for every phoneme and word was conducted in Tables 4 and 5, with the cross-covariance computed in frequency domain features (Table 4) and for the features computed by the CNNLSTM neural network after the first CNNLSTM layer (Table 5). This analysis aims to highlight the role of the CNNLSTM layers in  
To compute the Euclidean distance between the mean of each class, first the input matrix was reshaped into a matrix with the size N \(\times\) N\({}_{\text{features}}\), where N is the number of input vectors, and N\({}_{\text{features}}\) is the total number of features corresponding to N\({}_{\text{window}}\)\(\times\) N\({}_{\text{channels}}\)\(\times\) N\({}_{\text{depth}}\) for the cross-covariance matrix in frequency domain and N\({}_{\text{window}}\)\(\times\) N\({}_{\text{channels}}\)\(\times\) N\({}_{\text{channels}}\)\(\times\) N\({}_{\text{depth}}\) for the features computed by the first layer of the CNNLSTM neural network. All features were normalized in the range [0, 1], using Equation (12), before computing the Euclidean distances (13). \[{X_{ij}}^{norm}=\frac{{X_{ij}}-\underset{i}{\text{min}}{X_{ij}}}{{\underset{i} {\text{max}}{X-\underset{i}{\text{min}}{X_{ij}}}}} \tag{12}\]  
where \({X_{ij}}^{norm}\) is the input vector \({X_{ij}}\), \(i=\overline{0,\ N-1,j=\overline{0,\ N_{features}-1}}\), \(\underset{i}{\text{min}}{X_{ij}}\) is the minim value of the vector \(i\) for the feature \(j\), and \(\underset{i}{\text{max}}{X_{ij}}\) is the maximum value of the vector \(i\) for the feature \(j\). The Euclidean distance between the mean of utterance \(u1\), and utterance \(u2\) was computed as:  
\[D=\sqrt{\sum\limits_{j=0}^{N_{features}-1}\left({\mu_{u1,j}}-{\mu_{u2,j}}\right) ^{2}} \tag{13}\]  
where \({\mu_{u,j}}\) is equivalent to the mean of all utterance u, computed as:  
\[{\mu_{u}}=\frac{1}{{N_{u}}}\sum\limits_{i=0,\ {X_{i}}\in u}^{N_{u}-1}{X_{i}} \tag{14}\]  
It can be seen from Tables 4 and 5 that the Euclidean distances of the means increased significantly after the first convolutional layer. This growth highlights the important role of the convolutional layer in transforming the feature space into a space with a better separability between classes.