# Imaginary Speech Recognition Using a Convolutional Network with Long-Short Memory
## 3 Materials and Methods
### Feature Extraction  
The feature extraction stage aims to decode the information hidden in the signal and pass the result to a neural network capable of classifying the words and phonemes. In this paper, we used as features the inter-channel covariation in frequency domain. Let \(FX^{ch}\) be the fast Fourier transform (FFT) corresponding to the channel \(ch\) of the input signal \(X^{ch}\), the FFT can be described by the following equation:  
\[FX^{ch}(f)=\sum_{t=0}^{n-1}X^{ch}_{t}e^{-\frac{2\pi ft}{n}} \tag{1}\]  
The cross-covariation between the channels in frequency domain is given by:  
\[Cov\Big{(}X^{c1}(t),X^{c2}(t)\Big{)}=E\Big{[}\Big{[}\ X^{c1}(t)-E(X^{c1}(t) \Big{]}\Big{[}\ X^{c2}(t)-E(X^{c2}(t)\Big{]}\Big{]} \tag{2}\]  
where \(X^{c1}(t)\) represents the EEG signal acquired for channel \(c1\), \(X^{c2}(t)\) is the EEG signal acquired for channel \(c2\), and \(E[X^{ch}(t)]\) represents the mean value (where \(ch\) is corresponding to the specific channel \(c1\) or \(c2\)) and is computed as:  
\[E(X^{ch}(t))=\frac{1}{W}\sum_{i=0}^{W-1}x^{ch}_{i} \tag{3}\]  
The \(W\) value of Equation (3) corresponds to the window dimension for which the features are computed. After extracting the features, the resulting matrix for a signal will be of size [\(N_{ch}\times N_{ch}\)], where \(N_{ch}\) is the number of channels. The 0.25 s segmented channels were further cropped into 0.1 s window with 50% overlap to preserve the time evolution of the signal, which was passed to the CNN-LSTM architecture. An example of the computed features over 0.1 s window length for each phoneme and word is presented in Figure 1.  
Further, we investigated the features by computing the linear discriminant analysis (LDA) to reduce the features' dimensionality to only two components for visual inspection of the computed features. Figure 2 presents a comparison between the results obtained by applying LDA to the computed features cross-covariance in frequency domain (a), the raw signal in time (b), and computed features cross-covariance in time domain (c). It can be easily seen that, when using the frequency domain cross-covariance, the features are very well-structured into separate clusters, compared to the results obtained using the time domain signal or the time domain cross-covariance as features, c  
Figure 1: Cross-covariance in frequency domain for 0.1 s window for all phonemes and words from the KODB. overlap and the vectors are mixed in the same region of feature space. Another observation regarding the LDA results is the separability of the phonemes versus words. It can be seen that words and phonemes occupy different regions of the feature space. In Table 1, the mean and standard deviation of the LDA components for each class were computed to quantify the results observed after the visual analyses. The computed means and the confidence ellipse for each class are pointed out in Figure 3 for all analyses features: cross-covariance in frequency domain (a), raw-signal features (b), and covariance in time domain (c). In time domain, the means were very close to each other, and the standard deviation was high for all classes, each one incorporating a large part of the other classes for phonemes and for words. However, there was a clear differentiation between the words and the phonemes in the feature space. When it came to the cross-covariance in time domain, the distances between the means were getting higher, but the separability between the phonemes and words was lost, too. In the case of cross-covariance in frequency domain, the separability between the phonemes and words was being preserved, and the distances between the means were bigger. However, the classes were still not fully separable.