# Imaginary Speech Recognition Using a Convolutional Network with Long-Short Memory
## 2 State of the Art
The researchers processed the neural signals by extracting the multichannel high-gamma band as features that were decoded by assigning the neural activity to a mel-scaled audio spectrogram. Further, a regularized LDA-classifier was trained to predict the energy level for each spectral bin. The results reported an average 0.62 \(\pm\) 0.15 Pearson correlation between the reconstructed signal and the real speech signal, but the reconstructed speech was not yet intelligible. However, even if ECoG and sEEG offered a good representation of the neural activity with a high signal-to-noise ratio, they were still **invasive** methods of acquiring signals from the brain, which leads to great limitations regarding collecting the signals and being easily accepted by the patients. The alternative to these invasive methods can be functional magnetic resonance imaging (fMRI), functional near-infrared spectroscopy (fNIRS), magnetoencephalography (MEG), and electroencephalography (EEG). All these methods have their limitations: fMRI is expensive, non-portable, and has a poor temporal resolution, fNIRS takes time to collect the information from the brain (aprox. 2 to 5 s), which makes it harder to aim a real-time system, and MEG is expensive and non-portable. Finally, EEG remains the best option for a real-time noninvasive wearable device, while having the limitation of small amplitude data with a higher signal-to-noise ratio, comparing to ECoG. In recent years, several studies have used EEG signals for imaginary speech recognition. In study [8], the researchers analyzed six different words: \"could\", \"yard\", \"give\", \"him\", \"there\", and \"toe\", acquired from 15 subjects. In the feature extraction stage, the signals were decomposed using the Daubechies-4 (db4) mother wavelet to eight levels corresponding to the delta, theta, alpha, beta, gamma bands, and additionally, three other bands, <2 Hz, 64-128 Hz and 128-256 Hz, where features such as root mean square, standard deviation, and relative wavelet energy were computed. The features were fed into a random forest (RF) classifier and support vector machine (SVM). The registered results rose above the chance for both classifiers, having a mean accuracy for all 15 subjects of 25.26% RF and 28.61% SVM. A great breakthrough in the field came along with the open access datasets, such as the Kara One Dataset [9] and [PERSON] et al. [10] research dataset. This helped the community to develop research in the imaginary speech field, without having to go through the complicated process of data acquisition, and made it possible to compare the results with other studies developed based on the same environment. Using the Kara One Dataset, [PERSON] et al. [11] managed to develop a **patient-specific** system based on computing a set of statistical features, root mean square, variance, kurtosis, skewness, and the 3 rd order moment, over the obtained signals after decomposition using db4 mother wavelet into seven levels. Afterwards, the signals were fed into a deep learning architecture with two layers of 40 neurons. The results offered a 57.15% mean accuracy for all subjects, significantly higher compared to the results obtained without using deep learning. With a more complex deep learning architecture, the researchers from University of British Columbia managed, in their paper [12], to present a generalized system for imaginary speech **binary classification,** achieving the best accuracy of 85.23% for discrimination between consonant and vocal phonemes. The results were achieved using the covariance matrix between the channels as a feature matrix and a combination between convolutional neural networks (CNN), long-short term memory (LSTM) neural networks, and a deep autoencoder (DAE). In 2021, a larger dataset was collected in Moscow, Russia, containing signals acquired from 270 healthy subjects during the silent speech of eight Russian words: \"forward\", \"backward\", \"up\", \"down\", \"help\", \"take\", \"stop\", and \"release\".