# Imaginary Speech Recognition Using a Convolutional Network with Long-Short Memory
## 2 State of the Art  
In recent years, recognizing silent speech from cortical signals attracted more and more attention of the researchers. Different approaches were tried over the years, aiming to achieve the best performances. The first attempts consisted of creating the desired words from letters using different methods of choosing the letters, such as moving a cursor on a monitor [1] or following a matrix of ASCII characters as the lines and columns are highlighted [2]. A successful real-time communication system, based on creating the words letter-by-letter, was developed by [PERSON] et al. [3] on a patient with amyotrophic lateral sclerosis (ASL). Due to the degradation of the motor functions, the only way of communication for the patient was by using brain signals. The researchers implanted a 64-microelectrode array, which allowed the subject to modulate neural firing rates after receiving audio feedback with a block of letters first and each letter from the block after to create words. After days of training, on the 245 th day after implantation, the patient was able to create complex sentences, such as, \"I would like to listen to the album by Tool loud\", with a mean of one character per minute. These methods offered significant results and have the advantage of improving over time as the subjects practice mastering the system. However, this type of communication is difficult, unnatural, and takes relatively a lot of time to form a word (one character per minute [3]). Other approaches targeted decoding the speech directly from the imagined words or phonemes. These methods assume that, during imaginary speech, the brain has different marks of electric activity, according to different pronunciations of each word. Trying to find different marks of imaginary speech, [PERSON] et al. [4] investigated the ECoG signals acquired during overt and imagined speech. In the conducted study, the researchers first tried to find similarities between overt and imagined speech. They computed the power spectrum of four frequency bands, theta (4-8 Hz), low-beta (12-18 Hz), low-gamma (25-35 Hz), and BHA (80-150 Hz), and they observed that the BHA power for both overt and imagined speech increased in the sensory and motor regions, while beta power decreased over the same regions. Nevertheless, when trying to differentiate different syllables in binary classes, articulatory, phonetic, and vocalic, the results for imaginary speech significantly dropped, compared to overt speech in the BHA band. Better performance was recorded when using power in the beta band, but with room for improvement. Significant improvements of the results were obtained when the trials were aligned on recorded speech onset, which was impossible to do for the imaginary speech. Other study based on ECoG signals managed to obtain important results when classifying five different words in a **patient-specific system**[5]. The researchers used high-gamma time features, which were aligned using dynamic time warping (DTW), and finally obtained a matrix feature by computing the DTW-distance between the realigned trials. These features, combined with support vector machine (SVM), offered encouraging results for a five-class classification task, obtaining a 58% mean accuracy for all five subjects. In 2019, [PERSON] et al. [6] managed to synthesize speech from **ECoG signals** by also using the **speech signal** acquired, together with the ECoG signal. The obtained results registered a correlation between the reconstructed speech signal and the actual speech, higher than the chance for all the analyzed subjects. Moreover, one of the subjects had a correlation of 0.69, which allowed for a good reconstruction of the vocal signal. Recent studies of stereoelectroencephalography (sEEG) signals regarding decoding the imaginary speech reported the development of a real-time system with speech synthesis for only one patient, due to the difficulty of data acquisition [7].