Deep convolutional recurrent neural network for short-interval EEG motor imagery classification

1 INTRODUCTION . The use of Deep Neural Networks ( DNN ) for the challenging task of interpreting and classifying the noisy and idiosyncratic signals produced by electroencephalography ( EEG ) is beginning to show promising results . Some of the best performing models , such as EEGNet ( Lawhern et al. , 2018 ) and ShallowFBCSP net ( Schirrmeister et al. , 2017 ) , have been used in end-to-end solutions that require little or no signal pre-processing . In this article , we propose an architecture for the classification of Motor Imagery ( MI ) signals ; viz. , the EEG signals elicited when participants imagine ( rather than physically perform ) a motor action , a topic of interest due to its potential as an interface modality , particularly for those with motor impairments . Traditional approaches to EEG signal classification , such as Common Spatial Pattern ( CSP ) ( Pfurtscheller & Neuper , 2001 ) and Filter Bank Common Spatial Patterns ( FBCSP ) ( Ang et al. , 2008 ) , have the advantage that they can interpret the EEG signals elicited by each MI event in terms of their spatial correlations ( e.g. , by decomposing eigenvalues by channel ( Pfurtscheller & Neuper , 2001 ) ) . Unfortunately , although these approaches yield high performance in two-class MI classification ( e.g. , left vs right ) , achieving good performance in four-class classification has been elusive , and it therefore seems unlikely that they can underpin more sophisticated BCI systems . Furthermore , CSP-based approaches are not end-to-end solutions , requiring a number of intermediate stages to achieve high performances . For instance , CSP approaches are sensitive to noise ( Devlaminck et al. , 2011 ) and require a priori feature selection , either automatically or through careful analysis to identify critical features ( such as optimal frequency bands ) . Furthermore , potentially useful information ( in terms of increasing classification accuracy ) could be lost when the signal is decomposed into covariance matrices and eigenvectors that other approaches may be able to capitalize upon . Approaches using the Discrete Wavelet Transform ( DWT ) ( Naeem et al. , 2006 ; Yusoff et al. , 2018 ) also typically require pre-processing steps for the interpretation of MI events in the frequency domain to improve signal-to-noise ratio ( SNR ) , such as spatial/temporal filtering and artefact removal . However , combining DWT with other approaches has the potential to enable more sophisticated analyses to be performed , and may help elucidate the association between observed EEG signals and the specific events , such as MI actions , that generated them . Nevertheless , DWT still requires a careful selection of the mother wavelet . A dimensionality reduction process is also commonly used , or features are calculated as metrics of the decomposition coefficients , implying a loss of potentially useful information and representing an obstacle to the construction of an end-to-end solution . Furthermore , the DWT features that would lead to the highest classification performance are best detected using longer trial windows , representing a challenge for their use in real-time systems . DNN interpretability is an emerging field that enables researchers to examine network decisions . Techniques to accomplish this include gradient-weighted Class Activation Mapping ( grad-CAM ) ( Selvaraju et al. , 2017 ) and attention mechanisms , used previously in computer vision and natural language processing domains ( Vaswani et al. , 2017 ; Bahdanau et al. , 2014 ) . At the time of writing , grad-CAM has not been used for DNN-mediated MI classification , but has been used successfully in a small number of EGG signal classification studies using wavelet transformation features ( Andreotti et al. , 2018 ) . Furthermore , only a handful of studies have employed attention mechanisms for MI classification : for instance , Zhang and colleagues ( Zhang et al. , 2019 ) achieved classification accuracy comparable with state-of-art methods using a Convolutional Recurrent Attention Model ( CRAM ) that uses time domain signals and incorporates an attention mechanism . Data augmentation is a commonly used technique that aims to improve the accuracy of a trained model and reduce model overfitting by expanding the volume of training data . Most often , new training data are generated in the image domain by cropping , flipping , rotating , adding noise , or modifying color properties of already existing training data ( Cireşan et al. , 2010 ; Zeiler & Fergus , 2014 ) . In this article , we develop a similar approach to expand EEG training data and observe increased classification accuracy . We also experimentally determine the minimum time interval and optimal time window needed to confer state-of-the-art classification accuracy . 2 METHODS . 2.1 DATASET . The public BCI Competition IV 2A EEG dataset ( Brunner et al. , 2008 ) was used . These data were collected from 9 participants and have four-classes ( i.e. , four MI movements : left hand , right hand , feet , tongue ) . Participants completed 250 trials ( thus there were 2250 signals captured in all ) . Each trial was about 9s in duration . At the start of each trial , a fixation cross was presented ( t = 0 ) . After 2s , an arrow was presented to instruct participant which of the four imagined movements to perform ( arrow pointing left for left hand , right for right hand , down for feet , and up for tongue ) . Participants were instructed to maintain the imagined movement until the arrow disappeared ( 4s later , at t = 6 ) . Next , a random inter-trial break between 2s and 4s occurred , during which a blank screen was shown . Data were collected from 22 silver chloride ( Ag/AgCl ) electrodes , positioned in the international 10-20 system ( Klem et al. , 1961 ) . Electrodes were sampled at 250 Hz . Data were band-pass filtered from 0.5 Hz to 100 Hz and notch filtered at 50 Hz to eliminate noise generated from electrical interference . The 4s portion of each signal in which MI was performed was extracted for use here , giving 1000 samples per trial at the sample rate of 250 Hz . From here onwards , when using the term trial , we refer to the 4s period in which the movement was imagined only . The dataset was divided into a training ( 80 ) and validation ( 20 ) sets . A testing set , provided separately by the BCI competition organizers , was used to evaluate the generalizability of the model since it was unseen during training and validation . 2.2 APPARATUS . An NVIDIA GeForce GTX 1050 GPU was used for calculating model parameters ( stochastic gradient descent and model updating , see later ) . CUDA 8 and PyTorch ( Paszke et al. , 2019 ) were used for model implementations . 2.3 DATA REPRESENTATION . The proposed models were evaluated using the four-class BCI IV 2A Competition dataset ( described above ) . For each participant , raw EEG data were stored in three-dimensional tensor , X . The first dimension was trial t ∈ [ 1 .. T ] ( where T represents the total number of trials per participant , here 250 ) , the second dimension was electrode number e ∈ [ 1 .. E ] ( where E represents the total number of electrodes , here 22 ) , and the third dimension was sample s ∈ [ 1 .. Smax ] , where Smax represents the number of samples in the longest trial for that participant , up to 4×250 . A label vector y contained the true class for each trial , j ∈ [ 1 .. T ] , wherein each value was a member of set C = { 0 , 1 , 2 , 3 } corresponding to the four classes , left hand ( 0 ) , right hand ( 1 ) , feet ( 2 ) and tongue ( 3 ) , thus |C| = 4 . 2.4 AUGMENTATION BY SHUFFLING CROPS AND SLIDING WINDOWS . An augmentation technique that appears to be novel in BCI research was used . New trials were generated by taking two non-overlapping intervals of 500 samples ( 2s ) from each 1000 sample ( 4s ) trial for each participant and grafting them together in reverse order . This had the effect of doubling the volume of data available for training , validation , and testing . Furthermore , for the runs using two seconds samples ( Smax = 500 ) , the four seconds samples ( Smax = 1000 ) were divided into two-two seconds introducing double the number of trials . To simulate real-time signals , overlapped cropped intervals were generated from both the original and augmented 4s data signals . Two window sizes were tested , ω ∈ { 200 , 400 } samples . One forward jump ( stride ) of σ = 50 samples ( 0.2s ) was used . Intervals ( sub-vectors ) from within each trial for each participant were generated for the sequence of sample indices [ r .. r+w ] , where r starts at 1 and was increased by σ until no further complete intervals could be collected . These sub-vectors are referred to later as time-slices . For each configuration , P time slices were created ( Eq . 1 ) . P = [ Smax − ω σ ] ( 1 ) 2.5 CONVOLUTIONAL RECURRENT NEURAL NETWORK WITH TEMPORAL ATTENTION . ( CRNN-TA ) 2.5.1 CONVOLUTIONAL NEURAL NETWORK ( CNN ) BLOCKS . CNN Block 1 : First , a standard convolution layer that operates simultaneously on the samples at each time index from all electrodes was used . Its purpose is to find correlations between electrodes . A kernel size ofE×1 was used . After convolution , batch normalization was applied with momentum set to 0.993 . A Leaky ReLU activation function ( Maas et al. , 2013 ) was then used , a variant of ReLU that can deal with negative values ( with gradient constant , α , set to 0.01 ) . Next , an average pool with kernel size 1×3 and stride size 1×3 was added to reduce feature dimensionality . CNN Block 2 : Two replicas of the second block type were added to deepen the model , such that each block comprised separable convolutional layers ( Chollet , 2017 ) to reduce the number of trainable parameters . Batch normalization and Leaky ReLU were again added . Following the nonlinear activation function , an average pool layer with kernel size 1×3 and stride 1×3 was used . Finally a dropout layer with dropout probability p = 0.5 was added . This generated encoded features ( feature maps ) , which are to be denoted M . The CNN blocks operated as a time-invariant system , wherein different time-slice positions were treated in the same way . In other words , the features in each time slice were stacked on top of each other ; e.g. , the 4D input tensor ( of order T ×E × P × S ) was transformed to a 3D matrix ( of order TP × E × S ) . The encoded features are then transformed back for the RNN block ( see below ) to learn time-slices dependencies . 2.5.2 RECURRENT NEURAL NETWORK ( RNN ) GRU BLOCK . Two stacked Gated Recurrent Unit ( GRU ) layers with 64 hidden units were used after the CNN blocks described above , operating on the time-slices , such that the recurrent operation is applied between each second on the encoded features obtained from the CNN blocks . The 2D latent feature matrix ( of order P ×H ) is defined as F ∈ RPH where P is the number of time slices , and H is the number of units in each hidden latent representation . The output of the GRU contains the output for each time slice ( rows , P ) for each of the hidden units ( columns , H ) .

Summary: The authors introduce a deep learning approach for short-time motor imagery classification using EEG data.  Conventional CNN and RNN - GRU layers are used. Remarkably, a data augmentation strategy and a class-activation mapping approach are presented. Overall, the idea is interesting, but the paper presentation, the mathematical foundation, and the experiments provided are poor. Therefore, more details about the model and other experiments should be carried out to validate the proposal. Besides, the authors claim that short time interval EEG classification is achieved; however, 0.8s windows size does not seem to be a "short interval" compared to other state-of-the-art methods.