Deep convolutional recurrent neural network for short-interval EEG motor imagery classification
Authors: Ahmed Bahaa Selim
Ian van der Linde
Venue: ICLR 2022 Submitted (2022)
Keywords: Attention
Brain-Computer Interface (BCI)
Electroencephalography (EEG)
Convolutional Neural Networks (CNN)
Motor Imagery (MI)
Recurrent Neural Networks (RNN)
grad-CAM

Abstract: In this paper, a high-performance short-interval motor imagery classifier is presented that has good potential for use in real-time EEG-based brain-computer interfaces (BCIs). A hybrid deep Convolutional Recurrent Neural Network with Temporal Attention (CRNN-TA) is described that achieves state-of-art performance in four-class classification (73% accuracy, 60% kappa, 3% higher than the winner of the BCI IV 2A competition). An adaptation of the guided grad-CAM method is proposed for decision visualization. A novel EEG data augmentation technique, shuffled-crossover, is introduced that leads to a 3% increase in classification accuracy (relative to a comparable baseline). Classification accuracies for different windows sizes and time intervals are evaluated. An attention mechanism is also proposed that could serve as a feedback loop during data capture for the rejection of bad trials (e.g., those in which participants were inattentive).

Under review as a conference paper at ICLR 2022
DEEP CONVOLUTIONAL RECURRENT NEURAL NET- WORK FOR SHORT-INTERVAL EEG MOTOR IMAGERY CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, a high-performance short-interval motor imagery classifier is pre- sented that has good potential for use in real-time EEG-based brain-computer in- terfaces (BCIs). A hybrid deep Convolutional Recurrent Neural Network with Temporal Attention (CRNN-TA) is described that achieves state-of-art perfor- mance in four-class classification (73% accuracy, 60% kappa, 3% higher than the winner of the BCI 2A competition). An adaptation of the guided grad-CAM method is proposed for decision visualization. A novel EEG data augmentation technique, chuffled-crossonee cation accuracy (relative to a comparable baseline). Classification accuracies for different windows sizes and time intervals are evaluated. An attention mechanism is also proposed that could serve feedback loop during data capture for the rejection of bad trials (e.g., those in which participants were inattentive).
is introduced that leads to a 3% increase in classifi-
INTRODUCTION
The use of Deep Neural Networks (DNN) for the challenging task of interpreting and classifying the noisy and idiosyncratic signals produced by electrenncephalograpyy (EEG) is beginning to show promising results. Some of the best performing models, such as EEGNet (Lawhern et al. (2018) and ShallowFBCSP (Schirrmeister al. 2017). have been used in end-to-end solutions that require little no signal pre-processing.
In this article, we propose an architecture for the classification of Motor Imagery (MI) signals; viz, the EEG signals elicited when participants imagine (rather than physically perform) a motor action, a topic of interest due to its potential as an interface modality, particularly for those with motor impairments.
Traditional approaches to EEG signal classification, such as Common Spatial Pattern (CSP) PPurtscheller & Neuper 2001) and Filter Bank Common Spatial Patterns (FBCSP) (Ang et al. 2008). have the advantage that they can interpret the EEG signals elicited by each MI event in terms of their spatial correlations (e.g., by decomposing eigenvalues by channel (Pfurtscheller & Neuper 2001]) Unfortunately. although these approaches yield high performance in two-class MI classifica- tion (e.g.. left vs right), achieving good performance in four-class classification has been elusive, and it therefore seems unlikely that they can underpin more sophisticated BCI systems. Furthermore, CSP-based approaches are end-to-end solutions, requiring number of intermediate stages to achieve high performances. For instance, CSP approaches are sensitive to noise (Devlaminck et al., 201I) and require a priori feature selection, either automatically or through careful analysis to iden- tify critical features (such optimal frequency bands). Furthermore, potentially useful information (in terms of increasing classification accuracy) could be lost when the signal is decomposed into covariance matrices and eigenvectors that other approaches may be able to capitalize upon.
Approaches using the Discrete Wavelet Transform (DWT) (Naeem et al., 2006 Yusoff et al.) 2018) also typically require pre-processing steps for the interpretation of MI events in the frequency do- main to improve iignal-oonoiise ratio (SNR), such as spatial/temporal filtering and artefact removal. However, combining DWT with other approaches has the potential to enable more sophisticated analyses to be performed, and may help elucidate the association between observed EEG signals and the specific events, such as MI actions, that generated them. Nevertheless, DWT still requires

---

Under review as a conference paper at ICLR 2022
careful selection of the mother wavelet. A dimensionality reduction process is also commonly used, or features are calculated as metrics of the decomposition coefficients, implying a loss of poten- obstacle to the construction of an end-to-end solution.
tially useful information and representing Furthermore, the DWT features that would lead to the highest classification performance are best detected using longer trial windows, representing challenge for their in real-time systems.
DNN interpretabillty is an emerging field that enables researchers to examine network decisions. Techniques accomplish this include gradient-weighted Class Activation Mapping (grad-CAM) (Selvaraju et al. 2017) and attention mechanisms, used previously in computer vision and natural language processing domains (Vaswani et al., 2017] Bahdanau et al.) 2014). At the time of writing, grad-CAM has not been used for DNN- MI classification, but has been used successfully in small number EGG signal classification studies using wavelet transformation features (An- dreotti et al.] 2018). Furthermore, only handful of studies have employed attention mechanisms for MI classification: for instance, Zhang and colleagues (Zhang et al., 2019) achieved classification accuracy comparable with state-of-art methods using Convolutional Recurrent Attention Model (CRAM) that uses time domain signals and incorporates an attention mechanism.
Data augmentation is a commonly used technique that aims to improve the accuracy of trained model and reduce model overfitting by expanding the volume of training data. Most often, new training data are generated in the image domain by cropping, flipping, rotating, adding noise, or modifying color properties of already existing training data (Ciresan et al. 2010 /Zeiler & Fergus) 2014). In this article, we develop a similar approach expand EEG training data and observe increased classification accuracy. We also experimentally determine the minimum time interval and optimal time window needed to confer state-of-the-art classification accuracy.
METHODS
2.1 DATASET
The public BCI Competition IV 2A EEG dataset (Brunner et al. 2008 was used. These data were collected from 9 participants and have four-classes (i.e., four MI movements: left hand, right hand, feet, tongue). Participants completed 250 trials (thus there were 2250 signals captured in all). Each trial was about 9s in duration. At the start of each trial, a fixation cross was presented (t = 0)- After 2s, an arrow was presented to instruct participant which of the four imagined movements to perform (arrow pointing left for left hand, right for right hand, down for feet, and up for tongue). Participants were instructed to maintain the imagined movement until the arrow disappeared (4s later, at 6). Next, a random inter-trial break between 2s and 4s occurred, during which blank screen was shown. Data were collected from silver chloride (Ag/AgCI) electrodes, positioned in the international 10-20 system (Klem et al. 1961). Electrodes were sampled at 250 Hz. Data were band-pass filtered from 0.5 Hz to 100 Hz and notch filtered 50 Hz to eliminate noise generated extracted
from electrical interference. The 4s portion of cach signal in which MI was performed for use here, giving 1000 samples per trial at the sample rate of 250 Hz. From here onwards, when using the term trial, refer to the 4s period in which the movement was imagined only. The dataset was divided into a training (80) and validation (20) sets. A testing set, provided separately by the BCI competition organizers, was used evaluate the generalizability of the model since was unseen during training and validation.
2.2 APPARATUS
An NVIDIA GeForce GTX 1050 GPU was used for calculating model parameters (stochastic gra- dient descent and model updating, see later). CUDA 8 and PyTorch (Paszke et al. 2019) were used for model implementations.
2.3 DATA REPRESENTATION
The proposed models were evaluated using the four-class BCI IV 2A Competition dataset (described above). For each participant, raw EEG data were stored in dinee-iimeniional tensor, X. The first dimension was trial E [1.TT] (where T represents the total number of trials per participant, here 250), the second dimension electrode number [1..F] (where F. represents the total number
2

---

Under review as a conference paper at ICLR 2022
of electrodes, here 22), and the third dimension was sample E [1..Smaz]: where Smaz represents the number of samples in the longest trial for that participant, up to 4x250. A label vector contained the true class for each trial, j E [1.TT, wherein each value corresponding to the four classes, left hand (0), right hand (1), feet (2) and tongue (3), thus [C] = 4.
member of set C' {0, 2.
2.4 AUGMENTATION BY SHUFFLING CROPS AND SLIDING WINDOWS
An augmentation technique that appears to be novel in BCI research was used. New trials were generated by taking two non-overlapping intervals of 500 samples (2s) from each 1000 sample (4s) trial for each participant and grafting them together reverse order. This had the effect of doubling the volume of data available for training, validation, and testing. Furthermore, for the runs using two seconds samples (Smaz = 500), the four seconds samples (Smes = 1000) were divided into two-two seconds introducing double the number of trials. To simulate real-time signals, overlapped cropped intervals were generated from both the original and augmented 4s data signals. Two window sizes were tested, w E {200, 400) samples. One forward jump (stride) of 50 samples (0.2s) was used. Intervals (sub-vectors) from within each trial for each participant were generated for the sequence of sample indices [r..r+w], where starts and increased by a until no further complete intervals could be collected. These sub-vectors are referred to later as time-slices. For each configuration, P time slices were created (Eq. 1).
S Smaz w
P
(1)
2.5 CONVOLUTIONAL RECURRENT NEURAL NETWORK WITH TEMPORAL ATTENTION (CRNN-TA)
2.5.1
CONVOLUTIONAL NEURAL NETWORK (CNN) BLOCKS
CNN Block 1: First, standard convolution layer that operates simultancously the samples at each time index from all electrodes was used. Its purpose is to find correlations between electrodes. A kernel size of Ex 1 was used. After convolution, batch normalization was applied with momentum set to 0.993. A Leaky ReLU activation function (Maas et al.) 2013) was then used, a variant of ReLU that can deal with negative values (with gradient constant, a, set to 0.01). Next, average pool with kernel size 1x3 and stride size I 1x3 was added to reduce feature dimensionality.
CNN Block Two replicas of the second block type were added deepen the model, such that each block comprised separable convolutional layers (Chollet) 2017) to reduce the number of trainable parameters. Batch normalization and Leaky ReL.U were again added. Following the nonlinear activation function, an average pool layer with kernel size 1x3 and stride 1x3 was used. Finally dropout layer with dropout probability p = 0.5 was added. This generated encoded features (feature maps), which are to be denoted M.
The CNN blocks operated as time-invariant system, wherein different time-slice positions were treated in the same way. In other words, the features in each time slice were stacked on top of each other; c.g., the 4D input tensor (of order T x E x x S) was transformed to 3D matrix (of order TP x E x S). The encoded features are then transformed back for the RNN block (see below) to learn time-slices dependencies.
2.5.2 RECURRENT NEURAL NETWORK (RNN) GRU BLOCK
Two stacked Gated Recurrent Unit (GRU) layers with 64 hidden units were used after the CNN blocks described above, operating on the time-slices, such that the recurrent operation is applied between each second on the encoded features obtained from the CNN blocks. The 2D latent feature matrix (of order P x H) is defined F E PH where P is the number of time slices, and H is the number units in each hidden latent representation. The output of the GRU contains the output for each time slice (rows, P) for each of the hidden units (columns, H).
3

---

Under review as a conference paper at ICLR 2022
2.5.3 EXCITATION AND SQUEEZE TEMPORAL ATTENTION
Since participants were asked to maintain each imaginary movement for 4s over 250 trials, their ability to concentrate may have fluctuated from trial-to-trial. Consequently, data in which partici- pants had lost focus will inevitably feature in the dataset. In practice, providing feedback notify participants whose concentration wanes during training and then labelling (and potentially discard- ing) data from these trials may improve compliance and enable training data quality to be improved. To reduce the contribution of data corresponding to periods of lapsed concentration, a temporal at- tention mechanism was used based upon squeeze and excitation (Hu et al., 2018). However, the operation was reversed in relation to the way it is normally used, and is therefore better described as excitation and squeeze. The proposed method requires only around 800 learnable parameters, compared with 17,000 in CRAM (Zhang et al. 2019). The average for each time-slice, 2;, then calculated (Eq. 2).
H Σ
2 H
ftg,t=1..P
Next, 2 (of order P x 1) was excited using nonlinear ReLU activation function applied (Eq. 3), where W1 a matrix of order 64 containing learnable weights (i.e., mapping to 64 neurons).
= RcLU
The vector produced,z', is of order 1x64. To acquire an attention vector for each time-slice, and learn nonlinear dependencies between time-slices Æ' is squeezed using a sigmoid activation function (Eq. 4) where W22 a weight matrix of order 64 P.
sigmoid( (z'W2)
The attention vector, a (of order x P), is multiplied element-wise by each of the H columns of F' (where each column is of order P x 1) to emphasize the contribution of the features in of each time-slice to the model decision, rescaling the hidden features to create F (Eq. 5, where o denotes Hadamard product).
FL.9. = d' o FI.porJ = 1..
(5)
2.5.4 FULLY CONNECTED LAYER
The output of the attention mechanism, Eq. 5, was then flattened form an input vector that is connected to C' output neurons, where CC) is the number of classes. A sof fmaa activation function (Eq. 6) was then applied the output neurons for classification, where W3 is a weight matrix of order 64P x |C|.
i softmar FF
2.6 TRAINING, TESTING AND VALIDATION
A cross-entropy function was used to calculate the loss of the model. During training, the model attempts minimize loss and the model was penalized when the probability of the predicted class differed from the true class, Binary cross-entropy is defined shown in Eq. 7.
Loga (p) + (1 - 3) logg (1-p)
Where y is the true class (binary or 1) and p is the probability associated with the class prediction. Where multi-class classification is required, loss L is calculated for each class i, where each is a value e E C, separately and losses are summed to obtain single loss value (Eq. 8) for each observation.
4

---

Under review as a conference paper at ICLR 2022
ICI
L - -= - Mr log2
(8)
The weights and biases were updated using batch gradient descent. Batch size was set to 20, thus weights were updated 20 times per epoch (see below), with the total loss being the average loss over 20 trials. The final predicted class label, Ds. is calculated as the class with the maximum probability. Adam (Kingma & Ba 2014), was used for weight optimization and updating, with weight decay set to 0.4 to introduce 1.2 regularization (Krogh & Hertz) 1992). A total of 800 epochs were used for training. The reported accuracies are the mean of five runs, similar to five-fold cross validation in which the training samples are non-overlapped and randomly selected for each run. Accuracies were calculated using the unseen testing set.
2.7 MODIFIED GUIDED GRAD-CAM
Selvaraju and colleagues (Selvaraju et al. 2017) presents generalization of Class Activation Map- ping (CAM) (Zhou et al. 2016 that is usable for any CNN-based architecture. These visualization techniques were initially developed to identify image regions that contribute to model decisions. However, whilst image features may be readily interpretable, where EEG signals are used, visual inspection will not be meaningful. Consequently, the approach used here has two components. First, signal intervals that contribute significantly to the classification of a specific class are approximately localized, enabling them be used in feedback loop when the model is being trained. Secondly, identifying the spatial correlation (the relationship) between the electrodes may be exploitable to minimize the number of electrodes required in real life (c.g., to simplify BCI hardware). To accom- plish this, some modifications to the standard method are required.
In the general definition of CAM, the last convolutional layer before the softmaz in any CNN based architecture generates K feature maps, M R""" where u (rows) and v (columns) represent the activated units in a 2D plane indexed by and j. e.g. M is the activated unit location (i.j) of the htth feature map. Global Average Pooling (GAP) is applied to M followed by a linear transformation obtain weighed score Y for each class c, and W4 is a weight matrix of order K |C]. For ease of reading we will refer to each weight connecting tth feature map with ch class as w (Eq. 9).
YO
M
km
j=
In the proposed model, the spatial correlation between electrodes encoded in the first convolutional layer. resulting in ID matrix with v samples for each feature map ls (collapsed dimension after convolving over electrodes). In order to preserve the electrode information and apply Grad-CAM to the EEG signals per electrode (1D) matrix). instead of using GAP, an average pool was applied on M so the CAM can now be represented (Eq. 10).
yc Σ
M
(10)
To make the CAM weighted by gradients, instead of the final layer learned weights [19], the gradient of y is calculated with respect to the feature maps M (Eq. 11).
OY
of = 7 1=1 OM
(11)
Where the partial linearization of obtained by average pooling the gradients acquired through back- book-propagating from the feature maps of the target class Finally, the heatmaps generated are defined as the combination of the feature maps and are also followed by a ReLU to acquire only the features that contribute positively (i.e., more important electrodes). Eq. 12:
5

---

Under review as a conference paper at ICLR 2022
Table I: Independent participant's classification accuracy in four different configurations (windows size, stride, number of samples). Baseline column without excitation and squeeze or augmentation. Proposed method column using excitation and squeeze with data augmentation. Reported accuracies on unseen test data from BCI IV 2A. The four seconds samples Smar were divided into two-two seconds for the proposed method.
w
0
#Samples
Baseline Mean Acc (SD) CRNN-TA Mean Accuracy SD
400
50
1000
0.68 (0.01)
200
50
1000
0.65 (0.01)
400
50
500
0.65 (0.01)
0.70 (0.12) 0.73 (0.08)
200
50
500
0.65 (0.01)
Table 2: A comparison between the best performing method from BCI Competition IV 2A, measured Cohen's Kappa (s), CRAM, and the proposed model (last column, labelled CRNN-TA). For each of the participants, the best preforming model is highlighted in bold.
Participant 1st Place
2nd Place
3rd Place
CRAM
CRNN-TA
0.68
0.69
0.38
0.69
0.70
0.42
0.34
0.18
0.29
0.34
3
0.75
0.71
0.48
0.68
0.79
4
0,48
0,44
0.33
0.34
0.51
0.40
0.16
0.07
0.09
0.45
6
0.27
0.21
0.14
0.30
0.39
0.77
0.66
0.29
0.57
0.80
8
0.75
0.73
0.49
0.49
0.72
0.61
0.69
0.44
0.56
0.66
Mean
0.57
0.52
0.31
0.45
0.61
k Σ
Le Lorad-CAM) ReLU
a M²
(12)
The heatmap generated for the first convolutional layer, Ligrad- CAyy will have the size as the feature map (i.e., 22x400 in the suggested model). After applying the convolution operation the spatial layer (with kernel size 22x1), the feature maps become ID (Ix400).
RESULTS
Results for each configuration are shown in Table 1. For the baseline, no statistically significant dif- ference in mean accuracy between the four configurations was found: F(3, 241) 0.92, p 0.01. However, the configuration w 400, 50%, S 1000 yielded 68% average accuracy, 3% higher than the remaining groups. Between figurations, statistically significant difference in mean accuracy between the four configurations was found: F(3, 32) = 0.50, p > 0.01. The final configuration (w = 200, a 50, S 500) yielded the highest accuracy of 73%, an improvement of +5% compared to the best baseline score +8% over the same configuration in the baseline.
suggested model and the baseline for the bottom two con-
The suggested model has the highest performance as shown in Table 2 where it outperforms the 1st place winner by 4% and CRAM by 6%.

---

Under review as a conference paper at ICLR 2022
Excitation and Squeeze
60
Time slice: 1
Time slice: 2
Time slice: 3
50
Time slice:
Time
Time slice: 6
40
Density
30
10
0
0.6
0.8
Attention weight
Excitation and squeeze density plot showing weight distribution across all trials for par- ticipant 3 with number of time slices P = 6 where attention weights (x-axis) were calculated using Eq. and y-axis represents the density the distribution
Figure 1:
008
Channes
14 12 11
Samples
Figure 2: Example heatmap produced by the guided grad-CAM looking the first convolutional
layer, corresponding to participant 3 imagining a right hand movement (w = 400 and S = 500); note that the features were encoded in time invariant manner discussed earlier

---

Under review as a conference paper at ICLR 2022
4 DISCUSSION
In this study, two different window sizes (w = 200 and w = 400) using either 5 = 500 or S = 1000 (2s or 4s intervals) were supplied to the model for training and testing, and results reported are for the unseen test data. The model achieved mean exceeds state-of-art CRAM (x 0.45), and the winner of BCI IV 2A competition (x 0.57), as summarized in Table 2. The baseline yielded identical performance using 200 or = 400 with 50 and S 500 (Table Baseline), implying that classification could be performed twice as quickly since it used half as much data (i.e., window size 400 over a 4s interval would require a delay of 4s and an input of 1.6s input per one step of classification, compared with a delay and 0.8s input per one step of classification), representing slightly better than the current real-time EEG based classification applications. A further advantage of shorter windows and intervals is that less memory is required to store the signals, further increasing the potential for its use in real-time or embedded BCIs. The data augmentation method proposed, in which extra training samples were generated by exchanging the first and last 2s intervals (note: this was not done to the final testing set used to evaluate the model's performance) led to the highest classification accuracy, with a mean of 70% for the configuration w 400,
0.60 = 0,02, representing performance that
50, S 500 and 73% for configuration w 200, = 50, 5 = 500, suggesting that further expanding this augmentation technique could potentially further increase performance without requiring that more data is collected/used for training. The attention mechanism proposed (excitation and squeeze, Fig. 1) provided an informative time slice interpretation that highlights the contribution of each slice to the model's 's decision. The attention weights observed seem to indicate greater attention density for the latter three time slices (4..6). This approach is therefore potentially useful in architectures with a recurrent layer, since is (relatively) computationally inexpensive, with only 800 learned parameters learned, and the attention vectors produced are potentially useful in a feedback loop. Guided grad-CAM heatmaps (Fig. 2) show higher contribution of selected channels between the four classes, and highlight which samples contribute more significantly to the model's decision. The guided grad-CAM approach proposed seems to offer positive preliminary results but further investigation is needed to validate the outcome of the grad-CAM (e.g., by removing the channels that contributed least) since it is not possible to verify visually.
REFERENCES
Fernando Andreotti, Huy Phan, and Maarten De Vos. Visualising convolutional neural network de- cisions in automatic sleep scoring, In CEUR Workshop Proceedings, pp. 70-81. CEUR Workshop Proceedings, 2018
Kai Keng Ang. Zheng Yang Chin, Haihong Zhang, and Cuntai Guan. Filter bank common spatial pattern (fbcsp) in brain-computer interface. In 2008 IEEE international joint conference on neural networks (IEEE world congress on computational intelligence). pp. 2390-2399 2397. IEEE, 2008
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: arXiv:/409. 409.0473, 2014.
Clemens Brunner, Robert Leeb, Gernot Mtiller-Putz. Alois Schlögl, and Gert Pfurtscheller. Bci com- petition 2008-graz data set a. Institute for Knowledge Discovery (Laboratory of Computrrrr Interfaces), Graz University of Technology, 16:1 2008
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
Dan Claudiu Cireşan, Ueli Meier, Luca Maria Gambardella, and Jürgen Schmidhuber. Deep, big, simple neural nets for handwritten digit recognition. Neural computation, 22(12):3207-3220. 2010.
Dieter Devlaminck, Bart Wyns, Moritz Grosse Wentrup, Georges Otte, and Patrick Santens. Mul- tisubject learning for common spatial patterns in motor-imagery bei. Computational intelligence and neuroscience, 2011, 2011.
Jie Hu, Li Shen, and Gang Sun. soneezz-andd-xxcttttiee networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018
8

---

Under review as a conference paper at ICLR 2022
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
GH Klem, HO Luders, HH Jasper, and C Elger. The ten twenty electrode system: international federation of societies for electroencephalography and clinical neurophysiology. American J. EEG Technol, 1(1):13-19, 1961.
Anders Krogh and John A Hertz. A simple weight decay can improve generalization. In Advances in neural information processing systems, pp. 950 957, 1992.
Vernon J Lawhern, Amelia Solon, Nicholas R Waytowich, Stephen M Gordon, Chou P Hung, and Brent Lance. Eegnet: compact convolutional neural network for eeg-based brain wrain-computer interfaces. Journal of neural engineering, 15(5):056013, 2018.
Andrew L Maas. Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. ieml, volume 30, pp. 3. Citeseer, 2013.
Muhammad Naeem, Clemens Brunner, Robert Leeb, Bernhard Graimann, and Gert Pfurtscheller. Seperability of four-class motor imagery data using independent components analysis. Journal of neural engineering, 3(3):088, 2006.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32: 8026-8037, 2019.
Gert Pfurtscheller and Christa Neuper. Motor imagery and direct brain- crain-computer communication. Proceedings of the IEEE,
2001.
Robin Tibor Schirmeister, Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, and Tonio Ball. Deep learning with convolutional neural networks for eeg decoding and visualization. Human Brain Mapping, 38(11):5391-33533
doi: https:/dooi.org/ 10. 100.0.00000..0..3.1 23730. URL https: / //n// in iirrrrrrr... wi ey com/ doi /abs//////////// 10. 1002 22:00///mm...m 23730
2017.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- gradient-based based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Mohd Zuki Yusoff, Dalia Mahmoud, Aamir Saeed Malik, Mohammad Rida Bahloul, al. Dis- crimination of four class simple limb motor imagery movements for brain-computer interface. Biomedical Signal Processing and Control, 44:181-190, 2018.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference computer vision, pp. 818-833 Springer, 2014.
Dalin Zhang, Lina Yao, Kaixuan Chen, and Jessica Monaghan. A convolutional recurrent attention model for subject-independent eeg signal analysis. IEEE signal processing letters, 26(5):715- 719, 2019.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
9


Reviews:
[{"review_id": "ypzh9j0ByJ0", "reviewer": "Reviewer_uzaX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros.\n\n-An interesting data augmentation algorithm for EEG time-series classification.\n-A class activation mapping approach for EEG data is presented.\n\nCons. and comments\n\n-The paper presentation is poor. For example, the introduction is confusing, lacks suitable state-of-the-art analysis, and does not provide the proposal's paragraph.\n-The results should include other motor imagery databases. The influence of the number of EEG channels and the inter and intrasubject variability must be studied to test the availability of a \"short interval.\" For example, see:\nhttps://academic.oup.com/gigascience/article/6/7/gix034/3796323\nhttps://physionet.org/content/eegmmidb/1.0.0/         \n-The mathematical details should be enhanced for an iclr paper.\n-Please provide the codes and the experimental details.", "summary_of_the_paper": "The authors introduce a deep learning approach for short-time motor imagery classification using EEG data.  Conventional CNN and RNN - GRU layers are used. Remarkably, a data augmentation strategy and a class-activation mapping approach are presented. Overall, the idea is interesting, but the paper presentation, the mathematical foundation, and the experiments provided are poor. Therefore, more details about the model and other experiments should be carried out to validate the proposal. Besides, the authors claim that short time interval EEG classification is achieved; however, 0.8s windows size does not seem to be a \"short interval\" compared to other state-of-the-art methods.", "summary_of_the_review": "A good idea is presented concerning the data augmentation and the cam-based extensions for EEG data. Nonetheless, the short interval claim seems to be ambiguous. Several details regarding the mathematical background and more experiments should be conducted to test the proposal. In addition, the paper presentation needs to be enhanced.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "k3H6sem3WIJ", "reviewer": "Reviewer_M3gg", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The topic of EEG classification is meaningful. \n\n2. The modified class activation mapping (Grad-CAM) is interesting to me.\n\nWeaknesses:\n1. The representation is not clear. To me, the challenges, motivation, solutions, or contributions are not clearly expressed. For example, \n\n- Since the title mentioned 'short-interval EEG', then what's long-interval EEG? What's the difference and how do existing studies treat them differently?\n\n- Sec. 1 uses two long paragraphs to show the limitations of traditional approaches, which are all well known. Please condense them into one paragraph. \n\n2. Lack of novelty. Using standard deep learning models (and attention motivation) for EEG recognition is ok in 2018, but now, we expect more technical novelty. In specific, the CNN, RNN, excitation and squeeze, and FUlly-connected layers are very commonly used and intensively studied components in the last 5 years. \n\n3. Weak experiments. \n- This work only compared with one baseline, which is not enough. I am also confused that what is exact the baseline mentioned in Table 1? I cannot find descriptions about the baseline's model/structure/citation. \n\n- This work claims its effectiveness by 'outperforming the 1st place winner by 4%' (in Abstract and Results). However, the BCI IV competition is in 2008! It is not amazing that the proposed model is better than a winner in 2008.\n\n- The work lacks lots of studies on DL-based EEG analysis in recent 3 years. Please find my recent works and compare them. ", "summary_of_the_paper": "This paper proposes a model combining convolutional operation and recurrent operation along with temporal attention for EEG classification. ", "summary_of_the_review": "As I mentioned in the above weakness, this manuscript is not presented well, the technical novelty is not significant, and the experiments are not extensive. I suggest a strong rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "1mqthJIA6Ei", "reviewer": "Reviewer_tCsE", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is the opinion of the reviewer that following are the strengths and weaknesses of the paper.\n\nStrengths:\n- The paper is generally well-written and structured.\n- Good results are achieved.\n\nWeaknesses:\n- The proposed method is quite simple (Conv RNN + att), unfortunately with no novelty or innovation. Similar methods have been widely used in the past, even in the field of EEG representation learning, e.g. Zhang et al., Classification of Hand Movements from EEG using a Deep Attention-based LSTM Network, 2020\n- Only one dataset (BCI IV 2A) is used. Therefore, it is very unclear how the method generalizes to other datasets. \n- No in-depth analysis of the length of the window sizes (i.e., \"short interval\") is carried out.\n- The model details are not all give, and so the work is not reproducible. \n- A comprehensive comparison with other methods in the field is missing. \n\nBased on the shortcomings mentioned above, unfortunately the paper is very far from the level of ICLR.", "summary_of_the_paper": "The paper proposes a short-interval MI classification system. The model is a convolutional RNN with temporal attention.", "summary_of_the_review": "-", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper develops a deep convolutional network with RNN layers and 
a new data augmentation method for EEG motor imagery classification.

Reviewers agreed that the paper was not very clearly written, and that
without comparisons to other related methods or at least demonstration 
of the importance of each of the components of the model (through for 
example ablation analyses), it was hard to understand the generality
of the approach.

The authors did not respond to the reviews, so I am recommending not  
accepting this paper.
Decision: Reject