# Dual Head and Dual Attention in Deep Learning for End-to-End EEG Motor Imagery Classification
## 1 Introduction  
Motor imagery (MI) classification based on electroencephalogram (EEG) event-related synchronization (ERS) and event-related desynchronization (ERD) phenomena is a measure of the neuron extent when people image body movements [1, 2, 3, 4]. In recent years, two typical and general approaches make important achievements in EEG-MI recognition and brain-machine interface (BMI): optimizing the hand-crafted features and extracting the ERS/ERD features by deep learning. For the former approach, common spatial pattern (CSP) filters and Riemannian Manifold [5, 6, 7, 8] are two popular and effective methods. The CSP method is optimal for discrimination of the filtered time series data, and it forms a low dimensional spatial-subspace for the acquired multi-channel EEG data and derives a covariance matrix for each MI class [9]. [PERSON]'s group introduced Filter Bank Common Spatial Pattern (FBCSP) as an extension of the original CSP algorithm and gained attention by winning the 2008 BCI Competition IV-2a [10, 11]. The FBCSP algorithm recognized that not all frequency bands contain discriminative information, and it optimized the data-driven spectral filter and spatial filter. The Riemannian based dimension reduction algorithm is derived to construct a low-dimensional embeddingfrom high-dimensional Riemannian manifold. [PERSON]'s group used the geodesic distance of Riemannian manifold to determine the adjacency and weight in Riemannian graph, and then proposed bilinear regularized locality preserving (BRLP) to address the problem of high dimensions frequently arising from BMIs [6]. Ref. [7] proposed Riemannian distance and Riemannian mean was directly adopted to extract tangent space (TS) features from spatial covariance matrices of the MI EEG trials. Researchers in [12] utilized a scheme for transfer learning to use the Riemannian geometry of symmetric and positive definite(SPD) matrices, tightly connected to the BMI transfer learning work. Ref. [13] proposes a time-frequency decomposition-based weighted ensemble learning (TFDWEL) method, which aims to improve the classification performance of motor imagery EEG signals. In recent years, EEG power topography is used for MI classification [14; 15]. However, few of these methods are subject-independent explorations. There are currently a number of approaches targeting the subject-independent EEG signal analysis via machine learning. In this area, several studies had made progress via CNN (Convolutional Neural Network). [PERSON] et al. [16] combined multiple one-versus-rest CSP features on CNN for multi-classes MI classification. In [17], a 3D representation is generated by transforming EEG signals into a sequence of 2D arrays which preserve spatial distribution of sampling electrodes. Then, the work proposed a multi-branch 3D CNN and a corresponding classification strategy to preserve temporal-spatial features. In [18], a Convolutional Recurrent Attention Model (CRAM) is built to encode the EEG signals and a recurrent attention mechanism is proposed to explore the temporal dynamics of the EEG signals. [PERSON] et al. [19] recognized 10 volunteers MI activities with a feedforward, multi-layer perceptron network and convolutional neural network in combination with different data pre-processing methods. In addition, one dimension-aggregate approximation is also employed to extract effective MI signal representation for long short-term memory (LSTM) networks, such as [20; 21]. In [21], not only the time and frequency domain features but also a Random Forest (RF) was used to evaluate feature weights. Fields like Natural Language Processing (NLP) and even computer vision have been revolutionized by the attention mechanism. Recent advances in interpreting deep model behaviors, including the employment of attention mechanism [18; 21; 22] and utilization of several types of inputs from frequency or time domain or both, have significantly enhanced the classification accuracy.