MAtt: A Manifold Attention Network for EEG Decoding
Authors: Yue-Ting Pan
Jing-Lun Chou
Chun-Shu Wei
Venue: NeurIPS 2022 Accept (2022)
Keywords: Attention network
Riemannian geometry
SPD manifold
EEG
Brain-computer interface

Abstract: Recognition of electroencephalographic (EEG) signals highly affect the efficiency of non-invasive brain-computer interfaces (BCIs). While recent advances of deep-learning (DL)-based EEG decoders offer improved performances, the development of geometric learning (GL) has attracted much attention for offering exceptional robustness in decoding noisy EEG data. However, there is a lack of studies on the merged use of deep neural networks (DNNs) and geometric learning for EEG decoding. We herein propose a manifold attention network (mAtt), a novel geometric deep learning (GDL)-based model, featuring a manifold attention mechanism that characterizes spatiotemporal representations of EEG data fully on a Riemannian symmetric positive definite (SPD). The evaluation of the proposed mAtt on both time-synchronous and -asyncronous EEG datasets suggests its superiority over other leading DL methods for general EEG decoding. Furthermore, analysis of model interpretation reveals the capability of mAtt in capturing informative EEG features and handling the non-stationarity of brain dynamics.

MAtt: A Manifold Attention Network for EEG Decoding
Chun-Shu Wei National Yang Ming Chiao Tung University, Hsunchu, Taiwan wei@nycu. edu.
Yue-Ting Pan
Jing-Lun Chou
Abstract
Recognition of electroencephallgraahiic (EEG) signals highly affect the efficiency of non-invasive brain-computer interfaces (BCIs). While recent advances of deep- learning (DL)-based EEG decoders offer improved performances, the development of geometric learning (GL) has attracted much attention for offering exceptional robustness decoding noisy EEG data. However, there a lack of studies on the merged use of deep neural networks (DNNs) and geometric learning for EEG de- coding. We herein propose a manifold attention network (MAtt), a novel geometric deep learning (GDI.)-based model, featuring manifold attention mechanism that characterizes spatiotemporal representations of EEG data fully on Riemannian symmetric positive definite (SPD) manifold. The evaluation of the proposed MAtt on both lime-synchronous and -asyncronous EEG datasets suggests its superiority over other leading DL methods for general EEG decoding. Furthermore, analysis of model interpretation reveals the capability of MAtt in capturing informative EEG features and handling the non-stationarity of brain dynamics. Source codes are available at https: / gighttt........ com/ CEe//CCCNL/MAE
Introduction and related works
A brain-com computer interface (BCI) is a type of human machhne interaction that bridges pathway from brain to external devices. Electroenncehhalogram (EEG), non-invasive neuromonitoring modality with high portability and affordability, has been widely used to explore practical applications of BCI the real world D 3 For instance, disabled users can type messages through an EEG-based BCI that recognizes the steady- visual evoked potential (SSVEP) induced by flickerigg visual targets presented on a screen 4 b) Stroke patients who need restoration of motor function undergo motor-imagery (MI) BCI-controlled rehabilitation an active training 7 g Most EBG-based BCI systems are designed detect/recognize reproducible time-asynchronous
time-synchronous EEG patterns of interest, depending the schemes of BCI [9] For example, the MI EEG pattern is an endogenous oscillatory perturbation sourced from the motor cortex without an explicit onset time [0). On the other hand, synchoynchoooous EEG pattern is time-locked to specific event. For example, the pattern SSVEP synchronized to the change of brightness on flickering visual target. The efficiency of BCI systems largely relies on the accuracy and robustness of the EEG decoder. However, due to the low signal-to-nn -ggnal-to-noise ratio (SNR) [1] and non-stationarity 122 of EEG, translating perplexing EEG signals into meaningful information has been grand challenge in the field.
Recent advances in deep learning (DL. have contributed to the rapid development of DL-based EEG decoding techniques 113. DI. models are capable of extracting features automatically according to given training data. Convolutional neural network (CNN) is one type of the most common DL models and has achieved remarkable performance in tasks such as image recognition and object detection |4 1 [6] CNN models newly designed for EEG decoding use convolutional kernels that
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

analogously function as conventional spatial and temporal filters but with extra flexibility to optimize
the transformation of EEG data automatically through model training [7 18] 19.. In addition to the fast growth of DL-bbased EEG decoders, geometric learning (GL.) approaches, mostly hased on Riemannian geometry (RG), have been adopted in the field of BC1 20 RG is type of non-Fuclidean geometry that has different interpretation of Euclid's 's fifth postulate (i.e. parallel postulate) [2I]. In GL, geodesic between points on the manifold is a critical feature for classification tasks in BCL. The power and spatial distribution of a segment of multi-channel EEG signals can be coded into a covariance matrix that is symmetric positive definite (SPD) in general. The use of Riemannian geometry allows mapping of EEG data directly onto a Riemannian manifold where Riemannian metrics are insensitive to outliers and noise [22 20 RG can also avoid swelling effect [23], which is common issue when employing Euclidean metric. Furthermore, metries on Riemannian manifold have several types of invariance properties [24 22], which make the model have higher generalization capability complex EEG signals. In 2010, Barachant al. [25] proposed Minimum Distance to Mean (MDM) that maps target EEG data onto the SPD manifold to find the nearest class center. Later on, they developed TSLDA (26) that projects data from the manifold to specific tangent space where Euclidean classifiers are applicable. RG-based classífication for EEG decoding has shown extra robustness as the relationship between data samples can be stably preserved, leading to success in recent data competitions in the BCI field such as *DecMEG2014")
1 4an2 and the 'BCI challenge
The nascent field of geometric deep learning (GDL) [27] has expanded by emerging techniques to generalize the use of deep neural networks to non- -on-Euclidean structures, such as graphs and manifolds. Efforts have been made to transitioning useful operations from Euclidean to Riemannian spaces, including convolution [28 29 30. activation function (28 29,. batch normalization [31 32)
that facilitate the ongoing development of GDL tools. SPDNet 28 is Riemannian network for non-linear SPD-based learning on Riemannian manifolds using bi-linear mapping that mimics Euclidean convolution for visual classification tasks. ManifoldNet 29 offers high performance in medical image classification with manifold autoencoder. (33] characterizes 3D movement via the manifold polar coordinate with a geodesic CNN. 27 performs convolution on the manifold generalization of local graph or manifold pseudo- coordinate for vertex classification on graph and shape correspondence task. In contrast of the vast develop of GDL in many other scientific fields, only few studies focus on decoding EEG data with a merge use of GL and DL. 34 proposed a network architecture that integrates fusion of Euclidean-basse with multiple LSTM and attention structures to extract spatiotemporal information of EEG. (35 proposes a caamaannian-embbddingg baaks method that separates the entire embeddings into multiple sub-problems for learning spatial patterns of MI EEG signals based on the features extracted from the SPDNet. (36) combines federated learning and transfer learning on Riemannian manifold using the spatial information of EEG. (37) proposes deep optimal transport on the manifold to minimize the cost of domain adaptation from the source domain to the target domain. (38 extracts multi-view representations of EEG. These studies have established cornerstones toward the field of future GDL for EEG decoding, but the increment of performance is yet marginal. Most of the techniques can not map the temporal information of EEG onto the manifold, or still rely Euclidean tools to handle EEG features. We herein propose a manifold attention network, a novel GDL framework, which maps EEG features EEG patterns are fully characterized. The main contributions of the present study are the following:
module and manifold-base module
Riemannian SPD manifold where the spatiotemporal
a manifold attention network proposed for decoding general types of EEG data.
a lightweight, interpretable, and efficient GDL framework that is capable of capturing spatiotemporal EEG features across Euclidean and Riemannian spaces.
an empirical validation of our proposed model demonstrating its generalizable superiority over leading DL approaches in EEG decoding.
neuroscientific insights interpreted by the model that not only echo prior knowledge but also offer new look into the dynamical brain.
This article organized as follows: we first brief the essential background of RG and manifold attention mechanism; next, we leverage the proposed MAtt architecture with details of model design; we then validate our proposed model experimentally; lastly, we interpret our proposed model with neuroscientific insights. Our source code is released at
com/ wtwptthuwbbbooo///c///////
I DecMFG2004:
PBC challenge: http:://www.aggl..oo///mmmieeeee

---

2 Preliminary
manifold generalization of curves and surfaces in Euclidean space. It topological space that can be locally regarded an open set in Hilbert space. If a manifold is equipped with differential structure (i.e. a collection of charts with transition maps defined on the overlaps of the charts), is called a differential manifold [39]. Riemannian geometry is the study of differential manifolds equipped with Riemannian metric. We focus on the symmetric positive definite (SPD) manifold, which allows us directly manipulate manifold-valued data on the manifold. The spatial information of an EEG signal can be represented as specific covariance matrix, which encodes the relationships between channels and is a critical representation for understanding EEG signals. However, the solution of the Riemannian mean does not have a closed form when the manifold is equipped with an affine invariant metrie (AIM),
need calculate the approximate mean in an iterative manner 22 29 until convergence conditions are satisfied. Riemannian mean can be computationally expensive in deep learning due to its high complexity. Therefore, use approximation based on the Log--uuclidan metric 24 as described below.
2.1 Notations
GL,nn, IR) := {A "xx" determinantt()) # 0} is general linear group, which is the set of all real non-singular sqaure matrices. (M,g) denotes connected Riemannian manifold. Sym(n) == {S Maxa (R) ST S} is the space of all
real symmetric matrices, where M (R) specifies the space of all real square matrices, (.)T is the transpose operator, and Sym (n) {P Maxn (R) P PT, uT Pv 0, Vu E R" {0}} is the set of all n X symmetric positive definite(SPD) matrices.< < A, B >p means the Frobenius inner product, defined as Tr(A" B), where Tr(.) is the trace operator. Log(-) and Exp(-) are the principle logrithm operator for SPD matrix 40 and exponential operator for symmetric matrix respectively. Both of them can be computed using the orthogonal diagonalization. Exp: Sym(n) + Sym+ (ym+t(n). an operator maps symmetric matrix S E (n) to Sym (n) by:
Exp(S) "diaglerplo))........ exp(o
where V is the matrix of eigenvectors of S.
The inverse projection of Exp operation is Log operator: Log Sym (n) Sym(n) is operator that maps SPD matrix P E Sym+ (n) to Sym(n) by:
Log(P) (P)
U U'dig(log(ai)) log((a))))
where U is the matrix of eigenvectors of P, since P E Sym (n), > 0, 18;=1,... 1,..,,
2.2 Log-Euclidean metric
Log-Euch Log-Euclidean metric (LEM) offers elegant, analogous, and efficient generalization to calculate the center on the SPD manifold than the affine-invariant metric (AIM) [24] LEM is bi-invariant metric the Lie group the SPD manifold (24). The geodesic distance from P1 Pa on the
Sym (n) ) is also given by [24]
(p P2) = Log())) Log(P2))lF
(2)
Furthermore, we can also define the -ug-Edeli mean(g) via the Log- Euclidean distance:
Σ (P.P))
G(PI...
arg min
PESym (n)
where Pi...., Pt E (Smm+(m) Fortunately, the solution to the formula above has closed form to follow, given by 422
G = Erp Σ Log (Po)
We utilizes the weighted Fug-uueli mean that is endowed with different weights in different Pt in our work. We denote the weight of each P1 wt, where VI (1,2,...,k)) k). Here, (wr)k

---

ouget
Mennedd
X
J/1
22
(a)
Figure 1: (a) The overview of the proposed model architecture. (b) E2R operation: split latent feature into several epochs, and convert each one to specific SPD matrix.
satisfies the convexity constraint definition (i.e.
Σ wi and wy > 0). The definition and the
corresponding weighted Log-Euclidean mean be defined and derived as:
C(PI....PP
arg min reSym
(n Pt)
(m)
and
Exp Σ Lrg((
respectively.
3 Methodology
As shown Figure Ta), the architecture MAtt includes components of the feature extraction (FE), the manifold attention module, transitioning from Euclidean to Riemannian space (E2R), and transitioning from Riemannian to Euclidean space (R2E).
3.1 Feature extraction of EEG signals
We adopt two convolutional layers to extract information of raw EEG signals, where the first convolutional layer performs spatial filtering to the multi-channel EEG signals and the second convolutional layer extracts spatiotemporal features. Our parameter setting follows [99]
3.2 From Euclidean space to SPD manifold (E2R operation)
mightd
stanineityy
Tz
Exp
A1
M
Fin
(b)
Figure 2: (a) The architecture of the proposed manifold attention module. qi k2, 8, refer to the query. key, and value of the tt input matrix 인; respectively; of stands for the output of the proposed module. (b) Illustration of the operation and number of epoch is 3; di and ky refer to ith query and jth key respectively; d, denotes the distance between Qa and ky on the SPD manifold M: Tr refers the tangent space based on identity matrix I.
Log-Fucliden, mean used in proposed module

---

After we got q:, kG and eq by bilinear mapping, we define the similarity for measuring the qi and ky SPD matrices. In Euclidean space, there are several ways to define the similarity. A most common way is to use dot-product [3 to measure the similarity of query and key. However, key, and value are SPD matrices instead of vectors as regular attention. We define the similarity based the Log-Eucling Euglididi distance (Eq. [ between query and key. Suppose we have q, and kj. for some i.j E (1,..., m). The similarity sum(.) is a strictly decreasing f function of distance Oij- Then, the attention As illustrated in FigurdTbb,, b), we convert the embeddings from the feature extraction stage to the SPD data and map the feature embeddings from Euclidean space to the SPD manifold. Suppose f denotes the embeddings after the feature extraction stage, we divide the whole embeddings into several epochs f1; .m.. and calculate the sample covariance matrix(SCM) of each fi, Vi E 21, m). By doing so, we get sequence of covariance matrices that present the temporal information of the embeddings f in the form of SPD data, called SCM, f. SCM f...... SCM f.. After we get some datapoints, we do trace-normalizatiie
and add small number on each main diagonal element for SSCM, + e!) where i E {1, .... m}. / is the identity matrix, and we set as each SCM,, (i.e. to (r(CMM;
le-5 in our source code. The resulting SPD sequence is denoted as X = [ 42, ..,, am] We add a small identity matrix to ensure that each z;; is well-defined SPD matrix.
3.3 Manifold attention module
The input of this layer is sequence of SPD data. The overview of the manifold attention module is illustrated in Figure a). Motivated by 12 and 43). we capture the spatiotemporal information on the manifold. Suppose the module takes sequence of SPD matrices
Forward procedure:
[ 22, .... 2. denoted as X. Herein we have the query, key, and value in the form of SPD matrices on the manifold [33] We convert the Ji; to the di, ki, and ;; via bilinear mapping
[28] and exploit
non-linear and valid features from each segment. Suppose the shape of di de x de, and hg. hg. and h is the mapping from d; to 2i, kis and respectively. We have:
q: = (Z;;; Wo) Wa W? k; (2 We) WLE.WE e, (6, W.) W.S.W.
where T. E (de), Wg, Ws. and W, E Rdu (dh, < de) denotes transformation matrices. Moreover, make sure the output gi, kú and is also SPD matrices, transition matrices W Ws, and Wo are constrained as row-full rank matrices.
query,
[0, oo) + [0, 1] and is defined as: sim(g; ky) 1100 matrix is:
A = [aig]mxm
We then use Softmax function to shrink the range along the row direction, making values in row have converity constraint property. The final attention probability matrix A' is:
A' = Softmor (A) Softmaz((ass)mxxen
[a/ylmxm
where aly- Ets exp(aa) Vi,j 1, I,...,,, m. Finally, we combine the attention probability matrix
emp(as;)
and 11, 02, ..., Um to get the final output v/1, o/2, ..., o' and define the output n((vi 1, 2...... m) via Log-Fuclidean mean as:
Df = Exp
Σ Log(())
The forward procedure of proposed manifold attention module is illustrated in Algorithm
Backward procedure: In order to perform gradient descent parameter updating on the Riemannian manifold, we employed the Riemannian gradient descent method to update the parameters. The trainable parameters in this module are W;: WE, and W... We require the weight updated on Stiefel manifold [44] 28), denoted as St(p, n) (X E R [= X = Ip}. Since our manifold attention module has different mathematical architecture to those in Euclidean space, we herein extend Euclidean gradients onto a Riemannian space. To be precise, we expect our gradients on the Stiefel manifold to generate valid orthogonal weights. The Euclidean gradients of W,, Wx. and W, in the attention module can be derived using the chain rule. Suppose the L is the loss the query, key, and value generated in the manifold attention module are gı, K2, and v, Vi 1 respectively, and the
5

---

Algorithm 1 Manifold attention module
Require: A sequence of SPD data { transformation matrices W. Wa, W
1: for
m do
2: di W WT: k W. W.
3: end for
4: V E (1. mm. A: = (aijlmx Hoog(11
5: A' Softmar(A)
6: for = 1: do
7:
U/- Exp
Σ EdlLog(li)
8: end for
9: return sequence of SPD data {1/
output of the attention module is v Vi 1 ... m (All notations here are the same as the Algorithm
m JL did V Vw,C=
m OL dvl 0q: Ou² 0q. OW,
m
dc
m
Ook
dols Oq.
dy: aw,
do! OW,
du!
ac
dok oki
Σ dc
Ouk da''s Da's Ohi
V L
Σ
Σ
tug ow
Fk oW
Ou!
ow
1=1
1=1
m
OC dok
m
OC
m
dek do,
Vw.C C= Σ
Σ
Σ Duy aw,
do aw,
do! i=1
The way of the attention module to impose the Euclidean gradient to the Stiefel gradient on the manifold [44) is: Suppose Vw, C is du x de matrix and W, is on the manifold. Then the projection of Vw, C onto the normal space, which is the orthogonal complement of the tangent space based on the tangent point Wo is:
C) (Vw, C)
w. TN (V w. C) W. (W2 V Vw, C) f) W,
2
where the D Daym is defined as D+D The tangent component of the St(de, d.) can be defined as the 2
subtraction between the Euclidean gradient and normal component AN (V w. L) from Eq.
(T w L) + V W. WW
c VCC Ww. AN (Vw. C) W..C- W,
Now we have the Stiefel gradient Ww,C. C., which is the tangential direction that we update the parameters. We use the retraction operation, which is smooth mapping to map the final weight back to the St(de, du). Finally, We have the updated weight:
(neww) = r(W, - xV
where À is the learning rate, I' is the retraction operation defined in the QR decomposition [45). The backward procedure for W', and W' are the same. For more concepts about the optimization in Riemannian geometry, we recommend readers refer to 44 45)
As shown in Figure (b). the output of of our attention module can be comprehended as a projection that translates the weighted sum, on the tangent space, of three different matrices encoded by three different epochs and corresponding attention scores a' a12, a/3 (or weights) to specific representative matrix v/ on SPD manifold. Herein the weights for the weighted sum on the tangent space assigned by its query matrix Qi and corresponding keys kı k2, kg to generate the relevance score between 91 and ki, f2, k3 43 46).

---

3.4 From Riemannian manifold to Euclidean space (R2E) and loss layers
After passing through the attention module, the ReEig layer is used to imitate the ReLU function. However, unlike the ReLU function, which sets threshold to the value of the input. the ReEig layer sets a threshold to the eigenvalue of the input, as defined in (28]. The R2E operation aims to map the SPD data back to Euclidean space for the final classification, which is composed of logarithmic layer and regular flatten layer in [28]. The logarithmic layer is a common technique in geometric deep learning for projecting SPD data to Euclidean space, allowing us to reduce the manifold to a flat space using the Log(.) operation. We apply the Log(.) operation to the output from the attention module layer, v4,, 1/2, ... I'm E (dm)) Denote the R2E operation as hr. Sym +
Rdd x(du+))/2
hz(e)) flatten (Log(v\)) = flatten (S(diag(log(o))... (og((o..)))
where S is the eigenvector matrix of v/,, and oi, aA are the eigenvalues of e/.. The Log(.) operation is the same as Eq. and the t flaatten(A) A) operation flatten the upper triangle of the arbitrary
symmetric matrix A.
Finally, we set a fully connected layer and regular softmax operation on embeddings after R2E operation. Suppose the output from the whole model stream is y, the groundtruth is y, we define the loss as the cross-entropy loss of ú and y.
4 Experiments
Here we evaluate the proposed MAtt using both time-asynchronous and time-synchronous EEG data to give empirical evidence of the advantages. The performance in a general use for EEG decoding is compared against leading DL-based techniques. We incorporate the BCI Competition IV 2a Dataset (BCIC-IV-2a) 47 to assess the performance on time-asy aame-ayychhooouu motor- motor-imagery (MI) EEG decoding the MAMEM EEG SSVEP Dataset II [MAMEM-SSVEP-II] 48 and the BCI challenge error-related negativity (ERN) dataset (BCI-ERN) 44 to assess the performance on iime-synchronous SSVEP and ERN EEG decoding. Previous and current state-of state-of-the-ard ahe- DL-based models listed for comparison with MAtt include MBEEGSE [50], TCNet-Fusio (5]] FEG-TCNet
52
FBCNet [53] (9) EEGNet [17, and ShallowConvNet [18
4.1
Datasets and preprocessing
Data set I (MI): We employ the BCIC-IV-2a a representative dataset providing time asmehhsynnhooouus EEG data. The BCIC-IV-2a dataset is one of the most commonly used public EEG dataset released for the BCI Competition IV in 2008 [47. It contains EEG recordings in four-class motor-imagery task from nine subjects with two repeated session each on different days. During the task, the subjects were instructed to perform imagination of one of the four types of movements (right hand. left hand, feet, and tongue) for four seconds after an instructional cue. Each session consists of a total of 288 trials with 72 trials for each type of the motor imagery. The EEG signals were recorded by 22 Ag/AgCI EEG electrodes at the central and surrounding regions in a sampling rate of 250 Hz. We performed standard preprocessing procedures for the 22-channel EEG signals, including 1) nown-sampiig from 256 Hz to 128 Hz, 2) Band- -ass filtering at 4-38 Hz, and 3) Segmenting EEG signals at 0.5-4s (438 timepoints) of the onset of cue for each trial.
Data set II (SSVEP): The MAMEM-SSVEP-II dataset was incoporated as the representative dataset of time-synchronous EEG data. It consists of EEG recordings from 11 subjects performing a SSVEP- based task where the subjects were instructed to gaze at one of the five visual stimuli flickering at different frequencies (6.66, 7.50, 8.57, 10.00, and 12.00 Hz) for five seconds. Each subject executed five runs of five cue- based trials for each of the five stimulation frequencies. The EEG signals were acquired using the EGI 300 Geodesic EEG System (GES 300) equipped with 256 channels in sampling rate of 250 Hz. There are 5 sessions in this dataset. We assigned the early blocks (session 1, 2, and 3) as the training set, session 4 as validation set and the remaining (session 5) as the test set to maximize the efficacy of the cross cross-session validation. The preprocessing procedures for this dataset were 1) Band-pass filtering at 1-50 Hz, 2) Selecting eight channels (PO7, PO3, PO, PO4, POS, O1, Oz, and 02) in the occipital area, the location of visual cortex, and 3) Segmenting each trial into four 1 I-second segments at 1s-5s of the onset of cue, yielding a total of 500 trials of -second 8-channel SSVEP signals for each subject, thus the time length of input EEG data is 125.
7

---

Table 1: Performance comparison between MAtt and baseline DL methods on MI (BCIC-IV-2a), SSVEP (MAMEM-SSVEP and ERN (BCI-ERN) datasets. Bold fonts mark the highest overall performance (MI, SSVEP: accuracy, ERN: AUC). We adopted Wilcoxon-sign rank test with Bonfer- roni correction to perform the multiple comparison between all models. The statistical test result is available in Appendix.
Models ChallowConvNet
MI
SSVEP
ERN
61.84+6.339
56.93+6.97
71.86++2.64
EEGNet
6..43:16.25
53.72+7.23
74.28
SCCNet
71.95:15.05
62.1117.70
70.93+2.31
EEG-TCNet
67.09:14.66
55.45:17.66
77.05:12.46
TCNet-Fusion
56.52:13.07
45.00:16.45
70.46:12.94
FBCNet
71.45=4.445
53.09::5.67
60.47+3.06
MBEEGSE
64.58+16.07
56.45+7.27
75.46+2.34
MAt
74.7145.001
6,.50
76.01++2.28
Data set III (ERN): Another -ynchhonchoo BCI Challenge ERN dataset (BCI-ERN) is adopted to validate the capacity of presented MAtt. The BCI-ERN dataset 49] was employed in the BCI Challenge on Kaggle ! Twenty-six subjects participated in P300-based BCI spelling task and measured the ERN, a type of event-related potential (ERP), elicited from the feedback of erroneous input displayed by the BCI speller. The aim of the experiment was to detect and determine the type of signal perturbation evoked by the erroneous feedback from BCI speller to improve and assess the robustness. Therefore this task is ainary-andiimbalance decoding task because the BCI speller detects more amount of correct inputs than the amount of erroneous inputs. EEG recordings were recorded by 56 Ag/AgCI EEG electrodes (VSM-CTF compatible system) whose region followed the extended 10-20 system in sampling rate of 600 Hz. There are five sessions (60 trials for the first four sessions and 100 trials for the last session) per subject, and the duration a single EEG trial is 1.25 seconds. We used the 16 subjects released in the early stage of the Kaggle competition. The preprocessing steps include 1) Downsampling from 600 Hz to 128 Hz and 2) Band-past filtering at 1-40 Hz. Each trial has size of 56 channels by 160 timepoints after the preprocessing step.
Model training and validation: A series of experiments were conducted to evaluate the perfor- mances of the MAtt against other EEG decoders with the context of real-world BCI usage taken into account. In the real-world usage of BCI, a user usually needs to go through a training session for collecting a sufficient amount of individual EEG data for training the decoding model before executing the BCI system. To stick with the practical scenario, we performed an individual training scheme where a chunk of trials within a subject are assigned to the training set and the left-over trials within the same subject are used for testing 19 53]. For the BCIC-IV-2a dataset, we used the first session of subject to the training
where out of eight used for validation for MAtt with m = 3. The model with the lowest validation loss within 350 iterations was used for testing the second session of the same subject. For the MAMEM-
dataset, we assigned the first four sessions of a subject to the training set where one out of four was used for validation for MAu with m 7/m 3. The model with the lowest validation loss within 180/130 iterations was used for testing on the fifth session of the same subject. The classification performances for BCIC-IV-2a and MAMEM-SSVEP-II datasets were estimated by the mean accuracy across repeats for each subject. On the other hand, we use the same criterion as [17], herein the AUC score [54] is adopted to estimate the performance of BCI-ERN dataset due to the imbalanced issue.
4.2 Performance comparison
We validate the performance of MAt against other leading methods. The criteria of selecting the baseline model collection here is based on: 1) code availability and completeness and 2) solid evaluation (e.g. cross-session) without additional auxiliary procedures (e.g. manual feature extraction, data augmentation, pre-trained model, etc). As shown in Table [ MAtt outperforms all other leading DL methods on both Lime-synchronou (SSVEP) and -asynchronous (MI) EEG decoding, However, for the ERN dataset, MAtt seconds to the EEG-TCNet with 1% deficit but is least 7% higher than EEG-TCNet on other two kinds of EEG decoding tasks. As most DL-based EEG decoders are
PCI challenge: http::/wwww.aggll.omm//miaaaaa hhl.aaall...cmm////iiciccaccaaae
8

---

Table Overall accuracy (%) on BCIC-IV-2a and MAMEM-SSVEP and overall AUC score on BCI-ERN (%) with parts within the model appended. FE: feature extractor, MA: manifold attention module; SA: self-attention module.
Parts appended
MI
SSVEP
ERN
FE MA
26.08=0..70 60.7315.80
20.18
73.40+2.27
30.51+2.57
59.47=3.56
FE+SA FE+MA (proposed)
49.19+2.72
22.91+22000
63.77+1.71
74.71+5.001
65.50=8.20
76.01+22.28
designed for a single type of EEG decoding task due to high variability between different types of the experimental results suggest that MAtt provides sesh-performacce EEG decoding across types of EEG data. Our study attests the robustness proposed MAtt, which has strong generalization capacity adapt general types of EEG data compared other leading DL. models.
EEG data [53 12 191,
4.3
Ablation study
We assess the significance of each of the major component in MAtt via series of ablation analysis. As shown in Table 2 the accuracy reduces we only use one component in our MAtt to do the classification. However, the combination of proposed manifold attention module and feature extractor (FE+MA) achieves the best accuracy on all datasets. This implies that there is no redundant component in our proposed MAtt, and each part is needed for its non-negligible functions. The feature extractor aims to denoise and preprocess the EEG signals, and the attention module focus on integrating the preprocessed EEG signals and capturing underlying dynamics in the latent features. We can further compare the result between FE+SA and FE+MA to check the necessity of MA: the performance of FE+MA significantly outperform the regular self-attention based FE+SA all validation EEG datasets.
4.4 Model interpretation
Through analysis for the interpretation of the proposed model, MAtt, we are able to un- cover the underlying characteristics learnt from the data. Figure illustrates the gradient re- sponse for MI EEG decoding across channel and across time. We can see left/right hand MI responses strong C4 and C3 corresponding to right/left motor cortices that control the lateral motor functions of the contralateral side of the body [57] Both feet and tongue MI, that are not lateral movements, presents strong responses at CPz above the midline of motor cortex. The spatial distribution clearly exhibit asymmetric pattern for left/right hand MI and symmetric pattern for feet/tongue MI. The temporal infor- mation is available in Figure where all four types MI induce strong response at mu band around 10 Hz. This result is in line with the well-known association between motor function and mu rhythm EEG recordings [57). The visualization of our model for SSVEP decoding is exhibit the appendix.
Right
Feet
Tongue
sugaeeearg out @ 4 doy gogn grgge 11 -hee
Figure 3: Spatial topomaps for the mean absolute gradient response (computed as [55] (66) across time from the visualization of the model S3 in the BCIC-IV-2a dataset for the four motor-imagery classes (left hand, right hand, feet, and tongue). Dark red marks the brains region presenting strong gradient activation at C4 (over right motor cortex) for the left hand, C3 (over left motor cortex) for the right hand, CPz (over motor cortex) for the feet and the tongue motor imagery.

---

Frequency
0
Left hand
0
Right hand
0
Feet
Tongue
25
25
25
25
0.0
1.01,
02 Time
3.02
0.0
1.01
02
3.02
0.0
1.01
2.02
3.02
0.0
1.01
2.02
3.02
Time
Time
Time
Figure 4: Time-frequency spectrograms from the gradient-based visualization (computed in (61) of the model S3 the BCIC-IV-2a dataset for the four motor- motor-imagery classes (left hand, right hand, feet, and tongue). Strong response of motor imagery marked by dark red specific frequency bands and time intervals. Increased response of motor imagery is found at mu band ( 10 Hz) for all classes. The strong response of left/right hand motor imagery occurs at 1-2 seconds, the feet motor imagery is most vivid at 0.5-1 seconds, and the tongue motor imagery induced two peaks second and seconds.
Figure ddepicss the distribution at- tention scores across epochs between the (a) MI, (b) SSVEP, and (c) ERN EEG signals. Here, the attention score refers to the average of the rel- evance score of an attention network,
Attention
Attention
0-1.28
12-2233
(a) 2.3-3.5
0-0 148
1440 23t6
22--0
20-0-43B
03300.576
04330.55
07-0000711
0.71-0
0.88
18
0-0.25
0.42-0.83
0.831 25s
as described in [43, and has been applied interpreting an attention- based EEG decoder [46]. For MI EEG signals, attention score is the highest at the first epoch and decreases in the following epochs, which implies that the beginning of the motor imagery
(6)
(c)
Figure 5: The distribution of attention scores across epochs within trial based on the model interpretation for a sin- gle subject from the three datasets: (a) BCIC-IV-2a: (b) MAMEM-SSVEP-IL and (c) BCI-ERN.
may contribute a higher importance determined by the manifold attention module. The profile of attention score for SSVEP BEG signals presents similar traits that the earlier epochs relate to higher importance. For ERN EEG signals, attention score is the highest on the first two epochs. As we observe consistency cross EEG datasets that higher attention scores lie in earlier epochs, this may infers that the attention module relies largely on the similarity to the early stage of a trial, which is analogous to baseline correction, major common procedure in conventional EEG signal processing [58]. This analysis reveals the capability of MAtt in handling the non-stationarity of the dynamical brain activity.
Conclusion
We propose a manifold attention network as a novel GDL framework for both synhhsyyhhooouus and -asynchronous EEG decoding. Using back propagation based on the Stiefel manifold, the proposed MAtt capable of mapping EEG features onto Riemannian manifold, where spatiotemporal EEG patterns are captured and characterized, within lightweight architecture. The experimental results suggest the superiority of MAtt over current leading DL methods for both time-synchronous and -asynchronous EEG decoding. Spatial and temporal EEG patterns interpreted from the model in line with prior neuroscientific knowledge and shed light on potential possibility of tracking the brain dynamics. In sum, our privileged method, MAtt, improves the SOTA performance of EEG decoding, and is expected to impact on GDL-based EEG processing with generalizable efficiency and robustness for future development of various BCI systems.
Acknowledgement
This work was supported in part by the National Science and Technology Council (109-2222-E- 009-006-MY3, 110-2221-E-444--330MMM Education Sprout Project of the National Yang Ming Chiao Tung University and Ministry of Education of Taiwan.
and 110-2314-B-037-8 and in part by the Higher
10

---

References
[1] Michal Teplan et al. Fundamentals of EEG measurement. Measurement science review, 2(2)1-11,,000
[2] Inaki Iturrate, Javier Antelis, and Javier Minguez. Synchronous EEG brain-actuated wheelchair with automated navigation. 2009 IEEE International Conference on Robotics and Automation, pages 2318-2325, IEEE, 2009.
[3] D Puthankattil Subha, Paul K Joseph, Rajendra Acharya U, Choo Min Lim, et al. EEG signal analysis: survey. Journal of medical systems, 34(2): 195-212, 2010.
[4] ZT Al-Qaysi, BB Zaidan, AA Zaidan, and MS Suzani. A review of disability EEG based wheelchair control system: Coherent taxonomy, open challenges and recommendations. Com- puter methods and programs biomedicine, 164:221-233 2018.
[5] Yu-Te Wang, Masaki Nakanishi, Yijun Wang, Chun-Shu Wei. Chung-Kuan Cheng, and Tzyy- Ping Jung. An online brain-computer interface based on SSVEPs measured from non-hair- bearing areas. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 25(1):14- 21, 2016.
[6] Xiaogang Chen, Yijun Wang, Masaki Nakanishi, Xiaorong Gao, Tzyy-Ping Jung, and Shangkai Gao. High-speed spelling with a noninvasive computer interface. Proceedings of the national academy of sciences. 122(44)
2015.
[7] Kai Keng Ang, Karen Sui Geok Chua, Kok Soon Phua, Chuanchu Wang, Zheng Yang Chin, Christopher Wee Keong Kuah, Wilson Low, and Cuntai Guan. randomized controlled trial of EEG-based motor imagery brain-compun interface robotic rehabilitation for stroke. Clinical EEG and neuroscience, 46(4):310-32) 2015.
[8] Luz Maria Alonso- Alonso-Valerdi, Ricardo Antonio Salido-Ruiz and Ricardo A Ramirez-Mendoza. Motor imagery based brain brain-comea computer interfaces: An emerging technology to rehabilitate motor deficits. Neuropsychologia, 19:354-363. 2015.
[9] Fabien Lotte, Marco Congedo, Anatole Lécuyer, Fabrice Lamarche, and Bruno Arnaldi. review of classification algorithms for EEG-based brain-computer interfaces. Journal of neural engineering, 4(2):R1, 2007.
[10] Matteo Feurra, Patrizio Pasqualetti, Giovanni Bianco, Emiliano Santarnecchi, Alessandro Rossi, and Simone Rossi. State-dependent effects of transcranial oscillatory currents on the motor system: what you think matters. Journal of Neuroscience, 17444)417483317489, 2013.
[11] Don H Johnson. Signal-to-noise ratio. Scholarpedia, 1(12):2088, 2006.
[12] Gabriel Emile Hine, Emanuele Maiorana, and Patrizio Campisi. Resting-state EEG: A study on its non-stationarity for biometric applications. In 20/7 International Conference of the Biometrics Special Interest Group (BIOSIG), pages 1-5. IEEE, 2017.
[13] Yannick Roy, Hubert Banville, Isabela Albuquerque, Alexandre Gramfort, Tiago H Falk, and Jocelyn Faubert. Deep learning- based electroencephalographh analysis: a systematic review. Journal of neural engineering, 16(5):051001, 2019.
[14] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-basee learning applicd to document recognition. Proceedings of the IEEE, 86(1 (0(11))2278-2333 1998.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[16] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference machine learning, pages 6105-61 615-6114, 14. PMLR, 2019.
[17] Vernn J Lawhern, Amelia J Solon, Nicholas R Waytowich, Stephen M Gordon, Chou P Hung and Brent Lance. EEGNet: a compact convolutional neural nerwork for EBG-based brain-compute interfaces. Journal of neural engineering, 15(5):056013, 2018.
11

---

[18] Robin Tibor Schirmeister, Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann. Frank Hutter, Wolfram Burgard. and Tonio Ball. Deep learning with convolutional neural networks for EEG decoding and visualization. Human brain mapping, 38(11):5391-542) 2017.
[19] Chun-Shu Wei, Toshiaki Akino, and Ye Wang. Spatial component-wise convolutional network (SCCNet) for motor-imagery EEG classification. In 20/9 9th International IEFFF/EMBS Conference on Neural Engineering (NER), pages 328-331. IEER, 2019.
[20] Fabien Lotte, Laurent Bougrain, Andrzej Cichocki, Maureen Clerc, Marco Congedo, Alain Rakotomamonjy, and Florian Yger. review of classification algorithms for EEG-based brain-computer interfaces: a 10 year update. Journal of neural engineering, 15(3):031005, 2018.
[21] Roberto Torretti. Philosophy of geometry from Riemann to Poincaré, volume 7. Springer Science & Business Media, 2012.
[22] Marco Congedo, Alexandre Barachant, and Rajendra Bhatia, Riemannian geometry for EEG- based brain-computer interfaces; a primer and a review. Bruin-Computer Interfaces, 4(3):155- 174, 2017.
[23] Florian Yger, Maxime Berar, and Fabien Lotte. Riemannian approaches in brain-computer interfaces: a review. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 25(10):1753-1762 2016.
[24] Vincent Arsigny, Pierre Fillard, Xavier Pennec, and Nicholas Ayache. Log-Euclidean metrics for fast and simple calculus on diffusion tensors. Magnetic Resonance in Medicine; An Official Journal of the International Society for Magnetic Resonance in Medicine, 56(2):411444 2006.
[25] Alexandre Barachant, Stéphane Bonnet, Marco Congedo, and Christian Jutten. Riemannian geometry applied to BCI classification. In International conference on latent variable analysis and signal separation, pages 629-639-666 Springer, 2010.
[26] Alexandre Barachant, Stéphane Bonnet, Marco Congedo, and Christian Jutten. Multiclass brain- computer interface classification by Riemannian geometry. IEEE Transactions Biomedical Engineering, (9(4):920-92) 2011.
[27] Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model CNNs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1124,, 2017.
[28] Zhiwu Huang and Luc Van Gool. A riemannian network for spd matrix learning. In Thirty-Fiin AAAI Conference on Artificial Intelligence, 2017.
[29] Rudrasis Chakraborty, Jose Bouza, Jonathan Manton, and Baba C Vemuri. Manifoldnet: deep neural network for manifold-valued data with applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
[30] Jose J Bouza, Chun-Hao Yang, David Vaillancourt, and Baba C Vemuri. MVC-Net: Convo- lutional Neural Network Architecture for Manifold- Valued Images With Applications. arXiv preprint arXiv:2003.01234 2020.
[31] Daniel Brooks, Olivier Schwander, Frédéric Barbaresco, Jean-Yves Schneider, and Matthieu Cord. Riemannian batch normalization for SPD neural networks. Advances in Neural Informa- tion Processing Systems, 32, 2019.
[32] Minhyung Cho and Jaehyung Lee. Riemannian approach to batch normalization. Advances in Neural Information Processing Systems, 30, 2017.
[33] Jonathan Masci, Davide Boscaini, Michael Bronstein, and Pierre Vandergheynst. Geodesic con- volutional neural networks on Riemannian manifolds. In Proceedings of the IEEE international conference computer vision workshops, pages 37-45, 2015.
12

---

[34] Guangyi Zhang and Ali Etemad. RFNet: Riemannian fusion network for EEG-base brain- computer interfaces. arXiv preprint arXiv:2008. 2020.
[35] Yoon-Je Suh and Byung Hyung Kim. Riemannian embedding banks for common spatial patterns with EEG- based SPD neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI). pages 854-862, 2021.
[36] Ce Ju, Dashan Gao, Ravikiran Mane, Ben Tan. Yang Liu, and Cuntai Guan. Federated transfer learning for EEG signal classification. In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 3040-3045. IEEE, 2020.
[37] Ce Ju and Cuntai Guan. Deep Optimal Transport on SPD Manifolds for Domain Adaptation. arXiv preprint arXiv:2201.05745. 2022.
[38] Ce Ju and Cuntai Guan. Motor Imagery Classification. arXiv preprint a:Xiv:2202.02472, 2022.
A Novel Geometric Deep Learning Framework for
[39]
Wilhelm PA Klingenberg. Riemannian geometry, volume 1. Walter de Grayter, 2011.
[40] Maher Moakher. A differential geometric approach to the geometric mean of symmetric positive-definite matrices. SIAM Journal on Matrix Analysis and Applications, 26(3):735-747, 2005.
[41] Frédéric Barbaresco. Innovative tools for radar signal processing based on Cartan's 's geometry of SPD matrices information geometry. In 2008 IEEE Radar Conference, pages 1-6. IEEE, 2008.
[42] Vincent Arsigny, Pierre Fillard, Xavier Pennec, and Nicholas Ayache. Geometric means in novel vector space structure on symmetric positive-definite matrices. SIAM journal on matrix analysis and applications, 29(1):328-33) 347, 2007.
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[44] Alan Edelman, Tomás A Arias, and Steven T Smith. The geometry of algorithms with or- thogonality constraints. SIAM journal on Matrix Analysis and Applications, 20(2):303-333, 1998.
[45] P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms on matrix manifolds. In Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2009.
[46] Huy Phan, Kaare B Mikkelsen, Oliver Chen, Philipp Koch, Alfred Mertins, and Maarten De Vos. SleepTransformer: Automatic sleep staging with interpretability and uncertainty quantification. IEEE Transactions on Biomedical Engineering, 2022.
[47] Clemens Brunner, Robert Leeb, Gernot Müller-Putze Putz, Alois Schlögl, and Gert Pfurtscheller. BCI Competition 2008-Graz Graz, data set A. Institute for Knowledge Discovery (Laboratory of Brain-Computer Interfaces), Graz, University of Technology, 16:1-6, 2008
[48] Spiros Nikolopoulos. MAMEM EEG SSVEP Dataset II (256 channels, 11 subjects, 5 frequen- cies presented simultaneously). 2021.
[49] Perrin Margaux, Maby Emmanuel, Daligault Sébastien, Bertrand Olivier, and Mattout Jérémic. Objective and subjective evaluation of online error correction during P300-based spelling. Advances in Human- Computer Interaction, 2012, 2012.
[50] Ghadir Ali Altuwairri, Ghulam Muhammad, Hamdi Altaheri, and Mansour Alsulaiman. Multi-Branch Convolutional Neural Network with Squeeze-and-Exciiatle Attention Blocks for EEG-Based Motor Imagery Signals Classification. Diagnostics, 12(4))995, 2022.
[51] Yazeed K Musallam, Nasser AlFassam, Ghulam Muhammad, Syed Umar Amin, Mansour Alsulaiman, Wadood Abdul, Hamdi Altaheri, Mohamed A Bencherif, and Mohammed Al- gabri. Elecrneeneepallggaapyy based motor imagery classification using temporal convolutional network fusion. Biomedical Signal Processing and Control, 69: 102826, 2021.
13

---

[52] Thorir Mar Ingolfsson, Michael Hersche, Xiaying Wang, Nobuaki Kobayashi, Lukas Cavigelli, and Luca Benini. EEG-TCNet: An accurate temporal convolutional network for embedded motor-imagery brain-machine interfaces. In 2020 IEEE International Conference Systems, Man, and Cybernetics (SMC), pages 2958-2968 IEEE, 2020.
[53] Ravikiran Mane, Effie Chew, Karen Chua, Kai Keng Ang. Neethu Robinson, A Prasad Vinod, Scong- Whan Lee, and Cuntai Guan. FBCNet: A multi-view convolutional neural network for brain-computer interface. arXiv preprint arXiv:2104.01233.0000 2021.
[54] Saharon Rosset. Model selection via the AUC. In Proceedings of the twenty-first international conference on Machine learning, page 89, 2004.
[55] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:/312.6034, 2013.
[56] Jost Tobias Springenherg. Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: 2014.
[57] Gert Pfurtscheller, Clemens Brunner, Alois Schlögl, and FH Lopes Da Silva. Mu rhythm (de) synchronization and EEG single-trial classification of different motor imagery tasks. NeuroIm- age, 31(11:1133-133)) 2006.
[58] Burkhard Maess, Erich Schröger, and Andreas Widmann. High-pass filters and baseline correction in M/EEG analysis. Commentary How inappropriate high-pass filters can produce artefacts and incorrect conclusions in ERP studies of language and cognition". Journal of neuroscience methods, 266: 164-165, 2016.
14


Reviews:
[{"review_id": "TUw1NL8SAHA", "reviewer": "Reviewer_hJCd", "summary": "The paper proposes a manifold attention network (mAtt), that leverages the properties of the SPD manifold in order to perform spatiotemporal analysis on EEG signals. In order to achieve that, the authors introduce a specific covariance matrix representation of the spatial information of the EEG signal. Concretely, they calculate the sample covariance matrix for each of the embeddings derived from the output of two convolutional layers fed with EEG time series. Then, the sequence of the covariance matrices (which are SPD matrices) is fed into a manifold attention layer that applies the attention mechanism on the SPD manifold. ", "questions": "The capturing of temporal information is not very clear. How creating covariance matrices of consecutive EEG time-frames is better for capturing temporal features than using simple LSTM or GRU layers that have demonstrated excellent performance on similar tasks? (Zhang, Dalin, et al. \"Cascade and parallel convolutional recurrent neural networks on EEG-based intention recognition for brain-computer interface.\")\n\nSPD matrices have been widely used in medical data analysis. However, to our knowledge, the criteria for choosing a particular metric on the SPD manifold for a given application are vague. Therefore, it would be nice to see the reasoning behind the choice of the Log-Euclidean metric instead of other alternatives (i.e. affine invariant metric Thanwerdas, Yann, and Xavier Pennec. \"Is affine-invariance well defined on SPD matrices? A principled continuum of metrics.\"). Also, a comparative study demonstrating the advantages of one metric over the others would be an interesting addition to the paper. \n\nThe framework is tested on 9 subjects (asynchronous data) and 11 subjects (synchronous data). However, in order to be able to safely consider its generalization capabilities, it would be nice to evaluate how the framework performs given a bigger multi-subject dataset (i.e. EEG Motor Movement/Imagery Dataset from PhysioNet).\n\nFig 5 reveals some very interesting insights regarding neural dynamics with respect to the time duration. It would be interesting to see whether they can be verified by current neuroscientific knowledge. \n", "limitations": "Some of the limitations have been addressed in section 4.3.  Potential limitations that have not been addressed by the authors have been extensively discussed in the Questions section of this review.", "rating": 6, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe interpretability of the proposed framework is validated by the current neuroscientific knowledge. Concretely, Fig. 3 very nicely demonstrates brain regions corresponding to certain motor imageries of specific body parts. \n\nSignificance:\nThe proposed framework demonstrates non-trivial advances in terms of performance compared to sota methods.\n\n**Weakness**\n\nClarity:\nLine 114 there seems to be a plus (+) sign missing from the Sym(n) space.\nLine 119 A clarification regarding what AIM stands for, followed by the corresponding literature reference, would be useful. (I assume it stands for affine invariant metric)\nFig. 2 More information is required in the caption.\nSection 3.3  a very brief explanation of the terms query, key and value would give a nice ‘flow’ to the reading process of the paper, even though these terms have been extensively discussed in the referenced literature. \n\nOriginality:\nThe idea of representing EEG signals as covariance matrices and then projecting them on the SPD manifold has been widely used. The novelty of the approach lies mainly in the application of the attention mechanism on the SPD manifold. Therefore it would be interesting to see how the model performs when compared with very similar frameworks that exploit attention in different manners, i.e. Zhang, Guangyi, and Ali Etemad. \"RFNet: Riemannian fusion network for EEG-based brain-computer interfaces.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OqbWtDK68uR", "reviewer": "Reviewer_AD4a", "summary": "The proposed MAtt is a manifold attention-based model to capture EEG representations. I am not an expert in manifold and the 'geometric' in this paper is not have the same meaning in my domain (where geometric learning denotes graph neural networks). Therefore, please decrease the weight of my comments and consider more from other reviewers. \n\nI will provide the comments as a general researcher with expertise in EEG. ", "questions": "See Weaknesses above. \n\nBTW, It's better to move Sec. 3.5 to Experiments (Sec. 4).", "limitations": "The authors discussed their limitations in Sec. 4.3. ", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Considered both MI EEG and SSVEP. MI is a passive EEG while SSVEP is an event-evoked EEG. They are two key categories in EEG family. \n\nWeaknesses:\n\n1. The experimental results in Table 2 are kind of weird to me (lower than expected), so I checked the performances in STOA. The performance of MI-BCI competition IV-2a reported in the literature is higher than that reported in this work. \n\nFor example, [1] provides the accuracy of this dataset ranges from 76% to 79% (higher than the highest value in Table 2). [2] shows the accuracies ranges from 75% to 81%. Please check, compare, and explain. \n\n[1] S-EEGNet: Electroencephalogram Signal Classification Based on a Separable Convolution Neural Network With Bilinear Interpolation, 2020.\n[2] Advanced TSGL-EEGNet for Motor Imagery EEG-Based Brain-Computer Interfaces Lack of STOA baselines, 2021.\n\n2. Lack of SOTA baselines. This manuscript only compared with four baselines which are published in 2017, 2018, 2019, and 2021 (preprint) separately. More recent and powerful baselines are expected to be included in Table 2. \n\n3. In the Introduction, it is not clear why geometric deep learning is necessary in EEG classification. Manifold-based methods have boost performance in images is not a good idea for using the same methods in EEG. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "G1fbWKfXHXu", "reviewer": "Reviewer_xJ3M", "summary": "This study presented a novel method for the brain-computer interface, where a manifold attention mechanism is applied to Riemannian SPD. The results showed significant improvement over the SOTA methods on two typical datasets (motor imagery and SSVEP). The paper is clearly written with adequate analysis. The explanations provide interesting insights. ", "questions": "Q1: The attention scores are highest for the first part of the MI data and moderate for the middle part, are they linked to the ERD/ERS? \nQ2: For SSVEP, why the attention scores should be different for different parts of the data if the underline signals are the \"steady\" visual responses from the EEG?", "limitations": "The proposed method provides a novel treatment to BCI with promising performance. More experiments on different datasets should be tested to confirm its generalizability. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This study is a new development of geometric deep learning for the brain-computer interface. \n- The results are promising.\n\nWeakness:\n- The interpretations should provide more in-depth analysis to provide deep insight. For example, it shows the attention scores are high for the first part of the MI data and moderate for the middle part. Does it link to ERD/ERS? For SSVEP, why the attention scores should be different for different parts of the data if the underline signals are the \"steady\" visual responses from the EEG?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ByIFF5Wk-BP", "reviewer": "Reviewer_5euM", "summary": "The paper proposes a deep neural network architecture equipped with a Riemannian manifold attention mechanism, tailored for EEG decoding tasks. Specifically, following convolutional feature extractor layers, learned Euclidean space feature sequences are mapped to an SPD manifold where the attention module operates. Experiments are evaluated on the BCI Competition IV-2a and an SSVEP dataset. Performed model interpretation analyses reveal neurophysiology-consistent predictions performed by the proposed architecture.", "questions": "- The paper needs to expand their analyses both in terms of methods to compare against, as well as the datasets considered. Further supporting experiments on larger MI datasets are necessary (e.g., \"EEG datasets for motor imagery brain-computer interface\". Gigascience. 2017) to show consistency with the BCIC-IV-2a results. \n\n- While the MI-BCI experiments are designed well with cross-session test set separation, the SSVEP dataset uses a held-out experiment block of the same session as the test set, which is not a strong evaluation of generalization performance when it comes to EEG. Due to this limitation, I suggest another supporting time-synchronous EEG decoding experiment to be considered. For instance one could also consider an alternative experimental setup with P300 event-related potential detection experiments?\n\n- Please clarify what is done in the process of \"analysis of model interpretation\"? It is undefined throughout which explanation tools or methods from papers are used for this. Also Figure 3 does not really stand for a strong outcome of the experimental analyses. If one would look into such topographies for the other models, similar neurophysiologically-consistent results can be also observed. It is not clear what is the direct comparison here.\n\n- Discussion of results are very short. Method comparisons are only performed with respect to plain convolutional EEG decoding architectures (also, comparisons to DeepConvNet from [36] are missing?). There are no ablations of parts of the architecture to truly demonstrate its significance. At the most basic level, how relevant is the Riemannian manifold transformation? How does a CNN equipped with an attention module perform without SPD manifold projection? There are existing models exploiting temporal attention mechanisms that one can compare against:\n\"An end-to-end CNN with attentional mechanism applied to raw EEG in a BCI classification task.\" Journal of Neural Engineering, 2021.\n\"An attention-based deep learning approach for sleep stage classification with single-channel eeg.\" IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2021.\n\n- Figure 4 can be simplified by removing half of the Frequency range (y-axis values) from the figures, since the data was already preprocessed with a 4-38Hz bandpass filter? \n\n- Input dimensionality of EEG inputs to the network for both experiments should also be clearly stated, it was not clear.\n\n", "limitations": "Some methodological limitations are addressed in Sec 4.3.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper nicely combines the traditional Riemannian manifold EEG feature extraction methods with a deep attention mechanism. Methodology is explained and illustrated clearly.\n\nWeaknesses: Main limitation of the paper is its shallow experimental evaluations. This makes the proposal of the paper very weak as a general EEG decoding framework.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "72sIF3o1auj", "reviewer": "Reviewer_SYJH", "summary": "The authors submitted a well-written and inspiring paper proposing various attention networks for decoding general types of EEG data. It was a pleasure to read it, and the author presented a solid validation of their submission with two open-source BCI datasets. ", "questions": "Did the authors consider other BCI paradigms (ERP/P300, cVEP, etc.), which are not narrow-frequency bands localized as MI or SSVEP? ", "limitations": "Within-subject training and testing is a classical and limited ML application to BCI. Did the authors consider testing their method in transfer learning or leave-one-subject-out cross-validation setting, in which most BCIs fail these days? ", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed approach is novel, inspiring, clearly explained, supported with supplementary code, and ideally validated. The strength and solid contribution to ML in BCI is a lightweight, interpretable, and robust GDL framework capturing spatiotemporal EEG features across Euclidean and Riemannian spaces.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept