Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This study presented a novel method for the brain-computer interface, where a manifold attention mechanism is applied to Riemannian SPD. The results showed significant improvement over the SOTA methods on two typical datasets (motor imagery and SSVEP). The paper is clearly written with adequate analysis. The explanations provide interesting insights. 
strengths_and_weaknesses: Strengths:
- This study is a new development of geometric deep learning for the brain-computer interface. 
- The results are promising.

Weakness:
- The interpretations should provide more in-depth analysis to provide deep insight. For example, it shows the attention scores are high for the first part of the MI data and moderate for the middle part. Does it link to ERD/ERS? For SSVEP, why the attention scores should be different for different parts of the data if the underline signals are the "steady" visual responses from the EEG?

questions: Q1: The attention scores are highest for the first part of the MI data and moderate for the middle part, are they linked to the ERD/ERS? 
Q2: For SSVEP, why the attention scores should be different for different parts of the data if the underline signals are the "steady" visual responses from the EEG?
limitations: The proposed method provides a novel treatment to BCI with promising performance. More experiments on different datasets should be tested to confirm its generalizability.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The proposed MAtt is a manifold attention-based model to capture EEG representations. I am not an expert in manifold and the 'geometric' in this paper is not have the same meaning in my domain (where geometric learning denotes graph neural networks). Therefore, please decrease the weight of my comments and consider more from other reviewers. 

I will provide the comments as a general researcher with expertise in EEG. 
strengths_and_weaknesses: Strengths:

1. Considered both MI EEG and SSVEP. MI is a passive EEG while SSVEP is an event-evoked EEG. They are two key categories in EEG family. 

Weaknesses:

1. The experimental results in Table 2 are kind of weird to me (lower than expected), so I checked the performances in STOA. The performance of MI-BCI competition IV-2a reported in the literature is higher than that reported in this work. 

For example, [1] provides the accuracy of this dataset ranges from 76% to 79% (higher than the highest value in Table 2). [2] shows the accuracies ranges from 75% to 81%. Please check, compare, and explain. 

[1] S-EEGNet: Electroencephalogram Signal Classification Based on a Separable Convolution Neural Network With Bilinear Interpolation, 2020.
[2] Advanced TSGL-EEGNet for Motor Imagery EEG-Based Brain-Computer Interfaces Lack of STOA baselines, 2021.

2. Lack of SOTA baselines. This manuscript only compared with four baselines which are published in 2017, 2018, 2019, and 2021 (preprint) separately. More recent and powerful baselines are expected to be included in Table 2. 

3. In the Introduction, it is not clear why geometric deep learning is necessary in EEG classification. Manifold-based methods have boost performance in images is not a good idea for using the same methods in EEG. 
questions: See Weaknesses above. 

BTW, It's better to move Sec. 3.5 to Experiments (Sec. 4).
limitations: The authors discussed their limitations in Sec. 4.3.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a manifold attention network (mAtt), that leverages the properties of the SPD manifold in order to perform spatiotemporal analysis on EEG signals. In order to achieve that, the authors introduce a specific covariance matrix representation of the spatial information of the EEG signal. Concretely, they calculate the sample covariance matrix for each of the embeddings derived from the output of two convolutional layers fed with EEG time series. Then, the sequence of the covariance matrices (which are SPD matrices) is fed into a manifold attention layer that applies the attention mechanism on the SPD manifold. 
strengths_and_weaknesses: **Strengths**

The interpretability of the proposed framework is validated by the current neuroscientific knowledge. Concretely, Fig. 3 very nicely demonstrates brain regions corresponding to certain motor imageries of specific body parts. 

Significance:
The proposed framework demonstrates non-trivial advances in terms of performance compared to sota methods.

**Weakness**

Clarity:
Line 114 there seems to be a plus (+) sign missing from the Sym(n) space.
Line 119 A clarification regarding what AIM stands for, followed by the corresponding literature reference, would be useful. (I assume it stands for affine invariant metric)
Fig. 2 More information is required in the caption.
Section 3.3  a very brief explanation of the terms query, key and value would give a nice ‘flow’ to the reading process of the paper, even though these terms have been extensively discussed in the referenced literature. 

Originality:
The idea of representing EEG signals as covariance matrices and then projecting them on the SPD manifold has been widely used. The novelty of the approach lies mainly in the application of the attention mechanism on the SPD manifold. Therefore it would be interesting to see how the model performs when compared with very similar frameworks that exploit attention in different manners, i.e. Zhang, Guangyi, and Ali Etemad. "RFNet: Riemannian fusion network for EEG-based brain-computer interfaces."
questions: The capturing of temporal information is not very clear. How creating covariance matrices of consecutive EEG time-frames is better for capturing temporal features than using simple LSTM or GRU layers that have demonstrated excellent performance on similar tasks? (Zhang, Dalin, et al. "Cascade and parallel convolutional recurrent neural networks on EEG-based intention recognition for brain-computer interface.")

SPD matrices have been widely used in medical data analysis. However, to our knowledge, the criteria for choosing a particular metric on the SPD manifold for a given application are vague. Therefore, it would be nice to see the reasoning behind the choice of the Log-Euclidean metric instead of other alternatives (i.e. affine invariant metric Thanwerdas, Yann, and Xavier Pennec. "Is affine-invariance well defined on SPD matrices? A principled continuum of metrics."). Also, a comparative study demonstrating the advantages of one metric over the others would be an interesting addition to the paper. 

The framework is tested on 9 subjects (asynchronous data) and 11 subjects (synchronous data). However, in order to be able to safely consider its generalization capabilities, it would be nice to evaluate how the framework performs given a bigger multi-subject dataset (i.e. EEG Motor Movement/Imagery Dataset from PhysioNet).

Fig 5 reveals some very interesting insights regarding neural dynamics with respect to the time duration. It would be interesting to see whether they can be verified by current neuroscientific knowledge. 

limitations: Some of the limitations have been addressed in section 4.3.  Potential limitations that have not been addressed by the authors have been extensively discussed in the Questions section of this review.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors submitted a well-written and inspiring paper proposing various attention networks for decoding general types of EEG data. It was a pleasure to read it, and the author presented a solid validation of their submission with two open-source BCI datasets. 
strengths_and_weaknesses: The proposed approach is novel, inspiring, clearly explained, supported with supplementary code, and ideally validated. The strength and solid contribution to ML in BCI is a lightweight, interpretable, and robust GDL framework capturing spatiotemporal EEG features across Euclidean and Riemannian spaces.
questions: Did the authors consider other BCI paradigms (ERP/P300, cVEP, etc.), which are not narrow-frequency bands localized as MI or SSVEP? 
limitations: Within-subject training and testing is a classical and limited ML application to BCI. Did the authors consider testing their method in transfer learning or leave-one-subject-out cross-validation setting, in which most BCIs fail these days?
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a deep neural network architecture equipped with a Riemannian manifold attention mechanism, tailored for EEG decoding tasks. Specifically, following convolutional feature extractor layers, learned Euclidean space feature sequences are mapped to an SPD manifold where the attention module operates. Experiments are evaluated on the BCI Competition IV-2a and an SSVEP dataset. Performed model interpretation analyses reveal neurophysiology-consistent predictions performed by the proposed architecture.
strengths_and_weaknesses: Strengths: The paper nicely combines the traditional Riemannian manifold EEG feature extraction methods with a deep attention mechanism. Methodology is explained and illustrated clearly.

Weaknesses: Main limitation of the paper is its shallow experimental evaluations. This makes the proposal of the paper very weak as a general EEG decoding framework.
questions: - The paper needs to expand their analyses both in terms of methods to compare against, as well as the datasets considered. Further supporting experiments on larger MI datasets are necessary (e.g., "EEG datasets for motor imagery brain-computer interface". Gigascience. 2017) to show consistency with the BCIC-IV-2a results. 

- While the MI-BCI experiments are designed well with cross-session test set separation, the SSVEP dataset uses a held-out experiment block of the same session as the test set, which is not a strong evaluation of generalization performance when it comes to EEG. Due to this limitation, I suggest another supporting time-synchronous EEG decoding experiment to be considered. For instance one could also consider an alternative experimental setup with P300 event-related potential detection experiments?

- Please clarify what is done in the process of "analysis of model interpretation"? It is undefined throughout which explanation tools or methods from papers are used for this. Also Figure 3 does not really stand for a strong outcome of the experimental analyses. If one would look into such topographies for the other models, similar neurophysiologically-consistent results can be also observed. It is not clear what is the direct comparison here.

- Discussion of results are very short. Method comparisons are only performed with respect to plain convolutional EEG decoding architectures (also, comparisons to DeepConvNet from [36] are missing?). There are no ablations of parts of the architecture to truly demonstrate its significance. At the most basic level, how relevant is the Riemannian manifold transformation? How does a CNN equipped with an attention module perform without SPD manifold projection? There are existing models exploiting temporal attention mechanisms that one can compare against:
"An end-to-end CNN with attentional mechanism applied to raw EEG in a BCI classification task." Journal of Neural Engineering, 2021.
"An attention-based deep learning approach for sleep stage classification with single-channel eeg." IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2021.

- Figure 4 can be simplified by removing half of the Frequency range (y-axis values) from the figures, since the data was already preprocessed with a 4-38Hz bandpass filter? 

- Input dimensionality of EEG inputs to the network for both experiments should also be clearly stated, it was not clear.


limitations: Some methodological limitations are addressed in Sec 4.3.
review_title: null

Meta-review: metareview: The paper proposes a manifold attention network (mAtt), that leverages the properties of the SPD manifold in order to perform spatiotemporal analysis on EEG signals. The results showed significant improvement over the SOTA methods on two typical datasets (motor imagery and SSVEP). The paper is clearly written with adequate analysis. The idea of representing EEG signals as covariance matrices and then projecting them on the SPD manifold has been widely used. The novelty of the approach lies mainly in the application of the attention mechanism on the SPD manifold. The authors' responses have successfully addressed some reviewers' concerns and provided additional (supportive) experimental results, which convinced reviewers to update their evaluations. Some minor concerns about why geometric deep learning is necessary in EEG classification and the clarity of the paper could be further improved. The authors are encouraged to take reviewers' detailed comments into account in the final version. 
confidence: Certain
Decision: Accept