Learning Robust EEG Representations with a Large Spatiotemporal Transformer as a Foundation Model

Authors: Liuyin Yang, Marc M. Van Hulle
Primary area: applications to neuroscience & cognitive science
Keywords: Brain-computer interfaces, EEG representations, foundation model, masked autoencoder, spatial-temporal transformer

Abstract:
Electroencephalography (EEG)-based brain-computer interfaces (BCIs) serve many control paradigms by relying on a variety in active brain regions and EEG features. Developing a universal EEG foundation model has been challenging due to the large variety in recording setups and experimental tasks. Additionally, researchers often contend with limited labeled data, making it difficult to utilize large deep-learning models effectively. While there have been successful attempts to develop EEG foundation models, few studies have systematically evaluated their adaptability across diverse BCI control paradigms. To address this gap, we propose a novel, yet simple spatiotemporal EEG transformer (ST-EEGFormer) that projects segments (“patches”) of raw EEG data into an embedding space enriched with a spatial and temporal embedding, allowing the model to effectively handle EEG data exhibiting various channel set-ups and time lengths. To improve data efficiency, we first employed a masked autoencoder (MAE) task to pre-train the ST-EEGFormer in a self-supervised learning manner on a dataset combining six different motor imagery (MI) datasets, a P300 dataset, and a steady-state visual evoked potential (SSVEP) dataset, all of which are public. Next, we benchmarked the pre-trained model, after fine-tuning, on diverse downstream classification tasks. To evaluate the generalization capability, we conducted additional experiments on two public datasets, not used for pre-training: a seizure classification dataset and an online MI BCI dataset. We compared the performance against a simple linear model, EEGNet (a classic CNN-based benchmark model), the state-of-the-art supervised EEG Conformer model, and two foundation models, BIOT and Large Brain Model (LaBraM). The pre-trained ST-EEGFormers could learn robust EEG representations, achieving higher classification accuracies than the benchmarked models across all eight pre-training datasets and exhibiting strong generalization on new datasets with limited training data. Finally, we report several visualizations of the model including the features on which the results are based.

Decision: reject
Reviewer scores: 3, 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729511769194

thread_type:
official_review

<thread_object>
creator: reviewer_st9f

<content>
summary:
This paper proposes a universal spatiotemporal EEG transformer model, which is pretrained on segments of various EEG-based BCI experiment datasets (motor imagery, P300 speller, SSVEP). Pretraining is performed via a standard masked autoencoder (MAE) approach. The model is also shown to generalize well on a diverse set of downstream classification tasks, which were not used in pretraining (e.g., seizure classification).

strengths:
- This work is unique in its collection of a wide range of pretraining datasets on diverse tasks (i.e., both including motor imagery, P300 speller, or SSVEP datasets), without being constrained to the number of EEG sensors.
- The success of generalization experiments on alternative datasets on other tasks (seizure classification, online MI) is a strong point.

weaknesses:
- Methodologically, the work is solely based on the well-known MAE approach, and the main contribution appears to be only empirical.
- Some of the experimental results necessitate further details/clarifications (see questions).

questions:
- It appears like the ST-EEGFormer-large model is often necessary to achieve state-of-the-art performance, in comparison to existing models. To justify and validate this comparison, one should also include the total computational overhead and the training costs of obtaining each of these models (small, base, large) for final use.

- Finger MI classification experiments (Large-MI-5F) demonstrate quite strong results. It would be interesting to see how much of model generalization could be performed on this dataset, if it was not part of the pretraining corpus (i.e., pretraining without Large-MI-5F at all and only testing on this dataset for downstream classification).

- The MAE pretraining strategy should be discussed a bit more in the main manuscript to be self-contained.

- The paper only considers EEGNet and EEG Conformer models as state-of-the-art model examples. However, there has been a wide range of attempts to improve generalization performance in EEG-based BCIs along this line, and the authors should discuss (if not empirically compare against) these relevant pieces of work.

[1] "Learning invariant representations from EEG via adversarial inference."

[2] "Contrastive learning of subject-invariant EEG representations for cross-subject emotion recognition."

[3] "BENDR: Using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data."

[4] "SPD domain-specific batch normalization to crack interpretable unsupervised domain adaptation in EEG."
</content>
</thread_object>

</thread>
<thread>
creation date:
1730233748928

thread_type:
official_review

<thread_object>
creator: reviewer_z5h9

<content>
summary:
This paper addresses the challenges of developing a universal EEG-based brain-computer interface (BCI) by introducing a novel spatiotemporal EEG transformer (ST-EEGFormer). This work combines the objectives of enhancing existing applications in BCI technology and addressing the problem of limited labeled data in EEG research. By proposing a model that effectively handles diverse EEG recording setups and tasks through self-supervised learning, the authors aim to improve data efficiency and classification performance. Additionally, the study emphasizes the generalization capabilities of the ST-EEGFormer across various datasets, thus highlighting its potential impact on the field of EEG research and BCI applications. Overall, the paper seeks to provide both a practical solution for known challenges and a theoretical contribution to the understanding of EEG data representation.

strengths:
A strong point of this work is the demonstrated capability of the pretrained ST-EEGFormer to learn robust EEG representations that achieve higher classification accuracies compared to benchmark models across various pretraining datasets. This indicates not only the effectiveness of the model but also its adaptability, as it exhibits strong generalization on new datasets, even when limited training data is available. Additionally, the inclusion of visualizations of model features enhances the interpretability of the results, providing insights into how the model operates. The commitment to making the pretrained model weights and code publicly available further promotes transparency and encourages further research in the field, enhancing the overall impact of the study.

weaknesses:
One potential weakness of the study is the reliance on self-supervised learning, which may not fully address the challenges posed by the variability in EEG data across different tasks and recording setups. While the ST-EEGFormer shows strong performance, the model's effectiveness could be limited when faced with completely novel or atypical datasets that diverge significantly from those used in pretraining. Additionally, the complexity of the model may result in increased computational requirements, which could pose practical challenges for real-time BCI applications. Finally, while the paper reports improved classification accuracies, it may benefit from a more detailed analysis of the model's interpretability and potential biases in its learning process, ensuring that the results are not only accurate but also reliable across diverse populations and scenarios.

questions:
1- The introduction provides a comprehensive overview of EEG and its applications in BCIs, effectively contextualizing the research. However, there are a few areas that could be improved: First, while it outlines various EEG paradigms and traditional methods, it could benefit from a clearer statement of the specific gap or problem the proposed ST-EEGFormer addresses more explicitly. This would help readers understand the motivation behind the research. Furthermore, the transition between discussing existing methods and introducing the new model feels somewhat abrupt; a smoother connection would enhance the flow of the narrative.

Ican see huge gap here: Consequently, small models that typically rely on convolutional neural networks (CNNs) impose
restrictions on input shape (e.g., the number of EEG channels, and the number of samples), further
complicating the use of different datasets as they usually exhibit different data formats.
Contributions: This paper presents, for the first time, a large transformer-based ST-EEGFormer
model that is channel-aware, flexible, and applicable across various datasets and tasks. 

2- In the related work section, I suggest enhancing the discussion on recent advances in attention-based transformer models that have inspired BCI researchers. Specifically, you could include citations from foundational papers in this area, such as:

"A transformer-based approach combining deep learning networks and spatial-temporal information for raw EEG classification" (IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30, pp. 2126-2136).
"An end-to-end CNN with attentional mechanism applied to raw EEG in a BCI classification task" (Journal of Neural Engineering, 2021, 18(4), p. 0460e3).
Including these earlier works would provide valuable context, allowing you to discuss how they relate to or differ from the current studies. Additionally, a brief exploration of the evolution of transformer-based approaches in EEG classification would enrich the narrative and demonstrate the progression in this research area. This could help clarify the significance of the EEG Conformer model in relation to previous methodologies.

3-It may be beneficial to consider both non-overlapping and overlapping segments during the tokenization process. Non-overlapping segments simplify the model's input structure, potentially enhancing processing efficiency. However, overlapping segments can capture greater temporal context and variations within the data, which may lead to richer feature representations. By incorporating both approaches, the model could leverage the strengths of each method, thereby improving its ability to generalize across different EEG tasks and enhancing overall performance. This dual strategy would also provide more flexibility in handling EEG data, accommodating various recording setups and experimental conditions. I recommend discussing the potential trade-offs between these approaches and comparing their performance specifically in the context of the ST-EEGFormer model.

4- I encourage the authors to include a discussion section in this paper to address the results in depth. This section should explore the implications of the findings, highlight any limitations, and suggest directions for future research. Including these elements will enhance the overall impact of the paper and provide valuable insights for readers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730395466860

thread_type:
official_review

<thread_object>
creator: reviewer_rz9b

<content>
summary:
This paper explores the difficulties associated with creating a universal EEG-based brain-computer interface (BCI) by presenting a new spatiotemporal EEG transformer (ST-EEGFormer). The authors aim to enhance current BCI applications while tackling the issue of scarce labeled data in EEG studies. Their model is designed to effectively manage a range of EEG recording configurations and tasks using self-supervised learning, thereby aiming to boost data efficiency and classification accuracy. The research also highlights the ST-EEGFormer’s ability to generalize across different datasets, underscoring its significant potential for advancing EEG research and BCI technology. Ultimately, this work offers both a practical approach to existing challenges and a theoretical insight into EEG data representation.

strengths:
A key strength of this research is the ability of the pretrained ST-EEGFormer to develop robust EEG representations that yield superior classification accuracies compared to benchmark models across various pretraining datasets. This highlights not only the model’s effectiveness but also its adaptability, as it demonstrates strong generalization on new datasets, even with limited training data. Furthermore, the incorporation of visualizations of model features improves the interpretability of the results, shedding light on the model's functioning. By making the pretrained model weights and code publicly accessible, the study promotes transparency and encourages further exploration in the field, thereby enhancing its overall impact.

weaknesses:
Although the ST-EEGFormer demonstrates strong performance, its effectiveness may be constrained when encountering entirely new or atypical datasets that significantly differ from those used in pretraining. Moreover, the model's complexity could lead to higher computational demands, presenting practical challenges for real-time BCI applications. 
One potential limitation of this study is its reliance on self-supervised learning, which might not fully tackle the challenges arising from the variability of EEG data across different tasks and recording conditions. Lastly, while the paper highlights improved classification accuracies, a more thorough examination of the model's interpretability and any potential biases in its learning process would enhance the reliability of the results across various populations and contexts.

questions:
In the related work section, I recommend expanding the discussion on recent developments in attention-based transformer models that have influenced BCI researchers. Specifically, incorporating citations from key foundational papers in this field would strengthen your analysis.

Including a discussion section is much more beneficial than simply reporting the results. You can put limitations and suggest future directions for research within the same context.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704129982

thread_type:
official_review

<thread_object>
creator: reviewer_qnd3

<content>
summary:
The paper presents ST-EEGFormer, a large transformer-based model designed for learning representations from EEG data.  The model is channel-aware and flexible, capable of handling diverse datasets and tasks through self-supervised pretraining using the masked autoencoder approach.

strengths:
1. Foundation models for EEG (and time series data in general) are not inherently designed to handle varying numbers of channels. The paper solves this issue by using both temporal and spatial positional encodings for each token, which enables the model to work with a variety of different datasets and tasks.
2. Existing foundation models for EEG such as Neuro-GPT [1] and BENDR [2] pre-trained on a single large dataset (TUH) whereas the current paper utilizes a variety of different datasets for pre-training of their model because of its ability to handle varying numbers of channels.

weaknesses:
1. The novelty is not entirely clear. The authors claim that the novelty lies in their model’s ability to project segments of raw EEG data into an embedding space enriched with spatial and temporal embeddings. However, this appears to merge two existing and increasingly popular techniques in time-series research: (1) The paper by Xie et al. [3] published in 2022 uses a very similar method to encode both spatial and temporal position in a single transformer model. (2) Papers such as PatchTST [4] and Neuro-GPT [1] have performed chunking along the temporal axis.

2. While EEG-Conformer is a relevant model to compare with, It would be beneficial to the paper to conduct qualitative and quantitive comparisons with other papers that share the same spirit of foundation models for EEG such as Neuro-GPT [1], BENDR [2], and EEGFormer [5].

3. A major expectation from a foundation model is the ability to generalize on data it hasn’t seen before, i.e. data that it wasn’t pretrained on. The authors have evaluated the model on only two such datasets. It would be helpful to expand this evaluation to include more datasets that are not part of the training corpus, preferably alongside comparisons with other large EEG-models mentioned earlier.

Minor issues:
1. On line 149 of the text, it’s not very clear what the authors mean by “samll patches consist of all channels”. Perhaps, it is “small patches consisting of all channels”.
2. From 426-428, the following sentence is repeated: “However, differences were observed when additional clusters were introduced”.
3. In Table 2, it would be better to have the second-best accuracy underlined instead of bold for better distinction.

questions:
1. Referring to point 1 in Weaknesses, could the authors please elaborate on the novelty of their work?

2. The original EEG Conformer paper reports an accuracy of 78.66% for the BCI Competition IV Dataset 2a, while the current paper reports EEG-conformer’s accuracy for the same dataset as 56%. For BCI Competition IV Dataset 2a, the reported accuracies in both the original (84.63%) and current paper (83.10%) are more or less the same. Can the authors kindly explain the significant difference in the reported accuracy for EEG Conformer on the BCI Competition IV Dataset 2a?

References:
1. Cui, Wenhui, et al. "Neuro-GPT: developing a foundation model for EEG." arXiv preprint arXiv:2311.03764 (2023).
2. Kostas, Demetres, Stephane Aroca-Ouellette, and Frank Rudzicz. "BENDR: Using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data." Frontiers in Human Neuroscience 15 (2021): 653659.
3. Xie J, Zhang J, Sun J, Ma Z, Qin L, Li G, Zhou H, Zhan Y. A Transformer-Based Approach Combining Deep Learning Network and Spatial-Temporal Information for Raw EEG Classification. IEEE Trans Neural Syst Rehabil Eng. 2022
4. Nie, Yuqi, et al. "A time series is worth 64 words: Long-term forecasting with transformers." arXiv preprint arXiv:2211.14730 (2022).
5. Chen, Yuqi, et al. "EEGFormer: Towards transferable and interpretable large-scale EEG foundation model." arXiv preprint arXiv:2401.10278 (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705369751

thread_type:
official_review

<thread_object>
creator: reviewer_abb7

<content>
summary:
The paper proposes a spatio-temporal EEG transformer to build an EEG foundation model. The model is pretrained using a masked autoencoder (MAE) on various public datasets and demonstrates strong generalization on new datasets, outperforming existing work such as EEGNet and EEG Conformer in various classification tasks.

strengths:
1. The authors conducted model pretraining using very large amounts of data from 8 EEG datasets with various tasks. 
2. The proposed network effectively captures both temporal and spatial information from EEG data.
3. The method achieves state-of-the-art performance across multiple datasets.

weaknesses:
Overall, it is difficult to evaluate the contribution of this work due to (1) limited novelty and (2) missing comparison with recently published work. The proposed architecture is similar or a simple version of LaBraM [2] Please see details as follows:   

1. Questionable Contribution Claims: The authors make several problematic claims regarding the novelty and contributions of the proposed ST-EEGFormer. For example, in line 134-135, '...However, since their encoder module works with patches of EEG signals in fixed channels and time length, transferring between datasets remains unrealistic and challenging', this statement on current methods not being able to adapt to different datasets due to their reliance on fixed channels and time lengths is NOT true. The authors fail to mention recent published work, such as BIOT [1] (NeurIPS' 23) and LaBraM [2] (ICLR' 24), which were explicitly designed to handle diverse EEG datasets with varying channels and lengths. Therefore, the contribution claim 'This paper presents, for the first time, a large transformer-based ST-EEGFormer model that is channel-aware, flexible, and applicable across various datasets and tasks' (line 079-080) is inaccurate and misleading.

2. Inappropriate Statements in Contribution Section: In line 086, ' Upon acceptance of this paper, the code and pretrained model weights will be made open-source', this statement should NOT be in the Contribution section, as these resources are not currently available to  the EEG community or to reviewers who are evaluating the work. Similarly, in line 087-089, the authors claim 'We believe that the demonstrated effectiveness of the proposed ST-EEGFormer model can serve as a pioneering effort, encouraging further studies on developing better end-to-end EEG-BCI foundation models to accelerate BCI development and neuroscience research.' , which would be better suited to the discussion section rather than contributions.

3. Lack of Comparison with Closly-related Recent Work: This manuscript fails to conduct comparisons with recently published work, including BIOT [1] (NeurIPS' 23) and LaBraM [2] (ICLR' 24), which have made their code and pretrained weights publicy available, making direct implementation and evaluation easy. Please see the pretrained weights here: 
BIOT [1] pretrained weights: https://github.com/ycq091044/BIOT/tree/main/pretrained-models
LaBraM [2] pretrained weights: https://github.com/935963004/LaBraM/tree/main/checkpoints


[1] Yang, Chaoqi, M. Westover, and Jimeng Sun. "Biot: Biosignal transformer for cross-data learning in the wild." Advances in Neural Information Processing Systems 36 (2024).

[2] Jiang, Wei-Bang, Li-Ming Zhao, and Bao-Liang Lu. "Large brain model for learning generic representations with tremendous EEG data in BCI." arXiv preprint arXiv:2405.18765 (2024).

questions:
The authors should provide more details on the pretraining setup, including GPU usage and time used for pretraining.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732198660093

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overview of the revised manuscript

comment:
We sincerely thank the reviewers for their detailed feedback on our manuscript. We carefully considered all comments and revised our manuscript accordingly. Below, we detail how we addressed the reviewer comments:

Corrections and improvements:
o Corrected a mistake: the SSVEP dataset benchmarked in our study is the BETA dataset. (Liu, B., Huang, X., Wang, Y., Chen, X., & Gao, X. (2020). BETA: A Large Benchmark Database Toward SSVEP-BCI Application. Frontiers in Neuroscience, 14, 544547. https://doi.org/10.3389/fnins.2020.00627)

o Fixed typos and formatting inconsistencies throughout the manuscript for improved readability.

Inclusion of related studies:
o Added discussions on several additional related works, particularly focusing on similar EEG foundation models, to provide a broader and more comprehensive context for our study.

o In line with common practice, the Related Works section has been moved to the first chapter in the Appendix.

Comparison with new relevant models:
o Conducted thorough benchmarking of two recent foundation models, BIOT and LaBraM, using the same datasets as for our model.

Addition of a discussion section:
o Added a discussion section on the results of our findings, limitations of our study, and potential future research directions.

Addition of practical details:
o Included a new section in the Appendix providing practical details about pretraining the model, including GPU usage, training time, and the number of CPU cores utilized.

Major modifications are highlighted in yellow so that they can be easily spotted. We believe these enhancements significantly strengthen the manuscript by addressing concerns about novelty, by expanding the comparative evaluation, and by improving its structure.

We look forward to your feedback and suggestions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732320379994

thread_type:
official_comment

<thread_object>
creator: area_chair_anvh

<content>
title:
Rebuttal

comment:
Dear Reviewers,

I encourage you to review the rebuttal and reach out to the authors with any additional questions or requests for clarification.

Best,\
AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1732545260614

thread_type:
official_comment

<thread_object>
creator: area_chair_anvh

<content>
title:
Discussion

comment:
Dear Reviewers,

As we approach the end of the discussion period, I would like to encourage you again to review the rebuttal and engage with the authors if you have any additional questions or need further clarification. If you have no further questions, please acknowledge that you have received and reviewed the rebuttal.

Best,\
AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces ST-EEGFormer, a spatiotemporal transformer-based EEG foundation model trained using a self-supervised masked autoencoder for EEG. The paper has a number of notable strengths. First, it can handle diverse EEG configurations and tasks effectively with its use of temporal and spatial positional embeddings. It performs well on unseen datasets, demonstrating robustness with limited labeled data. The benchmarks used in the paper include diverse BCI applications, outperforming baselines like EEGNet and EEG Conformer. The pretrained weights and code are publicly available (upon publication). 

However, the paper also has a few shortcomings. First, it relies heavily on existing MAE and transformer methodologies, with minimal architectural innovation. Moreover, the benchmarking could have been more comprehensive, including additional key models such as Neuro-GPT and BENDR. Lastly, the key issue with the paper is its limited novelty following a large number of existing foundation models already available at this stage. When asked about this, the authors responded "we believe our proposed model distinguishes itself through its simplicity in both architecture and training process". I find this argument quite unconvincing. It should be clearly discussed, and backed by empirical experiments, that "how" exactly the model is more "simple", and more importantly, "why" is this simplicity important/useful.