CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding

Authors: Jiquan Wang, Sha Zhao, Zhiling Luo, Yangxuan Zhou, Haiteng Jiang, Shijian Li, Tao Li, Gang Pan
Primary area: applications to neuroscience & cognitive science
Keywords: Foundation Model; EEG; Criss-Cross Transformer; Downstream BCI tasks

Abstract:
Electroencephalography (EEG) is a non-invasive technique to measure and record brain electrical activity, widely used in various BCI and healthcare applications. Early EEG decoding methods rely on supervised learning, limited by specific tasks and datasets, hindering model performance and generalizability. With the success of large language models, there is a growing body of studies focusing on EEG foundation models. However, these studies still leave challenges: Firstly, most of existing EEG foundation models employ full EEG modeling strategy. It models the spatial and temporal dependencies between all EEG patches together, but ignores that the spatial and temporal dependencies are heterogeneous due to the unique structural characteristics of EEG signals. Secondly, existing EEG foundation models have limited generalizability on a wide range of downstream BCI tasks due to varying formats of EEG data, making it challenging to adapt to. To address these challenges, we propose a novel foundation model called CBraMod. Specifically, we devise a criss-cross transformer as the backbone to thoroughly leverage the structural characteristics of EEG signals, which can model spatial and temporal dependencies separately through two parallel attention mechanisms. And we utilize an asymmetric conditional positional encoding scheme which can encode positional information of EEG patches and be easily adapted to the EEG with diverse formats. CBraMod is pre-trained on a very large corpus of EEG through patch-based masked EEG reconstruction. We evaluate CBraMod on up to 10 downstream BCI tasks (12 public datasets). CBraMod achieves the state-of-the-art performance across the wide range of tasks, proving its strong capability and generalizability. The source code is publicly available at https://github.com/wjq-learning/CBraMod.

Decision: accept (poster)
Reviewer scores: 6, 8, 5, 8

Reviews and discussion:
<thread>
creation date:
1730391741230

thread_type:
official_review

<thread_object>
creator: reviewer_iyyg

<content>
summary:
* This paper proposes a new LLM model for EEG decoding, primarily trained on the Temple University EEG (TUEG) dataset and tested across different EEG decoding datasets.
* The authors introduce a new way to encode EEG data with various channel configurations, using asymmetric conditional positional dynamic encoding inspired by Chu et al. (2021) with different kernel sizes. They also present a new criss-cross transformer layer with temporal and spatial attention, improving upon the work of Huang et al. (2020) and adapting it effectively to the EEG context.
* The encoder's contribution lies in combining the BioT encoder for inputting frequency information via SFFT and the Labram temporal encoder utilizing Conv2D and GroupNorm.
* The pre-training strategy involves masked reconstruction with random masking of EEG patches instead of masking temporal and spatial components.
* The method is showcased on 12 datasets, spanning a variety of EEG decoding tasks.

strengths:
* Although the general idea of using language models for EEG data is not new, the neural network architecture presents some novelty by incorporating and combining multiple ideas from previous works, such as Patch from Visual Transformer, Criss-Cross Transformer, and Asymmetric Conditional Positional with different kernel sizes. 
* This work presents a large number of experimental results on 12 different datasets and tasks.
* Some of the experimental results are quite encouraging and show incremental results over the Labram and BioT works.
* The use of large language models in the EEG field is relatively under-explored, especially considering the revolution we are experiencing in parallel fields such as audio, NLP, and vision. Bridging the bridge connecting other communities to the EEG community is very refreshing!

weaknesses:
**Originality**

While the particular combination of methods is novel, it primarily combines existing ideas from other deep learning subfields. This limits the overall novelty of the work, especially given the lack of sufficient ablations to discern which of the numerous components is truly important. To be very clear, there is no problem using components from other fields; in my opinion, this is highly encouraged, but the study needs to go in-depth to understand what leads to compatibility and the "bridge construction."

**Clarity**

In my opinion, the paper's writing could be improved. The text needs refinement to better guide the reader through the findings rather than just describing tables. There is no discussion about other papers; while the engineering contributions are clear, the scientific perspective is significantly lacking.

**Major Concerns**

* The choice of datasets for motor imagery, emotion, and sleep stage tasks seems arbitrary. For instance, in the case of motor imagery, the most widely used dataset is BNCI 2014 version 004, and the largest is from Stieger et al. (2021) to the best of my knowledge. However, the authors chose Physionet-MI.
*  The results in Physionet-MI are significantly below the state-of-the-art. A quick search reveals accuracies of 88.6% ± 9.0 using EEGSym, 86.36% with Zoumpourlis et al. (2023), and 73.60% with TIDNet on unseen subjects, where we have 64.17%. While I understand that the exact reproduction of results in EEG decoding is impossible, the reported results here are more than 20% below the state-of-the-art. This concern extends to sleep stage models, where datasets like SleepEDF+, MASS, or SHHS are standard, and accuracies above 80% are expected. The same issue applies to emotion classification; for example, Zhang et al. (2024) achieved better results on the SEED dataset testing with different train-test splits.

* Furthermore, when critically analyzing the model's usefulness based on the metrics—and considering literature from the field, such as O. Alkob et al. (2018) and Combrisson and Jerbi (2015)—the metric values fall short of demonstrating real usefulness for BCI applications.

* The choice of baselines is inadequate and possibly inherits these shortcomings from the BioT and Labram studies. Although the authors cite well-established works in the literature, there is no comparison with neural networks like EEGNet, ShallowNet, EEGConformer, EEGSym, ATCNet, FBCNet, and many others that effectively capture temporal and spatial information in motor imagery tasks. By using neural networks with only frequency-based encoders (e.g., BioT), the spatial and temporal information critical for motor imagery is not learned. Employing weak baselines for classification tasks makes the gains appear larger than they actually are. This issue also applies to the sleep stage task, where models like USleep, DeepSleepNet, SleepTransformer, XSleepNet, SeqSleepNet, ChambonNet, and others should be considered as baselines.

* Using the TEUV and TUAB datasets for abnormal detection and event-type classification while employing TEUG for pre-training leads to data leakage. This is indicated on the Temple University dataset's own webpage: "The TUH EEG Events Corpus (TUEV: v2.0.1): This corpus is a subset of TUEG."

**References**

O. Alkoby, A. Abu-Rmileh, O. Shriki and D. Todder, "Can we predict who will respond to neurofeedback? A review of the inefficacy problem and existing predictors for successful EEG neurofeedback learning", Neuroscience, vol. 378, pp. 155-164, May 2018.

Combrisson E, Jerbi K. Exceeding chance level by chance: The caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy. Journal of neuroscience methods. 2015 Jul 30;250:126-36.

Stieger, J. R., Engel, S. A., & He, B. (2021). Continuous sensorimotor rhythm based brain computer interface learning in a large population. Scientific Data, 8(1), 98.

Zhang, Z., Zhong, S. H., & Liu, Y. (2024). TorchEEGEMO: A deep learning toolbox towards EEG-based emotion recognition. Expert Systems with Applications, 249, 123550.

Pérez-Velasco, S., Santamaría-Vázquez, E., Martínez-Cagigal, V., Marcos-Martínez, D., & Hornero, R. (2022). EEGSym: Overcoming inter-subject variability in motor imagery based BCIs with deep learning. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 30, 1766-1775.

Zoumpourlis, G., & Patras, I. (2024). Motor imagery decoding using ensemble curriculum learning and collaborative training. In 2024 12th International Winter Conference on Brain-Computer Interface (BCI) (pp. 1-8). IEEE.

Kostas, D., & Rudzicz, F. (2020). Thinker invariance: enabling deep neural networks for BCI across more people. Journal of Neural Engineering, 17(5), 056008.

questions:
All the questions are pointed out in the Major Concerns section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730538213535

thread_type:
official_review

<thread_object>
creator: reviewer_m8h6

<content>
summary:
In response to the limitations of existing EEG models, which primarily focus on whole-brain modeling while neglecting spatiotemporal dependencies, as well as the challenges faced by current EEG backbone models in handling EEG data in various formats, this study proposes an EEG foundation model based on a criss-cross transformer backbone. This work involves pretraining on a substantial dataset of EEG signals and fine-tuning on multiple downstream tasks.

strengths:
This work addresses the challenges currently faced in EEG models by proposing an effective solution for spatiotemporal modeling through enhancements in model architecture. The model has undergone pretraining on a substantial dataset, and the experimental results are comprehensive, providing support for several contributions.

weaknesses:
Due to the pretraining paradigm of CBraMod, it has not effectively addressed the challenges posed by the diversity of EEG data formats. This raises concerns regarding its suitability as a foundation model for EEG. The specific issues are outlined as follows.
1. This work acknowledges that current EEG foundation models still face challenges in handling EEG data of varying formats. However, the solution proposed by CBraMod during the pretraining phase, which involves the selection of 19 common EEG channels, appears to be a simplistic approach that does not effectively address the aforementioned issues.
2. To learn generic representations from both time-domain and frequency-domain EEG signals, this study conducted pretraining on the TUEG dataset, which primarily focuses on medical data related to conditions such as epilepsy. However, as a foundation model, CBraMod exhibits significant discrepancies between the pretraining data and the data used for downstream tasks. This raises questions regarding its theoretical validity.
3. The ablation studies related to pretraining, being a crucial experiment, should be included in the main text.

questions:
1. Given that this work is positioned as a foundation model, it is pertinent to inquire whether scaling laws exist at both the data level and the model size level. Have the authors conducted relevant experiments to explore these scaling laws?
2. It would be valuable to examine the loss function curves during the pretraining phase.
3. Could the authors elucidate their perspective on the motivation or value of foundation models within the EEG domain? Additionally, how does CBraMod balance the relationship between model size, computational cost, and performance?
4. During the pretraining phase, 19 common electrodes were utilized. How does the model handle discrepancies in the number of electrodes when applied to downstream tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707627989

thread_type:
official_review

<thread_object>
creator: reviewer_1dyx

<content>
summary:
The paper introduces CBraMod, a novel EEG foundation model that leverages a criss-cross transformer architecture to capture the spatial and temporal dependencies of EEG signals. It employs an asymmetric conditional positional encoding scheme to adapt to diverse EEG formats and is pre-trained on a large EEG corpus. The model demonstrates state-of-the-art performance across 10 downstream BCI tasks, showcasing its capability and generalizability.

strengths:
The paper proposes a novel transformer-based architecture for EEG feature extraction and uses it as a foundation model. Substantial comparative experiments have been conducted to show its good performance on different EEG prediction tasks. The presentation is clear, and superior results have been achieved. It’s a good exploration of using large-scale models to obtain EEG representations.

weaknesses:
The current version shows a good architecture for feature extraction instead of a real foundation model. It would be beneficial to give a clearer description of the motivation for proposing such a ‘foundation model’ and how this model facilitates the application of EEG analysis and even brain-computer interfaces. Could we use only a few data for finetuning or tuning specific model faster?

questions:
1. What features do the time-domain and frequency-domain branches capture? It would be beneficial to illustrate the specific frequency components learned by the frequency-domain branch within the patch encoder.
2. How does the CNN-based position encoder capture positional information? Does this approach primarily leverage local features, and have you compared it with more traditional positional encoding methods? 
3. It would be helpful to provide a comparison of the computation cost of criss-cross attention against other attention mechanisms mentioned in Figure 5.
4. Have you evaluated the impact of data scale for both pretraining and fine-tuning? It would be interesting to know whether the incorporation of a pre-trained foundation model enables effective learning with a reduced amount of data for specific tasks.
5. Many layers are employed, yet the feature maps are not large. Did each layer contribute significantly to the final results?
6. The paper mentions that only 19 channels were used during pre-training, whereas additional channels were introduced in the downstream task. Could you clarify how the balance between channels was managed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711847354

thread_type:
official_review

<thread_object>
creator: reviewer_k8cx

<content>
summary:
The paper introduces CBraMod, a novel foundation model designed for EEG signal decoding within Brain-Computer Interface (BCI) applications. CBraMod employs a novel criss-cross transformer architecture to combine modeling of mutual spatial and temporal dependencies inherent to EEG data. Additionally, the model incorporates an Asymmetric Conditional Positional Encoding (ACPE) scheme to learn to adapt to the diverse EEG data formats across various datasets. CBraMod is pre-trained on the extensive Temple University Hospital EEG Corpus (TUH) and evaluated across ten downstream BCI tasks using twelve public datasets. The results demonstrate that CBraMod achieves state-of-the-art performance, highlighting its strong generalizability and effectiveness across diverse EEG decoding tasks.

strengths:
1. The utilization of a criss-cross transformer to independently model spatial and temporal dependencies is a significant advancement. By partitioning attention mechanisms into Spatial-Attention (S-Attention) and TemporalAttention (T-Attention), CBraMod effectively captures the heterogeneous dependencies in EEG signals, which are often overlooked in traditional full EEG modeling strategies.
2. Asymmetric Conditional Positional Encoding (ACPE):  Dynamic Encoding: The ACPE scheme dynamically encodes spatial and temporal
positional information which may enhance the model’s adaptability to various EEG formats. 
3. Comprehensive Pre-training. Pre-training CBraMod on the TUH dataset, comprising over 27 000 hours of EEG puts this architecture in line with several other top-performing models. 
• Time-Frequency Encoding. Instead of relying on temporal domain convolution the authors decided to explicitly use a tf-decompositon to augment time-domain embeddings

weaknesses:
1. Limited Comparative Analysis with Recent Models. While the paper compares CBraMod with several non-foundation and foundation model baselines, the inclusion of more recent EEG foundation models, e.g. BrainWave (https://arxiv.org/abs/2402.10251) could provide a more comprehensive evaluation of CBraMod’s relative performance.  
2. Lack of interpretability. The authors did not provide any kind of interpretation of the obtained models. First of all, it would be very interesting to see what pieces of EEG (source topographies, frequency bands) contribute to the "powerful embeddings"  learnt by the foundation  model. Secondly, for the majority of the downstream tasks there is well defined defined hypothesis regarding the way the decoded information is encoded in the brain. For example motor imagery classification would require information in the 18-14 Hz and 15-25 Hz range on the sources located on the sensory-motor cortex with very distinct topographies. Addition of these would make the paper more convincing. 
3. Consistency of embeddings. We would expect that the obtained representation would somehow cluster in a mechanistically meaningful way.  For example, I would expect some clustering of the embeddings  with participant's age, gender, the downstream task, etc.  Demonstrating this consistency would significantly strengthen the presentation.
4. Ablation Studies on Architectural Components. although the paper discusses the contributions of the criss-cross attention mechanism and the ACPE scheme, more granular ablation studies isolating each component’s impact on specific tasks would strengthen the claims regarding their individual effectiveness and justify their inclusion in the architecture.

questions:
1. I think I missed this but how exactly the architecture adapts to each of the dataset? I.e. I an interested in a)  purely technical adaptation due to variation in the number of channels and b) more conceptual one and needed due to the variability of the electro-dynamic properties of the head as a volume-conductor resulting in quite different spatial patterns of EEG activity. 
2.  What is the effect of altering the temporal window size (1 s) for the resulting performance? Could it be that different downstream tasks require different values of this important parameter? 
3. How important is the time-frequency-based augmentation of the embeddings? What is the performance gain it brings about?
4. Could the authors provide additional details on whether the entire CBraMod model or only specific layers are fine-tuned for downstream tasks? 
5. Related to  (4). Could the authors be more explicit regarding the strategies to prevent overfitting during fine-tuning on smaller
downstream datasets (e.g., dropout rates, weight decay specifics beyond pre-training)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732085185057

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response and Revision of the Paper

comment:
We thank the reviewers for their insightful feedback. We appreciate the positive comments on the novelty of CBraMod (1DYx, iYYg), the effectiveness of proposed method (k8cX, m8H6), the comprehensive Pre-training (k8cX, m8H6),  the substantial comparative experiments (1DYx, m8H6, iYYg), the superior results (1DYx, m8H6, iYYg), the good exploration for large-scale models to obtain EEG representations (1DYx, iYYg), the time-frequency Encoding (k8cX), and the clear presentation (1DYx).

We acknowledge some reviewers' concerns regarding the interpretability, the performance of low-resource settings, the choise of datasets, the choise of baseline, the motivation of CBraMod and the value to EEG/BCI field.  **Therefore, we have addressed these concerns in the rebuttal and the revised version of our paper, aiming to alleviate the reviewers' concerns through further comparative experiments, interpretability analysis and discussion.** Due to the extensive computational resources and time required for model pre-training, this is the best effort we can make within the short timeframe provided for the rebuttal. 

Here, we will outline the specific revisions made in the revised version of our paper:
1. In response to the suggestions of multiple reviewers, we have enhanced the description of our motivation in the introduction section. **(Line 80-102, Page 2)**
2. As reviewer iYYg suggested, we have added two supervised-learning baselines, EEGNet and EEGConformer, for all the downstream datasets. **(Line 365, Page 7)**
3. As reviewer m8H6 suggested, we have moved the section "Ablation Study on Pre-training" into the main text. **(Line 517, Page 10)**
4. We have added a description in the Conclusion section highlighting the insights our work provides for the BCI field. **(Line 537-539, Page 10)**
5. As reviewer m8H6 suggested, we have added the pre-training loss curve. **(Line 965-988, Page 18-19)**
6. We have added the detailed description for the additional baseline in the appendix. **(Line 1001-1005, Page 19)** 
7. As reviewer iYYg suggested, we have added two baselines for sleep staging task, DeepSleepNet[3] and USleep[4]. **(Line 1092, Page 21)** 
8. As reviewer iYYg suggested, we have added the experiment setup, CBraMod (excluding TUEV) **(Line 1269, Page 24)** and CBraMod (excluding TUAB) **(Line 1309, Page 25)**, for TUEV and TUAB datasets, respectively.
9. As reviewer iYYg suggested, we have added the experiment on BCIC-IV-2a dataset for motor imagery classification. **(Line 1329-1355, Page 25-26)**
10. As reviewer m8H6 suggested, we have added the section "Scaling Data Size and Model Size" in the appendix. **(Line 1360-1391, Page 26)**
11. In response to the suggestions of multiple reviewers, we have added the section "Low-resource Comparison with Existing Methods" in the appendix. **(Line 1494-1525, Page 28-29)**
12. As reviewer k8cX suggested, we have added a more granular ablation studies on criss-cross splited heads in the appendix. **(Line 1526-1549, Page 29)**
13. As reviewer 1DYx suggested, we have provided a comparison of the computation cost of criss-cross attention against other attention mechanisms in the appendix section "Parameters and FLOPs Comparison". **(Line 1634, Page 31)**
14. In response to the suggestions of multiple reviewers, we have added the new section $\textcolor{red}{\textbf{``Interpretability Analysis''}}$ in the appendix. **(Line 1638-1750, Page 31-33)**
15. In response to the suggestions of multiple reviewers, we have added a section "Implication" to discuss the value of our work on EEG and BCI field.  **(Line 1754-1781, Page 33)**
16. As reviewer iYYg suggested, we have added the performance comparison with EEG-SimpleConv[5] under the leave-one-subject-out (LOSO) protocol on the BCIC-IV-2a datasets.  **(Lines 1811-1835, Pages 34)**
17. In response to the concerns of reviewer m8H6, we have added a section "Comparison on Convergence Speed with Supervised Model" in the appendix to demonstrate that CBraMod achieves significantly faster convergence. **(Line 1837-1862, Page 35)**


We hope that our work contributes valuable insights and inspiration to the EEG and BCI research communities, addressing critical challenges in this field and opening new avenues for future exploration. **We sincerely look forward to the reviewers' feedback and constructive insights to further refine and enhance our study!**

### References

[1] EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces[J]. JNE, 2018, 15(5): 056013.

[2]  EEG conformer: Convolutional transformer for EEG decoding and visualization[J]. IEEE TNSRE, 2022, 31: 710-719.

[3] DeepSleepNet: A model for automatic sleep stage scoring based on raw single-channel EEG[J]. IEEE TNSRE, 2017, 25(11): 1998-2008.

[4] U-Sleep: resilient high-frequency sleep staging[J]. NPJ digital medicine, 2021, 4(1): 72.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733207579351

thread_type:
official_comment

<thread_object>
creator: reviewer_k8cx

<content>
title:
E4 - depression classification

comment:
Dear Authors,

Thanks for the very detailed, intensive and breathtaking rebuttal. I have two questions.

1. Section E.4  where you classify depressed vs. normal subjects. What will happen to the performance of all (or some of) the models when you pre-filter the data in the 0.3-30 Hz  range rather than 0.3-75 Hz as you (and perhaps other authors) did in the mental disorder classification task?

2. While I do appreciate you willingness to demonstrate the superiority of your model over other solutions I feel a bit perplexed. Just purely from the statistical reasons, having performed so MANY additional comparisons you demonstrate that  your model beats all the other solutions, don't you think that there should be some tests where some other models perform better?  May be I missed smth.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This works proposes a masked autoencoder strategy for model pretraining on a large collection of EEG data. Evaluation is also quite comprehensive with 10 downstream tasks. Importantly authors engaged with reviewer iYYg who did a very careful check of the reported performances and their comparison with the literature.

The work has convinced that it contributes to the state-of-the-art for the field.

As an AC, I would highly encourage authors to contribute to shared evaluation benchmarks to facilitate tracking the progress of the field on all these tasks that go beyond NLP and computer vision.