EEGPT: Pretrained Transformer for Universal and Reliable Representation of EEG Signals
Authors: Guangyu Wang
Wenchao Liu
Yuhong He
Cong Xu
Lin Ma
Haifeng Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Electroencephalography
self-supervised learning
representational learning
masked autoencoder
transformer
brain-computer interfaces
Primary area: neuroscience_and_cognitive_science
TL;DR: EEGPT, a novel over 10-million-parameter pretrained transformer for universal EEG feature extraction,  achieving SOTA on various tasks through efficient, robust, and scalable representation learning.

Abstract: Electroencephalography (EEG) is crucial for recording brain activity, 
  with applications in medicine, neuroscience, and brain-computer interfaces (BCI). 
  However, challenges such as low signal-to-noise ratio (SNR), high inter-subject variability, and channel mismatch complicate the extraction of robust, 
  universal EEG representations. 
  We propose EEGPT, a novel 10-million-parameter pretrained transformer model designed for universal EEG feature extraction. 
  In EEGPT, a mask-based dual self-supervised learning method for efficient feature extraction is designed. 
  Compared to other mask-based self-supervised learning methods, 
  EEGPT introduces spatio-temporal representation alignment. 
  This involves constructing a self-supervised task based on 
  EEG representations that possess high SNR and rich semantic information, 
  rather than on raw signals. 
  Consequently, this approach mitigates the issue of poor feature quality typically 
  extracted from low SNR signals.
  Additionally, EEGPT's hierarchical structure processes spatial and temporal information separately, 
  reducing computational complexity while increasing flexibility and adaptability for BCI applications. 
  By training on a large mixed multi-task EEG dataset, we fully exploit EEGPT's capabilities.
  The experiment validates the efficacy and scalability of EEGPT, 
  achieving state-of-the-art performance on a range of downstream tasks with linear-probing.
  Our research advances EEG representation learning, offering innovative solutions for bio-signal processing and AI applications.
  The code for this paper is available at: https://github.com/BINE022/EEGPT

EEGPT: Pretrained Transformer for Universal and Reliable Representation of EEG Signals
Guagnyu Wang Faculty of Computing Harbin Institute of Technology wangguangyu@stu. hit edu.
Wenchao Liu Faculty of Computing Harbin Institute of Technology 23b903096astu hit edu. cn
Yuhong He
Cong Xu Faculty of Computing Harbin Institute of Technology congxuGhit. edu. cn
Faculty Computing Harbin Institute of Technology 1950030020stu hit edu. cn
Lin Ma
Haifeng Li Faculty of Computing Harbin Institute of Technology lihaif fenghhit. edu, cn
Faculty of Computing Harbin Institute of Technology malin lan__i@hhhi li@hit. edu. cn
Abstract
Eleetroencephallggrpphy (EEG) is crucial for recording brain activity, with applica- tions medicine, neuroscience, and brain-computer interfaces (BCI). However, challenges such low signal-to-noise ratio (SNR), high inter-subject variabil- ity, and channel mismatch complicate the extraction of robust, universal EEG representations. We propose EEGPT, novel 10-million- partmmeerr pretrained transformer model designed for universal EEG feature extraction. In EEGPT, a mask-based dual self-supervised learning method for efficient feature extraction is designed. Compared to other mask-ban based self-supervised learning methods, EEGPT introduces spatio-temporal representation alignment. This involves con- structing self-supervised task based on EEG representations that possess high SNR and rich semantic information, rather than on raw signals. Consequently, this approach mitigates the issue of poor feature quality typically extracted from low SNR signals. Additionally, EEGPT's hierarchical structure processes spatial and temporal information separately, reducing computational complexity while increasing flexibility and adaptability for BCI applications. By training on a large mixed multi-task EEG dataset, we fully exploit EEGPT's capabilities. The experiment validates the efficacy and scalability the-art performance
EEGPT, achieving state-of- range of downstream tasks with linear- linear-probing, Our research advances EEG representation learning, offering innovative solutions for bio-signal processing and AI applications. The code for this paper is available at: https: htt:://gtttbbbbuu com/ Btt///INEEO2//// EEGPT
Introduction
Electroencephalography (EEG) dynamically reflects the brain's 's functional state by recording electrical signals from the cerebral cortex 11. EEG essential for studying brain activity and is pivotal in
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

brain-computer interface (BCI) applications due to its non-invasive and portable nature [2] Despite its potential, EEG-based methods face practical challenges due to low signal-to-noise ratio (SNR)
3. high inter-subject variability, and significant task-dependent variations in EEG signals [4] Self-supervised learning, as described by Yann LeCun in his AAAI 2020 keynote [5] has shown
advantages in natural language processing (NLP) [6], computer vision (CV) 7 g and speech analysis
[9]. More and more state-of-the-art (SOTA) models are pretrained by self-supervised learning on
large datasets and fine-tuned for specific applications, effectively reducing the need for extensive labeled data. Masked autoencoders, a type of self-supervised learning method, have been successful in NLP 6 and CV [8] by recovering masked patches based on context.
Recent advances in EEG analysis using self-supervised learning techniques have shown promising results. In Falck et al. [IO), framework for learning EEG representations through contrastive learning was proposed. This framework extends the SimCLR framework to time series data, training a channel feature extractor. The model achieved an accuracy of 85. on the Sleep-EDF dataset [1] but was only trained on EEG data collected during 20-second tasks, making it unsuitable for shorter tasks such motor imagery. BENDR [12 applied self-supervised learning based on masked autoencoders and contrastive learning to EEG data. This approach addresses the challenges of multi-task and multi-paradigm EEG data characterization, enhancing the model's universality. BENDR uses convolutional encoder to extract features from local time windows, masks some features, and then transformer decoder predicts the information in the masked parts. EEG2VEC [13] introduced self-supervised model that learns EEG representations based on contrastive loss and reconstruction loss. The pretrained model is used BENDR utilize convolutional neural network and transformer networks to learn local and global features. EEG2VEC validated in EEG
feature extractor for downstream tasks. Both EEG2VEC and
and EEG regression tasks of the auditory EEG challenge [1]. The Biosignal Transformer (BIOT) [15] model addresses the challenges of cross-data learning with mismatched channels, variable lengths, and missing values in biosignals such as EEG, ECG, and human activity sensory signals. BIOT tokenizes each channel separately into fixed-length segments containing local signal features and then re-arranges the segments to form long "sentence". In the CHB-MIT seizure detection task [16]. the pretrained BIOT models achieved a 4% improvement. The Large Brain Model (LaBraM) [17 addresses EEG-based deep learning model limitations by enabling cross doosset learning. It segments EEG signals into channel patches and uses vector-quantized neural spectrum prediction for training a neural tokenizer. This tokenizer encodes raw EEG patches into neural codes, which pretrain transformers to predict the original neural codes for masked patches. LaBraM outperformed SOTA methods in abnormal detection, event type classification [1]) emotion recognition [9]. and gait prediction 22).
Universal models have clearly made progress in EEG data analysis. However, the extremely low SNR of EEG signals and the complexity of brain activities during tasks make it challenging to learn abstract features using masked autoencoders, which are commonly used in NLP and CV [12). Additionally, the inconsistent sampling rates of different EEG acquisition devices and variations in electrode channel locations (15) hinder the convolutional encoder's ability to decouple the correlation between electrode channels and EEG signals, resulting in robustness and scalability issues.
To address these issues, we propose dual self- self-supervieed EEG universal representation method based on the spatio-temporal consistency of EEG signals 211. introducing the EEG Pretrained Trans- former (EEGPT) for efficient feature extraction. Our method includes spatio-temporal representation alignment and mask-based reconstruction, enhancing representation quality and model robustness. Our method adopts BERT-style masked recovery task (22) (not autoregressive task) objectives for pretraining. Beyond the original waveform recovery, we align the predicted EEG signal features of the masked parts with full EEG signal features. Additionally, a local spatio-temport embedding method improves compatibility across different EEG acquisition devices.
EEGPT, with over 10 million parameters, is pretrained on mixed multi-task EEG dataset, including data from PhysioMI (23), HGD (24), and M3CV 255 This pretraining enables EEGPT to extract universal representations for various tasks. For downstream tasks, we employ linear-probing method, which achieves SOTA performance while also reducing computational resource consumption and preventing overfitting. Our experiments demonstrate EEGPT's superior performance in motor imagery classification (26). event-related potential (ERP) detection [27). and sleep stage detection (28). showcasing its capability to extract high-level abstract features across spatio-temporal dimensions.
2

---

Contributions of this paper:
Proposal of EEGPT, 10-million-panne partmmetrr model for EEG universal feature extraction,
leveraging a mixed dataset to enhance performances across tasks and subjects.
Development of a dual self-supervised method for EEG signals, combining spatio-temporal representation alignment and mask-based reconstruction, improving feature quality and convergence.
Design of a hierarchical structure for decoupled processing of spatial and temporal in- formation, reducing computational complexity and enhancing model flexibility for BCI applications.
Implementation of local spatio-temporal embedding method, increasing robustness and compatibility across different EEG acquisition devices.
Conduct of comprehensive experiments on downstream datasets, demonstrating EEGPT significantly outperforms existing models across multiple EEG tasks and that larger models exhibit improved performance.
2 Method
Background:
According to Kong and Zhang [29], a masked autoencoder learns features through form of denoising autoencoder: input signals occluded with random patch masks are fed into the encoder, and the decoder predicts the original embeddings of the masked patches:
min E 7 (dæ(z), x (1 M)), 2 fo(æ @ M)
(1)
0.0 x>
where "@" denotes element-wise product; M is the patch mask; fo(-) and d.(-) are the encoder and decoder, respectively; is the learned representation; and H(- is the similarity measurement. By minimizing the loss function, the model learns the optimal representation of the input signal. However, in practice, there is no explicit representation z (no split encoder and decoder) in the BERT-style model [22], and the model must be fine-tuned to locate effective representations. In contrast, we add a spatio-temporas representation alignment branch to explicitly represent 2, which changes Equation to Equation (dual self self-supervised method):
min E 7 (d+(z), (1 M)) H(z fo())) fo M) 0.6 E~D
This method encourages the encoded representations to take on larger extent of semantics, similar to the minimal sufficient representation in the Multi-Vier Entropy Bottleneck (MVEB) approach 300, thereby improving the encoding quality and generalization 31].
The structure of the EEGPT model is shown in Figure which includes operations such as patching, embedding, masking, encoder, predictor and reconstructor. First, the model chunks the input EEG signal æ E IR Mx XT (M channels and T time points) into patches Pi.j. and embeds each patch as a token tokeni,j by local apatio-temporal embedding (Section 2.3) followed by splitting into masked parts M and unmasked parts M, respectively. Then, we pretain the model using a dual self-supervised learning method, including spatio-temporal representation alignment (Section 2.1 and mask-based reconstruction (Section 2.2 Finally, the lincar-probing method is used in downstream tasks (Section 2.4).
EEG Pretrained Transformer (EEGPT):
2.1 Spatio-temporal Representation Alignment
The spatio- meatio-eempora representation alignment method aligns the predicted features with the momen- tum encoder's output, enhancing the encoder's ability extract robust features and ensuring that the encoder's 's output contains high-quality global features. We employ the encoder and the predictor to decouple spatial and temporal features, significantly reducing computational complexity.
Encoder: The encoder integrates spatial information from masked patches. Equation 3 describes how the encoder (ENC) processes all masked token,,j at time input and produces the corresponding output feature ency:
(tokensss) (t.))MM
enc, = ENC
(3)
3

---

BezedraseefTT:
recaptal, pond
Racreirsce
pig
pood pww.ree
Tracstutiee
SeNt
Alegeses Logs
quarry
Podlicur
Tokus
Escodar
Mecraciari Eraodde
Tranforter
Exbeediing
Tallnoo
Polcted EBG
Figure 1: The EEGPT structure involves patching the input EEG signal as Ps,j through masking (50% time and 80% channel patches), creating masked part M and unmasked part M and then embedding tokens.j by local spatio-temporal embedding. The encoder processes the masked part, extracting features (eney) consisting of {er ) for each time segment in the M part with summary tokens {s. The predictor predicts features (pred,) for all time segments, aligning with the Momentum Encoder output (mency). ). Based on features extracted by the predictor and encoder, the reconstructor generates rec;,j to reconstruct the EEG signal of the M part.
Predictor: As in Equation the predictor (PRED) utilizes features ency of the masked part from the encoder, combined with temporal position information posj. to predict the complete encoded feature. Embedding and unembedding 332 are performed linearly on the input and output. We adopt the rotary position embedding method 33 to generate posj. introducing relative positional and temporal information. To generate prediction features belonging M, learnable vector query is used as the query token. Through self-supervised training, the encoder is encouraged to extract more information about the correlation among tokens:
(Lene; + pos;} +onec++pss)) 35.(1.))MM
{prede} = PRED (E........N)
Momentum Encoder: The structure of the momentum encoder is identical to that of the encoder. Equation 5 outlines how the momentum encoder (MENC) processes all token,,y at time input and produces the corresponding output mency. After each training iteration, the parameters of the encoder are accumulated into the momentum encoder with factor of 0.01.
menc; = MENC
(tokensa)
(5)
((.))MUMM)
We employ an alignment loss based on Mean Square Error (MSE) 34 to achieve spatio-tempont representation alignment:
N N
L, =
predy LN(mnn,,
(6)
In Equation LN denotes layer normalization which helps to mitigate the effects of extreme values and covariate shift, allowing the model to focus on more important features.
2.2 Mask-based Reconstruction
The mask-based based reconstruction method aligns the reconstructed patches generated by the reconstructor with the raw patches Pi,j in the M part.

---

Reconstructor: As shown in Equation 7 the reconstructor (REC) utilizes features enc, from the M part encoded by the encoder and features pred; of the M part predicted by the predictor, along with temporal position posj, to generate the reconstructed patch recu,t- We establish a "skip connection" between the encoder and the reconstructor to help maintain the features and accelerate convergence.
(rec,,)) = REC (encc + poss} U (pred, + pos;}
(u.))MM
31.(1.MMMM Vi.(..)MM
Mask-based reconstruction is achieved using a reconstruction loss based on the Mean Square Error (MSE):
CR =
Σ
LN(p,,,)
(8)
M
(K
The complete pretraining loss L is constructed by summing both LA and Ln:
== LA Ln
(9)
2.3 Local Embedding
Charnel Embedding
Pitching
pethhpt
patchpss
B30
patchp
Embel(pu)
Tobens
Figure Illustration local spatio-temporal embedding. The EEG signal is divided into equally sized patches in the spatio-temporal dimensions. Each patch represents time segment for specific channel without overlap. The patches are linearly embedded and incorporated with channel embedding information to obtain corresponding feature.
The local spatio-temporal embedding method first patches and embeds the EEG signal in the spatio- We denote the set of EEG signal channels as { where o; is the name of each channel. Firstly, the EEG signal is divided into equally sized patches in the spatio-temporal dimensions, denoted as pajj 21,2, ..., M M).j. E (1, 2....NN.
temporal dimension before feeding it into the encoder, as shown in Figure 2
Pi.j 15.0--1)dd
(10)
where represents patch's time length, and = T/d is the number time patches. Next, the patches linearly embedded, combining the channel embedding information. We construct Codex book
{sa E RR= M (dg is the embedding dimension) containing all learnable channel embedding vectors and a mapping from channel names to channel embedding vectors R. [c - { This mapping flexibly corresponds the channels the EEG data the channels of the model inputs, allowing the model to adapt multiple datasets and improve channel adaptation. The patch's content linearly embedded as Embee((p.s) W Pi.j + by, where W, E Idxd, and b, Rd. are leamable parameters. Embedded token denotes as token., Idde;
36
token,, Embed(p..)) 53
(11)
Based on self- self-supervised learning task, the extracted features of patches are mutually predictable and can ignore noise signals at smaller scales. The method aims to extract macroscopic features that span larger scales, which are believed to be more casily recognizable and considered meaningful features.

---

Logib
Encoder (Fozee))
Erbedd Tokers
dagt Spataa Filter
M
pqr
B56
Figure 3: Linear- probing method.
Figure 4: Electrode locations.
2.4 Linear-Probing Method
In the downstream tasks, we apply the pretrained encoder and concatenate additional modules to solve classification tasks. As shown in Figure 3 we introduce the linear-probing method, which freezes the parameters in the pretrained model and only changes the parameters in the additional linear modules. These modules include adaptive spatial filters (1 x convolution) for aligning channels between EEG and the model, and linear layer map the features logits. The encoder passes the output tokens corresponding to summary tokens to the linear classification head. This approach helps avoid the overfitting problem when large parameter model fine-tuned using a limited number of samples. Since the additional module is straightforward, the performance is solely determined by the encoder, allowing us assess the model's capability.
Experiments
3.1 Datasets and Data Processing
Table 1: Datasets for pretraining and downstream tasks
Datasets
Paradigms
Subjects
Targets
PhysioMI
MI&ME
109
5
HGD
MI
14
pretraining Datasets
TSU
SSVEP
35
40
SEED
EMO
15
3
M3CV
MULTI
106
BCIC-2A
MI
10
4
BCIC-2B
MI
Sleep-EDFx
SLEEP
197
5
Downstream Datasets
KaggleERN
ERN
26
2
PhysioP300
P300
9
2
TUAB
Abnormal
2383
TUEV
Event
288
6
We curated EEG public datasets of various paradigms for model pretraining as shown in Table [
It
contains motor imagery (MI) and execution (ME) datasets PhysioMI [23], HGD 24 steady-
visual evoked potential (SSVEP) dataset TSU [37]. emotional classification dataset SEED [38 and multi-subject
multi- multi-parade paradigm dataset M3CV 25). To assess the practical utility of
the learned representations in downstream tasks, we curated a list of datasets shown in Table I contains MI datasets BCIC-2A (39), BCIC-2B [40], sleep stage detection dataset Sleep-EDFx
11 error related negativity (ERN) dataset KaggleFRN 41 and event-related potentials dataset
PhysioP300 23 We also curated dataset of abnormal EEG signals dataset TUAB and event
type classification dataset TUEV from Temple University EEG Corpus (18). These diverse datasets

---

enable comprehensive evaluation of the proposed EEGPT model across various tasks. Each dataset underwent similar and distinct preprocessing steps. including cropping (4s), re-referencing (average), channels selecting, scaling (mV) and resampling (256Hz). There are 0-38Hz bandpass filtering in MI datasets for downstream tasks. More details refer to Appendix C
3.2 Implementation and Settings
Model implementation. The implementation of encoder, predictor and reconstructor in EEGPT adopts the vision transformer (VIT) [42] and sets S learnable summary tokens (similar to [CLS] token) for summarizing information within the same time patches. We used 58 electrodes (M = 58), as shown in Figure 4 The input signal has a sampling rate of f. = 256 Hz, and the input signal time length is T 1024. Each patch has time length of d = 64, corresponding to a 250ms time window. The 50% time and 80% channel patches of the patches are masked during training.
pretraining strategy. In pretraining, for each training dataset, we randomly sampled 10% of the samples as the validation set. As shown in Table we trained 8 variants with different embedding dims, layers of transformer models (encoder, predictor, reconstructor) and summary tokens S. The AdamW optimizer was employed with the OneCycle learning rate strategy [43) (initial learning rate of 2.5e-4, maximum of 5e-4, minimum of 3.13.. The training was conducted for 200 epochs, with a batch size of 64 and 16-bit mixed precision training on 8 Nvidia 3090 GPUs.
Evaluation strategy. For the data splitting of TUAB and TUEV, we strictly follow the same strategy as BIOT 15] to compare all methods fairly. other downstream tasks, we use the same experimental configuration as BENDR [2]. in which used. Specially, KaggleERN uses 4-fold cross-validation with 10 subjects for testing. Dleep-EDFx uses 10-fold cross- validation and ratio 6:2:2 in training, validation and test splitting. We use linear- probing method for downstream tasks. Particularly, the sleep stage detection task, we use a 4-layer transformer encoder model as classifier that integrates the output of our model for every 0.25s for the purpose of processing a long task of 30s. We used the optimal large model in Section 3355] for testing on all downstream tasks. To ensure the reliability of the experiments, repeated each experiment three times and calculated the standard deviation.
One-Snnject-OOO Out (LOSO) validation method are
Baselines & Metrics. For the TUAB and TUEV datasets, we use the same baselines from BIOT which are fully fine-tuned models. In other tasks, we use the pretrained BENDR [2], BIOT 15 and LaBraM [17] as the baselines. The following metrics are used for comparison: 1) Balanced Accuracy (BAC), 2) AUROC, 3) Weighted FI, 4) Cohen's Kappa. We use AUROC only for binary classification tasks and Weighted Fl only for multi-class classification tasks. More details refer to Appendix D
3.3 Downstream Experiment Results
Table 2:
The results of different methods on TUAB.
Methods
Model Size
Balanced Accuracy
AUROC
SPaRCNet
44
0.79M
0.789620.0011
0,867620.0012
ContraWR
45
1.6M
0.7746#0.0041
0.845640.0074
CNN-T 46
3.2M
0.7777+0.0022
0.8461#0.0013
FFCL 47
2.4M
0.7848+0.0033
0.856940.008
ST-T
48
3.5M
0.796640.000 .0223
0,8707+0.0015
BIOT [15]
3.2M
0.7959+0.00%
0.881 5+0.0043
Ours-Tiny
4.7M
0.7959=0.002
0.87 18400001
Ours
25M
77983+0.0030
We conducted a comparative analysis with other large models on the Temple University datasets, using the same configuration as BIOT [15] for experiments the TUAB and TUEV datasets. The results are presented in Tables 2 and 3 In the TUAB dataset, the performance of EEGPT is comparable to that of the BIOT model. In the TUEV dataset, EEGPT improves the balanced accuracy by 9.5%, and the weighted Fl score by 6.9% compared to BIOT.
To further validate the effectiveness of our model, we conducted comparative experiments with BENDR, BIOT, and LaBraM. The results are shown in Table 4 On the BCIC-2A and BCIC- 2B datascts for motor imagery tasks and the Sleep-EDFx dataset for sleep stage detection, our
7

---

Table 3: The results of different methods on TUEV.
Methods
Model Size
Balanced Accuracy
Weighted F1
Cohen's Kappa
SPaRCNet
0.79M
0.4161#0.0262
0.702420.0104
0.423520:0181
Contra WR 45
1.6M
0.4384+0.0344
0.6893=0.0138
0.3912+0.0237
CNN-T
46
3.2M
0,4087=0.0161
0.6854+0.cm 02293
0.3815#0.0134
FFCL
2.4M
0.3979#0.0104
0.6783#0.0120
0.3732+0.0188
ST-T
48
3.5M
0.3984+0.0228
0.6823+0.00 .0190
0.3765#0.0306
BIOT
[15
3.2M
0.528 1
0.7492+0.000 00822
0.5273+0.024
Ours- Ours-Tiny
4.7M
0.5670+0.0000 0006
0.7535+0.000%
0,5085#0.0173
Ours
25M
0.6232+0.0114
0.818740.0063
0.6351+0.0134
Table 4: The results of universal EEG models on various datasets.
Datasets
Methods
Balanced Accuracy
Cohen's Kappa
Weighted FT AUROC
BENDR
0.4899 0.0000
0.3199+0.0000
0.4836#0.0076
BIOT
0,4590+0.0199
0.2787=0.00 12261
0.4282+0.0288
BCIC-2A
LaBraM
0.561340.00
0.4151+0. 0.0069
0.5520+0.0052
Ours
0.5846+0.0077
0.4462+0.0094
0.5715#0.0051
BENDR
0.7067+0.0011
0.4131+0.000
0.7854+0.000 0002
BIOT
0.6409#0.011
0281770.0220
0.7095#0.0144
BCIC-2B
LaBraM
0.6851#0.000 0003
0.3703+0.01
0.7576+0.0067
Ours
0.7212+0.0019
0.442620.0037
0.8059#0.0032
BENDR
0.6655+0.000 0043
0,6659#0,00 00043
0.7507#0.0029
BIOT
0.6622+0.00 3010
0.6461 +0.017
0.7415
Sleep-EDFx
LaBraM
0,6771#0.008
0.6710ł0.0000
175922+00000
Ours
0.6917+0.000.
0.6857+0.0019
0.7654+0.0023
BENDR
0.56724 0020
1.1461=0.000
0,6030=000004
BIOT
0.5118+0.00%
0.0297 .20244
0.5495+0.016
KaggleERN
LaBraM
0.5439+0.000
0.0944+0, 00066
0.5693+0.00 00552
Ours
0.5837+0.0066
0.1882+0.0110
0.6621#0.0096
BENDR
0.6114+0.0118
0.2227=0.0237
0.6588+0.0163
BIOT
0.5485 03325
0.0968+0.064
0.5308#0.005
PhysioP300
LaBraM
0.6477+ 10
0.2935+000227
0.7068+0.011
Ours
0.650240.0063
0.2999+0.0139
0.7168+0.0051
model exhibited accuracy improvements of 9.4%, 1.5%, 2.6%, respectively, compared to BENDR. Considering that BENDR used full model fine-tuming while model only fine-tuned additional linear layer, this suggests that our model extracts richer and more universal features. We used the linear-probing method for BIOT and LaBraM. EEGPT improved by 2.3%, 3.6%, and 1.4%, respectively, compared to LaBraM, and by 12.5%, and 8.1%, %, and 2.9%, respectively, compared to BIOT. For the ERP-type task datasets, model outperforms BENDR 2.6% and 3.9% the KaggleERN and PhysioP300 datasets, respectively. The performance our model on the PhysioP300 dataset is comparable to that of LaBraM, but higher by 3.0% KaggleERN. Our model also outperforms BIOT on the KaggleERN and PhysioP300 by 7.2% and 10.2%, respectively. On all tasks, our model EEGPT achieves competitive results compared BENDR, BIOT, and LaBraM. This demonstrates that EEGPT learns consistent representational features over the temporal-spiie -spotialspatill dimensions, enabling the model to be more widely applied to multiple paradigm tasks and to achieve better classification performance.
Combining the above experimental results, demonstrate that the method proposed in this paper addresses the issues of poor EEG channel adaptability, poor quality of EEG representations extracted by existing self-supervised learning methods, and the lack of universality of representations across multiple paradigms. Our method effectively extracts high-quality universal EEG representations.
3.4 Ablation Experiment Results
We conducted ablation experiments using the large model with four different configurations, and the results presented in Table 5 In the absence of alignment loss (CA). the reconstruction loss

---

Table 5: The results of the ablation study.
Variants
LA
LR
BCIC-2A-BAC
BCIC-2B-AUROC
KaggleERN--AURO
w/o La
37.13
0.57
0.528740.0088
0.726440.0381
0.575240.0164
B: w/o LN
0.15
0.002
0.5567+0.0088
0.7920#0.0012
0,5891#0.0227
w/o skip
0.12
0.56
0.579640.0011
0.7702+0.0122
0.635640.02999
D: with all
0.24
0.56
0.5846+0.0070
0.805940.0033
0.6621#0.00%
Table The results of pretrained models.
variants
de
layers
S
params
C
C
BCIC-2A-BAC (%)
tiny1
64
2/2/4
0.4M
0.32
0.60
49.19
tiny2
64
2/2/4
0.5M
0.36
0.60
50.03
tiny3
64
8/8/8
1.6M
0.17
0.59
51.58
little
128
8/8/8
4
6,4M
0.18
0.57
54.18
basel
256
6/6/6
19M
0.24
0.56
54.53
base2
256
8/8/8
25M
0.33
0.56
56.48
base3
512
6/6/6
76M
0.14
0.58
54.47
large
512
8/8/8
4
101M
0.24
0.56
58.46
comparable to that of the version model, but there a significant performance degradation of 6%%~9% in the downstream task. Without layer normalization on the targets of reconstruction loss, the version model demonstrated lower pretrain loss L, but was affected by extreme values and covariate shift (35) resulting in 3%,1 1% and 7% reduction in downstream tasks performance. The version C model, which removes the skip connection and uses all {prede ) from the predictor as inputs to the reconstructor, exhibited lower alignment loss and comparable reconstruction loss compared to the version D model, but showed 1%~~3% lower performance in the downstream task. These results suggest that the dual self-supervised method proposed in this paper is effective, the spatio-temporal alignment improves the quality of the EEG representations extracted by the model.
3.5 Pretrain Experiment Results
-052
-055
30000
2022
047
4018
Figure Scaling laws with EEGPT parameter size N. Axes are all on logarithmic scale.
We designed 8 variants to investigate the effects of model size (embedding size d, / layers) and summary token (S) on pretraining loss and downstream task accuracy. To evaluate the performance of the pretrained models, we used the anneargprobing method to test cross-subject classification tasks on the BCIC-2A dataset. The experimental results are presented in Table As the model size (embedding size de llyers)) and summary tokens (S) increased, the reconstruction loss (CR) gradually decreased, and the performance on the downstream task improved. For models with the same S, larger models exhibited lower alignment loss LA and higher performance. We also investigated the trends accuracy and reconstruction loss the model size increased, as shown in Figure 5 These trends can be summarized by the scaling laws: ACC = (33.6 0.00. and L (0.72 N) -0.014 where N the parameter count of the model. The results the downstream task experiments indicate that larger models generally achieve higher accuracy, with the large model, featuring an 8-layer, 512-embedding dimension, and 4 summary tokens, exhibiting the highest accuracy. For simplicity, used the optimal large model for testing on all downstream tasks.

---

4 Conclusion
In this paper, propose self-supervised EEG Pretrained Transformer (EEGPT) model with over 10 million parameters for universal EEG representation learning. We employ a dual self- supervised approach for pretraining, involving spatio-temporal representation alignment and mask- based reconstruction. The spatio-temporal representation alignment aligns masked patches' features with full patches' features, enhancing the quality of EEG representations and concentrating key information in the encoder output. The mask-based reconstruction leverages the spatial and temporal consistency exhibited by EEG signals extract complementary features in both dimensions. We design hierarchical structure for EEGPT, which first extracts stable spatial representations from short-term EEG signals, then captures the temporal correlations among long-term EEG signals. This structure not only reduces computational complexity but also enhances the flexibility and adaptability of EEGPT in BCI applications. Experiments demonstrate that our dual self-supervised pretrain model significantly outperforms the popular universal feature extraction models BENDR, BIOT, and LaBraM on tasks such as motor imagery, sleep stage detection and ERP-type classification. Compared with BIOT on the TUEV dataset, we achieved 9.5% performance improvement. These tasks, which have different channel configurations and sampling rates, suggest that EEGPT is scalable. In the future, we plan further enrich the pretraining EEG dataset and expand the model size and applicability.
References
[1] Gernot R Müller-Putz. Electroenneepaalggraphy Handbook of clinical neurology, 168:249-266 2020.
[2] Sharlene N Flesher, John Downey, Jeffrey M Weiss, Christopher I. Hughes, Angelica J Herrera, Elizabeth C TTyler-Kabara Michael L Boninger, Jennifer L. Collinger, and Robert A Gaunt. brain-computer interface that evokes tactile sensations improves robotic arm control. Science, 372(6544):831-82)) 2021.
[3] Eunjin Jeon, Wonjun Ko, Jee Seok Yoon, and Heung-ll Suk. Mutual information-driven subject- invariant and class-relevant deep representation learning in bei. IEEE Transactions on Neural Networks and Learning Systems, 34(2):739-749, 2023. doi: S..1009/NNNSS.0011 25 000583..
[4] Christian O'Reilly, Nadia Gosselin, Julie Carrier, and Tore Nielsen. Montreal archive of sleep studies: an open-access resource for instrument benchmarking and exploratory research. Journal of Sleep Research, 23(6)x628-633 635, 2014. doi: https://doi.org/// https://onniieelbrrrrr wiley wwwm//oo/// abs/10 1111/jsr. 12169
12169. URL
[5] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5149-511 2022. doi: 10. 10.1199//AMM..22113377022
[6] Jacob Devlin, Ming-Wor Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
[7] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervise vision transformers, 2021.
[8] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners, December 2021. htt::/ttt//arrxvvoor /bs/
2111.06377 arXiv:2111 11.6677 [cs].
[9] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations, 2020.
[10] Fabian Falck, Suproteem K. Sarkar, Subhrajit Roy, and Stephanie L. Hyland. Contrastive representation learning for electroencephallgramm classification. In MLAH@NeurIPS, 2020.
semaamiccccaattccclcc.........te hrrlltttppe/pppppeeeaaattcccttor wwtpg//orpssI:::000 :29778191
[11] B. Kemp, A.H. Zwinderman, B. Tuk, H.A.C. Kamphuisen, and J.J.L. Oberye. Analysis of a sleep-dependent neuronal feedback loop: the slow-wave microcontinuity of the eeg. IEEE Transactions Biomedical Engineering, 1759):11851119, 2000. doi: 10.1199/10.869928.
10

---

[12] Demetres Kostas, Stéphane Aroca- Ouellette, and Frank Rudzicz. BENDR: Using Transformers and a Contrastive Self Superviupei Learningg Task Learn From Massive Amounts of EEG Data. Frontiers in Human Neumoscience, 15, 2021. ISSN 1662-5161. doi: 10.3389/fnhum. 2021.653633 URL irrnoiiiei. htt.:aaawi...rrrrrrrreeeeeeeeeee 3389 (a009/ffhum... 2021
653659
[13] Qiushi Zhu, Xiaoying Zhao, Jie Zhang. Yu Gu, Chao Weng, and Yuchen Hu. Eeg2vec: Self-Supervised Electrreenceehaalggaphhe Representation Learning, May 2023. URL http:
//arxiv. org.lv..org/ao www.:/ggg//bb/b/eeeooooo awwg//bb/2305..... 13957 (XXi:23305.1357 [eess].
[14] Simon Geimaerr, Tom Francart, and Alexander Bertrand. Time-adaptive unsupervised auditory attention decoding using eeg-based stimulus reconstruction. IEEE Journal of Biomedical and Health Informatics, 26(8):3767-3778, 2022.
[15] Chaoqi Yang. M. Brandon Westover, and Jimeng Sun. BIOT: Cross-data Biosignal Learning the Wild, May 2023. URL Uttp://::/arxxi wwtp:/wwww.rrrra///////// 10351 arXiv:2305. 10351 [cs, eess].
[16] Ali Hossam Shoeb. Application of machine learning to epileptic seizure onset detection and Ireatment. PhD thesis, Massachusetts Institute of Technology, 2009.
[17] Wei-Bang Jiang, Li-Ming Zhao, and Bao-Liang Lu12. Large brain model for learning generic rep-resentations with tremendous eeg data in bei.
(18) Iyad Obeid and Joseph Picone. The temple university hospital eeg data corpus. Frontiers in Neuroscience, 10, 2016. ISSN 1662-453X. doi: 2..338//iins.22116001122 URL BtRhhtts::/ :/opp:://wwrr fnins: 2016.00199
frontiersin.orr
[19] Wei Liu, Jin-Lin Qiu, Wei-LLong Zheng, and Bao-Liang Lu. Comparing recognition performance and robustness of multimodal deep learning models for multimodal emotion recognition. IEEE Transactions on Cognitive and Developmental Systems, 14(2):715-729, 2021.
[20] Yongtian He, Trieu Phat Luu, Kevin Nathan, Sho Nakagome, and Jose Contreras- Vidal. mobile brain-body imaging dataset recorded during treadmill walking with a brain-computer interface. Scientific data, 5(1):1 2018.
[21] Alexander A Fingelkurts, Andrew A Fingelkurts, Victor A Ermolaev, and Alexander Ya Kaplan. Stability, reliability and consistency of the compositions of brain oscillations. International Journal of Psychophysiology, 59(2):116-126, 2006.
[22] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North, Jan 2019, doi: 10.18653/v 11t////////1911///// 1/11//199911...... URL http: //dx: doi. org/ 10/g/00.11. 10. 18653/ v1/n19- 1423
[23] Ary I. Goldberger, Luis A. N. Amaral, Leon Glass, Jeffrey M. Hausdorff, Plamen Ch. Ivanov, Roger G. Mark, Joseph Mietus, George Moody, Chung Peng, and H. Eugene Stanley. Physiobank, physiotoolkit, and physionet. Circulation, 101(23) 2000. doi: 10.1161/01 CIR, I.I............ 101.23..11.. URL BtLhhtts::/ http:///wwwwaaaorraasl..ooo org httpg//do/aaba//// 1161/
01. 10w.111....00 e215
[24] Robin Tibor Schirrmeister, Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, and Tonio Ball. Deep learning with convolutional neural networks for eeg decoding and visualization. Human Brain Mapping, aug 2017. ISSN 1097-0193. doi: 10. 10.1002/hbm.233300 http://: /ww. org 10. htm2/hbm........ 23730
[25] Gan Huang, Zhenxing Hu, Weize Chen, Zhen Liang, Linling Li, Li Zhang, and Zhiguo Zhang. M3CV:A Multi-subject, Multi-session, and Multi-task database for EEG-based Biometrics Challenge. bioRxiv, 2022. doi: 10. 10.1100/2022.6..66 28.497624. URI w/www biorxiv prg/connent/eaan /002 /22.0//22//////2222222222222 06 2976722

---

[26] Ali Al-Saegh, Shefa A Dawwd, and Jassim M Abdul-Jabbar. Deep learning for motor imagery ceg-based classification: A review. Biomedical Signal Processing and Control, 63:102172. 2021.
[27] Fotis Kalaganis, Elisavet Chatzilari, Spiros Nikolopoulos, loannis Kompatsiaris, and Nikos A. Laskaris. An error-aware gaze-based keyboard by means of hybrid bei system. Scientific Reports, Aug 2018. doi: 10. 10.1088/44159 011-5988--88-------1 URL. http: doi. org/10 10
1038/ s41598-018 018-31425-2
[28) Khald Ali Aboalayon, Miad Faezipour, Wafaa S Almuhammadi, and Saeid Moslehpour. Sleep stage classification using eeg signal analysis: a comprehensive survey and new investigation. Entropy, 18(9):272, 2016.
[29] Xiangwen Kong and Xiangyu Zhang. Understanding masked image modeling via learning occlusion invariant feature, 2022.
[30] Liangjian Wen, Xiasi Wang, Jianzhuang Liu, and Zenglin Xu. Mveb: Self-supervised learning with muewi-view entropy bottleneck. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[31] Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autocncoder for self-supervised representation learning. International Journal of Computer Vision, 132(1))208-2235 2024.
[32] Mary Phuong and Mareus Hutter. Formal Algorithms for Transformers, July 2022. URL http://arxiv.o orge/iiiiorraeeee htrggabb/20/0/// 09238 Number: arXiv:2207.09221 urXiv:2207.0922 [cs].
[33] Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021.
[34] Jiawei Ren, Mingyuan Zhang, Cunjun Yu, and Ziwei Liu. Balanced mse for imbalanced visual regression, 2022.
[35] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. 2016.
[36] Yiqun Duan, Jinzhao Zhou, Zhen Wang, Yu-Kai Wang, and Chin-Teng Lin. We Discrete EEG Waves Encoding for Brain Dynamics Text Translation, October 2023. URL http
//arxiv. wawlvv.org/bbba 2309. 14030 arXiv:2309.1400 [cs].
[37] Yijun Wang. Xiaogang Chen, Xiaorong Gao, and Shangkai Gao. benchmark dataset for ssvep- based brain-computer interfaces. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 25(10))1746- 1752, 2017. doi: 10. N0.1109//SSR..206..666
[38] Wei-Long Zheng and Bao-Liang Lu. Investigating critical frequency bands and channels for eeg-based emotion recognition with deep neural networks. IEEE Transactions on Autonomous Mental Development, 763):162-175, 2015. doi: 10.1 11199/AM..201..44331111
[39] Michael Tangermann, Klaus-Robers Müller, Ad Aertsen, Niels Birbaumer, Christoph Braun, Clemens Brunner, Robert Leeb, Carsten Mehring, Kai Miller, Gernot R. Müller-Putz, Guido Nolte, Gert Pfurtscheller, Hubert Preissl, Gerwin Schalk, Alois Schlögl, Carmen Vidaurre, Stephan Waldert, and Benjamin Blankertz. Review of the bei competition iv. Frontiers in Neuroscience, 6, 2012. URL https::/app: http:://app.semannicccholo.e.. wwwpcchhoaarro//ooorooeeeo atg/CorrpssD::70255.
[40] David Steyrl, Reinhold Scherer, Josef Faller, and Gernot R. Müller-Putz. Random forests in non-invasive sensorimotor rhythm brain-computer interfaces: practical and convenient non-linear classifier. Biomedical Engineering Biomedizinische Technik, 61:77 - 86, 2016.
URL httts:://ppppi ddemanticacholess org/ htt.p//.......rooooorrs.D//////. :
[41] Perrin Margaux, Emmanuel Maby, Sébastien Daligault, Olivier Bertrand, and Jérémie Mattout. Objective and subjective evaluation of online error correction during p300-based spelling. Advances in Human- Computer Interaction, 2012, 01 2012. doi: 10.1155/2012/78922
12

---

[42] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, June 2021. URL http:://axiii org/ httg/abb/201..11922 arXiv:2010.1 1999 [es].
[43] Leslie N. Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates, 2018. URL https://arxiv, http://axxxvv...aaai/a barg/abs/1708 07120
[44] Jin Jing, Wendong Ge. Shenda Hong, Marta Bento Fernandes, Zhen Lin, Chaoqi Yang, Sungtae An, Aaron F Struck, Aline Herlopian, loannis Karakis, al. Development of expert-level classification of seizures and rhythmic and periodic patterns during eeg interpretation. Neurology, 100(17)=e1500=e1100) 2023.
[45] Chaoqi Yang, Cao Xiao, M Brandon Westover, Jimeng Sun, et al. Self-supervised electroen- cephalogram representation learning for automatic sleep staging: model development and evaluation study. JMIR AI, 2(16694 2023.
[46] Wei Yan Peh, Yuanyuan Yao, and Justin Dauwels. Transformer convolutional neural networks for automated artifact detection in scalp eeg. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 3599-3602. IEEE, 2022.
[47] Hongli Li, Man Ding, Ronghua Zhang, and Chunbo Xiu. Motor imagery eeg classification algorithm based enn-lstm feature fusion network. Biomedical signal processing and control, 72: 103342, 2022.
[48] Yonghao Song, Xueyu Jia, Lie Yang, and Longhan Xie. Transformer- based spatial-temporal feature learning for eeg decoding. arXiv preprint arXiv:2106.11170, 2021.
[49] He He and Dongrui Wu, Transfer learning for brain-computer interfaces: euclidean space data alignment approach. IEEE Transactions on Biomedical Engineering, 67(2):399 2019.
[50] Jianlin Su. Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced Transformer with Rotary Position Embedding, August 2022. URL http:// //arxivaxvvvv org abs/ babs/2104 09864 arXiv:2104.0988 [cs].
13

---

A
Additional results
This section provides additional experimental results to support the claims in the main paper.
A.1
Ablation study for the predictor
0.68
0.66
0.64
0.62
0.60
0.58
0.56
baseline
prpddcctoorr
5000
10000
15000
20000
25000
35000
40000
step
Figure 6: Validation loss curves during pretraining, where 'baseline" is the standard EEGPT model and 'no_ no_prrddicoo' is the model without the predictor.
We conducted pretraining experiment after removing the predictor, the variation of Ln loss with the number of iteration steps during training is shown Figure 6 In this figure, 'baseline' is the standard EEGPT model and 'no po__rrdictorr is the model without the predictor. We can see that the reconstruction loss LR of the 'no cno_preddicor model after removing the predictor does not decrease, which indicates that directly aligning the outputs of the encoder and the momentum encoder does lead to the problem of representation collapse, resulting in the reconstruction task not being able to learn meaningful representation.
A.2 Ablation study for pretraining methods
Table 7: The results of the ablation study for pretraining methods.
Variants
LA
LR
BCIC-2A-BAC
BCIC-2B-AUROC
KaggleERN-AAUROC
A: w/o CA
37.13
0.57
0.5287#0.0088
0.726420.0381
0.575220.0164
w/o LNN
0.15
0.002
0.556740.000 00088
0.7920+0.000
0.589140.0227
C: w/o skip
0.12
0.56
0.5796+0.000 10111
0.7702+0.0122
0.6356+0.0296
D: with all
0.24
0.56
0.5846+0.0070
0.8059+0.0032
0.6621+0.0096
In the ablation experiments, we used the BCIC-2A, BCIC-2B and KaggleERN datasets to test the model with the linear probing method, the results are shown in Table 7 We can conclude that (1) the model performance degradation on all datasets without LA loss is significant (6% to 9%); (2) without layer normalisation the reconstruction target, the performance of the version B model degraded by 3%, 1%, and 7% on BCIC-2A, BCIC-2B, and KaggleERN, respectively; and (3) the performance of version C model (removed the skip connection) reduced by 1%, 3% and 3% on these datasets, respectively.
A.3 Ablation study for fine-tuning methods
We have added experiments comparing linear probing method and full fine-tuning method, as well as experiments comparing with and without adaptive spatial filter. The experimental results are displayed Table In T "ASF" stands for with an adaptive spatial filter, in contrast to feeding the signal directly into the model; 'L-P" stands for using linear probing, in contrast to using full fine-tuning of the model. Model variants and are the models with full fine-tuning and linear probing after excluding the adaptive spatial filter, respectively. Model variants B and D are models
14

---

Table 8: The results of the ablation study for fine-tuning methods.
Variants
ASF
L-P
BCIC-2A-BAC
BCIC-2B-AUROC
KagglcERN-AUROC
A
0.5774+0.0072
0787110000554
0,6078+0.0101
BC
0.5183+0.0155
0.7541
0.61 10=0.0019
0458660000088
0.797440.0030
0.6463#0.000
0.5846+0.0070
0.8059#0.0032
0.6621#0.0096
with full fine-tuning and linear probing after using adaptive spatial filter, respectively. The results show that on the BCIC-2B and KaggleERN datasets, variants C and D tested with linear probing achieved better results than variants A and using full fine-tuning: on the BCIC-2A dataset, variant A with full fine-tuning and no adaptive spatial filter is close to variant D with linear probing and an adaptive spatial filter, but overall the linear probing used by models C and outperform and The results show that variants B and D using adaptive spatial filter achieve better results compared to variants and C without adaptive spatial filter.
A.4 Scaling laws experiments
-0.235
Test Accuray
0245
-0.255
165
10f
107
Figure 7: Results on TUAB dataset. Scaling laws with EEGPT parameter size N. Axes are all on logarithmic scale.
-0.2
-324
Test AUROC
-0.30
-032
165
10c
107
Figure 8: Results on BCIC-2B dataset. Scaling laws with EEGPT parameter size N. Axes are all on logarithmic scale.
We added the results of the scale law experiments on the TUAB dataset and the BCIC-2B dataset, as shown Figure andd The results on the TUAB dataset show that the scaling law of the test balanced accuracy metric with model size (N) for TUAB is: BAC (0.74 0.0001, the results
15

---

on the BCIC-2B dataset show that the scaling law of the AUROC metric with model size (N) for BCIC-2B is: AUROC (0.61 NN0.0167.
A.5
Effect of pretrain data size
Text Actorccrrryy
005
-060
064
10
Size
Figure 9: Results on BCIC-2A dataset. Scaling laws with pretrain data size D. Axes are all on logarithmic scale.
-022
-024
Tels AUROC
-026
028
-030
2x
3xx00
14x10
6x110
10'
Size
Figure 10: Results on BCIC-2B dataset. Scaling laws with pretrain data size D. Axes are all on logarithmic scale.
We added pretraining experiments using 100%, 50%, 25%, and 12.5% of the training data and tested them on the downstream tasks of BCIC-2A and BCIC-2B. The results are presented in Figure 9 and
10 The results show that the scaling law of the balanced accuracy metric with the total amount of data D (the percentage of the training data used) for BCIC-2A is: ACC = (0.58 * 0.0461, and the scaling law of the AUROC metric with the total amount of data D for BCIC-2B is:AUROC
(0.79 D 0.0325
A.6 Results of LaBraM on TUAB and TUEV
Table 9: The results of different methods on TUAB.
Methods
Model Size
Balanced Accuracy
AUROC
BIOT [5]
3.2M
0.7959=0.0057 0.8140+0.0019
0.881540.0043
LaBraM [17]
5.8M
0.9022+0.000
Ours-Tine Tiny
4.7M
0.7959+0.000
0.8716+0.004
Ours
25M
0.7983+0.0030
0.8718+0.0050
16

---

Table 10: The results of different methods on TUEV.
Methods
Model Size
Balanced Accuracy
Weighted F1
Cohen's Kappa
BIOT [[5]
3.2M
#0.0225
0,749210.0082
0.5273=0.0249
LaBraM 17
5.8M
0.6409=0.0065
0,8312+0.0053
0.6637#0.0093
Ours-Tiny
4.7M
0.5670=0,0066
0.7535=0.0097
0.5085=0.0173
Ours
25M
0.6232+0.0114
0.8187#0.0063
0.6351 +.6311+0.0134
LaBraM employs larger pretraining dataset than ours, which also contains the TUEG dataset that is similar to the TUAB and TUEV distributions, and whose paper illustrates the scale effect of the amount of pretraining data, which may have facilitated the learning of richer and more downstream lask-adaptable representation for LaBraM.
A.7 Ablation study of self-supervised learning on TUAB
Table 11: The results of different methods on TUAB.
Methods
Model Size
Balanced Accuracy
AUROC
BIOT [1S]
3.2M
0.7959=0.0057
1.8815=0.00 0.043
Ours (no pretrained)
25M
0.7553#0.001
0.8260#0.M 0018
Ours
25M
0.7983#0000030
0.871 0.8718+0.0050
We tested the model with randomly initialized parameters (no pretrained model) on the TUAB dataset and the test results are shown Table II In Table 'Ours (no pretrained)' represents the model that is not loaded with pretrained parameters. By comparison, we see that the model without using self-supervvsed pretraining has the worst performance on TUAB, with 4% reduction in balance accuracy and 5% reduction in AUROC compared to the model loaded with pretraining parameters.
B CODE ACCESS
The code for this paper is available at:
C DATASET DESCRIPTION
C.1 PRETRAINING DATASET DESCRIPTION
C.1.1 Motor Imagery and Motor Execution tasks: PhysioNetMM 23
The PhysioNetMI datase P consists of over 1500 one-minute and two-minute EEG recordings obtained from 109 volunteers. Subjects performed different motor execution/imagery tasks while recording 64-channel EEG using the BC12000 system.
Preprocessing: All eight tasks were used during the pretraining of this paper, using global average reference, intercepting data from to 6s after the start of each trial, and randomly intercepting data from a 4s time window these during pretraining.
C.1.2 Motor Imagery task: HGD 24
The HGD dataset, 128-electrode dataset, was taken from 14 healthy subjects with approximately 1,000 trials each. The four-second executive movement trials were divided into 13 runs. The four types of movements were left-handed movements, right-handed movements, bipedal movements, and rest, The training set consisted of approximately 880 trials for all but the last two runs, and the test set consisted of approximately 160 trials for the last two runs. We access this dataset by MOABB
Preprocessing: All four tasks were used in the pretraining process of this paper, first downsampled 256 Hz. and then standardized to mV, using global average reference to intercept the data from
17

---

to 10s after the start of each trial, and randomly intercepting the data from a 4-s time window of these during pretraining.
C.1.3 SSVEP task: TSU 37
The TSU datase is a benchmark dataset for brain-computer interfaces based on steady-state visual evoked potentials (TSUBenckmark), This dataset collects SSVEP-BCI recordings from 35 healthy subjects using a brain-computer interface (BCI) speller with 40 characters for experimental data acquisition. SSVEP-BE ESVEP-BCIII recordings were made for 40 characters that flashed different frequencies (8-15.8 Hz with 0.2 Hz intervals).
Stimuli: Each trial began with visual cue (red square) indicating the target stimulus. The cue appeared on the screen for 0.5 seconds. Subjects were asked to shift their eyes to the target as soon as possible within the cue duration. After the cue was offset, all stimuli began flashing on the screen simultancously for 5 seconds. After the stimulus offset, the screen was blanked for 0.5 seconds, and then the next trial began, which allowed subjects to have short rest period between successive trials.
Signal: EEG 35 subjects (64 channels, 250 Hz). Each subject's experiment consisted of 6 blocks. Each block contained 40 trials corresponding to all 40 characters displayed in randomized order. Total: 35 persons x blocks x 40 trials.
Preprocessing: All 40 tasks were used in the pretraining process of this paper, first downsampled to 256 Hz, and then standardized to mV, using a globally averaged reference intercept the data from to 4s after the start of each trial for pretraining.
C.1.4 Emotion Recognition task: SEED
38
The SEED dataset, Shanghai Jíao Tong University Emotion EEG dataset (SEED is an EEG dataset provided by the BCMI lab led by Prof. Baoliang Lu.
Stimuli: 15 four-minute long movie clips from six Chinese movies.
Signal: EEG (62 channels, 200 Hz) from 15 subjects and eye movement data from 12 subjects. Three experiments were conducted per subject, each approximately one week apart, for total of 15 subjects 3 sessions 45 subjects.
Scores: Positive (1), negative (-1), and neutral (0).
Preprocessing: All three tasks used in the pretraining process of this paper, first downsampled to 256 Hz and then standardized to mV, using a global average reference to intercept the data from 10s after the start of each trial for pretraining.
C.1.5 Identification task: M3CV 25
6 is a reliable brainprint identification system designed to withstand changes in the mental state of the subjects pcross-paradigg test) and successfully identify individuals even after several days (cross-session test). The Multi-Session Multi-Paradigm EEG database (Multi-Subject Multi-Session Multi-Paradigm Commonality and Variability (M3CV)) contains 106 healthy subjects, two sessions, and six types of EEG paradigms.
The M3CV datase
Experimental paradigms: Resting state, event-related potentials, evoked stimuli, P300, motor execution, and SSVEP.
Preprocessing: The full task-state data were used in the pretraining process of this paper, first upsampled to 256 Hz, and then standardized to mV, using a globally averaged reference, intercepting data from to 4s after the start of each trial for pretraining.
"hut:://hi.mede http::/hh.i.mm.tsss.....o www.geww...n..dd....eemmmm "htts://hemi.s///// wwt..scu.uuu.u...ooo.....e edu. cwthome/scee//
"htpss//aistudioo baidu, wwwp://wwww......ooooooooooo htt.://///ooommiiooooo/////////a material
18

---

C.2 DOWNSTREAM DATASET DESCRIPTION
C.2.1 Motor Imagery Task: BCIC-2A
39
This Eraasiscoonsistt of EEG data from 10 subjects and includes four different motor imagery tasks: motor imagery of the left hand (Class 1), right hand (Class 2), feet (Class 3), and tongue (Class 4). Each subject performed two sessions on different days, with total of 288 trials per session.
Preprocessing: Uniform units; O to 38 Hz filtering was used; resampling to 256 Hz; EA normalization
49] was used for each session. EA normalization is a common normalization method applied to data from motion imagery tasks such as BCIC2A.
Experiments using the LOSO method. In the BCIC2A dataset 4- category cross-subject experiment, the accuracy of the model in this paper is lower than that of the task-specific SOTA model. This may be related the high individual variability of the BCIC2A data. The fine-tuning method of combining the aask-specific model with the model in this paper was used for testing. The MSConnNNt model, which has multi-scale convolutional EEGNet-like structure, was used as the motor imagery task-specific model. The model branch first maps the input 22-channel EEG signal into 10-20 system standard channel data using an adaptive spatial filter (convolution); then a pretrained encoder is used to extract the generic representations; and after collocating the generic representations and the in-domain representations of Wenchao's 's model branch, they mapped into category logits using linear layer. Training is performed using the Adam optimizer, OneCycle learning rate strategy [43] (starting learning rate 5c-4, maximum 1e-3, minimum 4.22c-7), 100 rounds of training, and batch size of 72.
Experimental Configuration:
C.2.2 Motor Imagery Task: BCIC-2B 40
This d of EEG data from 10 subjects. Each subject was given five training opportunities, the first two of which were training data without feedback (screening) and the last three with feedback. The three bipolar recordings (C3, Cz, and C4) were sampled at frequency of 250 Hz. The dataset consisted of motor imagery (MI) for two categories: left-handed (category 1) and right-handed (category 2). Each subject participated in two sessions without feedback recordings on two different days within a two-week period. Each session consisted of 120 category-balance trials.
Preprocessing: Uniform units: 0 to 38 Hz filtering was used: resampling 256 Hz: EA normalization [4 was used for each session. EA normalization is a common normalization method applied to data from motion imagery tasks such as BCIC2A.
Experimental Configuration: Experiment using LOSO validation method. The fine-tuned modeling approach used to test cross- 2-categorization task the BCIC2B dataset for the motor imagery task. The input data are 3-channel (C3, Cz, and C4 channels) EEG data with 4s, 256Hz sampling rate; to fully utilize the feature extraction capability of the model, spatial filters (convolution) were used to map the 3-channels of the input data into 7-channels, which corresponds to the C5, C3, C1, CZ. C2, C4, and C6 channels of the pretrained encoder; the features extracted using the pretrained encoder and then mapped into 7 channels using the linear layer mapped to category logits. The training process is performed using AdamW optimizer, OneCycle learning rate strategy [43] (starting learning rate 1.6e-5, maximum 4e-4, minimum 1.51e-7), 100 rounds of training, and batch size of 64.
C.2.3 Sleep Stage Detection Task: Sleep-EDFx
This d gives further insight into the generalizability of the model, as BCI data are typically categorized in the context of a specific trial or event, whereas SSC is a much more continuous problem that requires labeling the specific sleep stage that subject is in over long period of time. The Sleep-EDFx dataset contains 197 (78 healthy subjects) all-night sleep recordings, which include EEG, electrooculogram, chin EMG, and event markers.
Preprocessing: The preprocessing method of Banville et al. (2020) was referred to, which first converts the data unit to mV, and then uses 30Hz low-pass filter, intercepts the samples in 30s
hhtps://ww.bhcii wttp:h/h...cccommpiiooo/i//aaaaa http:://hhhiinnttooooooooeeeeeep
19

---

non-overlapping windows, and uses channel-wise (channel (chamnel-independl) indepeedenn)) z-score normalization for each sample.
Experimental Configuration: Subjects were randomly divided according to the ratio of 60
C.2.4 ERN task: KaggleERN
In this datase 10 each subject is presented with letters and numbers (showing 36 possible items on matrix) to spell words. Each item of the word is flashed in random order grouped together and selected one at time. The selected item the one that the online algorithm most likely to recognize as typical target response. The goal of this challenge was to determine whether the selected items were correct by analyzing the brain signals of subjects after receiving feedback. Two transcription-spellinn conditions were used: fast mode with more errors (4 blinks per item); and slower, less error-prowd condition (8 blinks per item).
Preprocessing: Uniform units; downsampling to 256 Hz; global averaging reference; intercepting 2 8 of data starting at -0.7 s before the onset of flicker.
Experimental Configuration: For comparison with the BENDR paper, 10 of the subject data were used as the test dataset, and the remaining subject data used for 4-fold cross-validation method training. KaggleERN uses fine-tuning method 1, with sliding window of 125ms in steps, to firstly map the 56 channels the 19-channel EEG data of the 10-20 system by adaptive spatial filters (convolution). and then features are extracted using pretrained encoder, and finally mapped into classification logits using a linear layer. The training process is performed using the Adam optimizer, the OneCycle learning rate strategy (43] (starting learning rate of 1.6e-5, maximum of 4e-4, and minimum of 1.5le-7), and 100 rounds of training with batch size of 64.
C.2.5 P300 task: PhysioNetP300 23
In this datase each participant asked to spell total of 20 characters using traditional matrix speller (Donchin speller). The target characters were randomly selected before the start of the run. Each row and column of standard 6x6 character matrix 50 ms intervals with approximately 20 flashes. During this time, subjects were asked to focus their attention the target character and mentally count the number of times the target character was highlighted.
randomly augmented for 100 ms at
Uniform units; 120Hz low-pass filtering was used; downsampling to 256Hz; and 28 of data starting at -0.7s before the onset of the flicker was intercepted.
Preprocessing:
Experimental Configuration: For comparison with the BENDR paper. subjects 8, 10, and 12 were removed and the data from the remaining 9 subjects were retained. Experiments were performed using the LOSO cross- validation method. The PhysioNetP300 uses a fine-tuning method with a sliding window of 125ms step size, using all channels used in the pretraining phase; the data from each EEG channel is first udaptively scaled by an adaptive spatial filter (learnable scaling factor), then features are extracted using the pretraining encoder, and finally mapped using the linear layer mapping for classification logits. The training process uses AdamW optimizer, OneCycle learning rate strategy [K] (starting learning rate 3.2e-5, maximum 8e-4, minimum 3.02e-7), 100 rounds of training, and batch size of 64.
C.2.6 TUAB (abnormal detection) and TUEV (event type classification)
18
TUAE is a corpus of EEGs which are 23-channel and sampled at 256 Hz. All data have been annotated as normal abnormal. There are total 409,455 10-second samples that we use for binary classification to predict normal/abnormal.
TUEV 13
is a subset of TUEG that contains annotations of EEG segments as one of six classes: (1) spike and sharp wave (SPSW), (2) generalized periodic epileptiform discharges (GPED), (3) periodic lateralized epileptiform discharges (PLED), (4) eye movement (EYEM), (5) artifact (ARTF) and
11 "htts://pyyiinnttooooootteeeenee httpp//wwww../ppp///////c/////// shtml
12 1 htttss//sspppiconeprssssss
20

---

(6) background (BCKG). The EEG signals contain 23 channels at 256 Hz and are segmented into 112,491 5-second samples.
We conducted experiments using linear-probing on these two datasets. Since these datasets use different electrodes and sampling rates compared the model pretraining. used two layers of convolution (spatial convolution and temporal convolution) to adaptively fit the data. The spatial convolution used 1xl convolution, and the temporal convolution used depthwise convolution. The convolution kernel size for TUAB was (I, 15), and for TUEV, it was (1, 55). Both experiments used the same optimizer and learning rate of 5e-4. Due GPU memory limitations, the batch size for TUAB was 100, and for TUEV, it was 500.
Table 12: Model design for TUAB dataset.
Input Size
Operator kernel
stride
groups
padding
23 x 2000
convid
1
0
20 x 2000
batchnorm.gehe
20 x 2000
convid
15
20
7
20 x 2000
batchnorm. geeln
20 x 2000
dropout(O.25)
20 x 2000
eegpt-encoder
64
64
31 x 4 512
flatten, linear
output
Table 13: Model design for TUEV dataset.
Input Size
Operator kernel
stride
groups
padding
23 x 1000
convid
1
0
x 1000
batchnorm.gehl
20 x 1000
convld
55
20
27
x 1000
batchnorm.get
20 x 1000
dropout(0.5)
20 x 1000
cegpt-encoder
64
64
15 x 4 512
lanten,linear
output
The detailed modelling structure used in the downstream tasks is presented in Table T2 The 23-channel input is first to reduce the number of channels to 20 by the convolution. Then, the cegpt-encoder uses the following 20 channel embeddings the inputs' channel embeddings: [FP1, FPZ, FP2, F7, F3, FZ, F4, F8, T7, C3, CZ, C4, T8, P7, P3, PZ, P4, P8, O1, O2]. The eegpt-encoder maps 64-length window segments of the input signals to 4 (number of summary tokens) x 512- dimensional features. Finally, the flatten and linear layers are used to output the final classification score.
T2 and Table 13
D METRICS DESCRIPTION
The following metrics are used for comparison: Balanced Accuracy (BAC): The mean recall across all classes, applicable to both binary and multi-class classification. 2) AUROC: The area under the receiver operating characteristic curve, primarily used for binary classification. 3) Weighted F1: The harmonic mean of precision and recall, with equal emphasis on both metrics, employed for evaluating multi-class classification. 4) Cohen's Kappa: A statistic measuring the agreement between categorical variables X and Y, derived from the observed and expected frequencies in the diagonal of square contingency table. We set AUROC as the monitor score for binary classification and Cohen's Kappa the monitor score for multi-class classification.
21

---

E MORE IMPLEMENTATION DESCRIPTION
For model implementation, the BENDR code is downloaded and modified upon the githut the BENDR code is downloaded and modified upon this githut and the LaBraM code is downloaded and modified upon this github 16 Other baselines implementations on the TUAB and TUEV datasets are also referred to this repo The EEGPT structure implementation is mainly modified upon the
github [8] and also references the code of BENDR. The ROPE embedding is implemented using the
code of Roformer [50] github
We mainly use MNE packag
20
braindecode package and torcheeg package to load and prepro-
cess the data.
F MORE EXPERIMENT RESULTS DESCRIPTION
In the comparative experiments, the baseline model BIOT demonstrated accuracy of 54% in the binary classification task of EEG P300 signals on PhysioP300 datase[4 suggesting its ineffectiveness.
The P300 signal is event-related potential (ERP) component commonly observed in EEG record- ings. is characterized by a positive deflection voltage occurring approximately 300 milliseconds after the presentation of stimulus. The P300 component is primarily concentrated around the 300 ms mark and is crucial for tasks involving attention and stimulus evaluation.
One plausible reason for the suboptimal performance of the BIOT model is the inherent nature of the P300 signal classification task, which predominantly relies on time-domain waveform features. The BIOT model, however, applies fast Fourier transform (FFT) to the input signals, thereby primarily extracting frequency-domat
features. This fundamental discrepancy between the feature domain utilized by the model and the domain that is most pertinent the classification task likely accounts for the observed lack of efficacy. Additionally, while the P300 component is concentrated around the 300 ms mark, the BIOT model applies FFT on I-second length patches, which is overly extensive and results in significant information loss. The "significant information loss" mainly reflected the fact that the BIOT model only retains the spectral energy information for Is of each patch after the FFT, and discards the phase information. similar situation also occurs in the binary classification task for ERN signals [7 the KaggleFRN dataset.
G VISUALIZATION
G.1
Channel Relationship Visualization
To further demonstrate the effectiveness of the model implementation, this paper visualizes the trained model. During the pretraining process, channel information is encoded through learnable channel embeddings. This section visualizes the similarity between positional encodings to show the relationship information learned by the model from the data.
Figure (a) shows the channel relationship diagram after model pretraining, using the cosine similarity between channel embeddings to represent the relationships between channels. The left figure shows relationships with similarity greater than 0.5, where channels are clustered based on their front, back, left, and right positions. The right figure shows relationships with similarity greater than 0.4, revealing not only nearby relationships but also long- long-distance relationships across brain regions.
htap::///ittbboooooooooooooorreo transformmm
19 "anggc//ggttticccmm/////yiiilllo
2 http://giithh..mm//haaacccco///i
22

---

b
(a) Channel Embedding Similarity 2
(b) Channel Embedding Similarity 2
Figure Channel Embedding Similarity Relationships
Figure 12: Channel Embedding Similarity Connection Diagram
23

---

Figure 12 shows the cosine similarity relationships between channel embeddings, with electrode positions placed according their actual locations. Solid lines indicate relationships with similarity greater than 0.3, while dashed lines indicate relationships with similarity between 0.1 and 0.3. It can be seen that channels close proximity have higher similarity, and there also significant similarity between distant electrodes on opposite sides.
G.2 BCIC2A Experiment Results Visualization
Left Hand
Right Hand
Foots
Tongue
Figure 13: BCIC2A Channel and Class Pearson Correlation Diagram
Figure 13 shows the correlation between channels and motor imagery classes detected using the channel perturbation method after training the BCIC2A dataset. Gaussian multiplicative random noise is randomly added to the signal amplitude of each channel, and the Pearson correlation between the noise intensity and the changes in the corresponding class logits is calculated and presented as a heatmap. Symmetric relationships are observed for electrodes related to left and right hand movements, with bilateral electrodes corresponding foot movements, and distinct channels corresponding to the four different classes.
T ENE projection
Left Tand Hand
Torrju
-10
Figure BCIC2A I-SNE Diagram
Figure shows the t-SNE diagram of features learned by the model. It can be seen that the features of the four classes clustered in four different regions, demonstrating linear separability.
Figure 15 shows the confusion matrix, indicating that the recognition performance for foot movements the best, while samples from other classes are often misclassified as foot movements.

---

Figure 15: BCIC2A Confusion Matrix Diagram
Attenice
Mon Allation
DXT Altanias
1-22
Mour Altorion
Attennee
Figure 16: P300 peatio-temporal Attention Diagram
G.3 PhysioP300 Experiment Results Visualization
Figure 16 shows the model's s attention distribution for the P300 task. For temporal attention, the model focuses more on the -0.1 Is time period. For spatial attention, the model focuses more on the signals from the F1, FZ, F2, and FCZ channels.
T-SNE projection
1-
-2-
Figure 17: P300 t-SNE Diagram
Figure 17 shows the t-SNE diagram of features extracted by the model for the two classes of samples.
The features the two classes are distributed on two sides, with a confusion belt in the middle, but overall the classes linearly separable.
25

---

130
Figure 18: P300 Confusion Matrix Diagram
The confusion matrix shows that the performance for the non-target class is better.
H LIMITATIONS
Firstly, although we have collected a large-scale multi-task mixed EEG dataset and utilized a model structure with extensive parameters during the pretraining phase, there remains significant disparity compared to today's 's large vision and language models. Our work is still in the exploratory phase of training large EEG models learn general representations. Encouragingly, shown that large EEG models can continue to be optimized, achieving considerable performance improvements compared existing methods developed for specific BCI tasks and general large EEG models. Secondly, while EEGPT requires only linear-probing to adapt to small-scale downstream tasks, may still result in high memory costs. Finally, EEGPT was pretrained using 4s EEG data. This may strict model's 's capability. It is worth exploring the training of large EEG models with long recordings.
experiments have
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We point out the limitations of the work and explain why they are important to consider in H
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiiatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
27

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: This paper has detailed instructions how to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
28

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We release all code and data preprocess scripts the supplemental material and figshare
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https//ccppbbccc/uuuddeeeoo Coaee//ddeeeebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specify all the training and details in 3.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results are accompanied by error bars, confidence intervals, or statistical significance tests.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provide sufficient information on the computer resources in 3.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Question:
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
The paper discusses the potential positive societal impacts, such as advance- ments in BEG-based diagnostics and therapeutic applications, which could improve health- care outcomes.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper of existing assets.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
31

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We release code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "IfMlSTJfF0", "reviewer": "Reviewer_DgXb", "summary": "The paper presents a new pretrained model called EEGPT (EEG pretrained transformer), designed to improve the analysis of EEG (electroencephalography) signals. EEGPT is a model with over 10 million parameters (up to 100M for the largest model) that aims to solve common problems in EEG analysis, where the challenges are low signal quality and high variability between individuals. The model introduces a special method called \"dual self-supervised learning\" to extract better features from EEG data. This method includes two main techniques: the usual masked reconstruction loss, and additionally the alignment loss. The alignment loss is a special loss introduced inside the masked auto-encoder to “force” the learned features of the masked patches inside the encoder (and predictor) to be aligned with spatial and temporal patches (all masked and unmasked). This is achieved by adding a momentum encoder and computing the loss between the predictor (global integration of summary patches of each time j) and momentum encoder (spatial integration of all unmasked patches for each time j).\n\nThe model was trained on several large EEG datasets and tested on various tasks like classifying motor imagery, detecting event-related potentials, and identifying sleep stages. EEGPT performed better than existing models in these tasks, showing it is effective and scalable.", "questions": "Even though there is the ablation study for the alignment loss, it is not clear from the paper what is the main motivation behind it, and why this particular design is used. To be more concrete, we have the summary tokens from the encoder which are only created from patches at specific time point j. By design we have now an encoder that integrates all the patches spatially (for each timepoint). The confusing part for me is the predictor: on the one hand it tries to capture global features between all tokens in the time domain, on the other hand the alignment loss forces the tokens to only capture the local features at specific timepoint. What is the main motivation behind this? Note that from the experiments it is shown that the method works quite well, but it was confusing for me in the beginning to understand how did you come up with this design choice. Stated differently: how does the model perform if we do the alignment loss before the predictor, or if we remove the predictor. \n\nHow did you evaluate the TUAB dataset since your model can only work 4s windows. The TUAB dataset can have many minutes to hours of EEG data. Same goes for TUEV.\n\nThe evaluation for TUAB seems to not include in the comparison the best performing models in the literature (see limitations). Are there also other models for the other tasks that perform better and are not included in the paper?", "limitations": "The model is only trained with 4s of EEG data and the pretraining set is very small. Although the initial experiments are promising, It is not clear how the model generalises in other datasets and downstream tasks. The model is not evaluated how it performs if we finetune the model (instead of linear probing).", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The model introduces the “dual self-supervised technique” and to my knowledge it is novel.\n2. The model performs better in the proposed tasks presented in the paper. However, the model is also significantly larger than the competition.\n3. The paper performs an ablation study showing the importance of their decision of using the alignment loss (L_A). Without L_A, accuracy drops from 58.46 to 52.87.\n4. The paper presents a range of experiments besides the model training and evaluation, providing a better understanding of the model design and behaviour. This includes methods like scaling laws, relationships of the channel embeddings (which are learned), and visualizations on the correlations of the motor imagery tasks and the channels for the BCI competition (BCIC2A dataset).\n5. The downstream tasks are performed only with linear probing (no need for full finetuning) and the model still outperforms other models.", "weaknesses": "1. The experiments done for the model design and behaviour such as scaling laws, ablation studies, channel embedding relationships, etc, are done only on one downstream task: namely the BCIC2A dataset. How does the model perform on the other tasks, what is the reason for not presenting the results on the other datasets?\n2. The paper presents interesting scaling law experiments, showing potential in the method to scale to larger model size. However the experiments are too limited to be able to state the scaling laws as given in the paper (e.g. the ACC = (33.6*N)^0.029). I would argue one has to test it either to more tasks, or have a unified metric that uses many datasets for the scaling law.  Also, how does the law depend on the amount of training data used?\n3. The design of the model was at first confusing and I still have trouble understanding the main motivation from some of the decisions. The ablation studies are a great help here, but I think a few more ablation studies or explanations on the design of the model would clarify the paper better (see Questions section).\n4. I have a few doubts on the evaluation of the baselines. Although I am not certain what are the best achievable accuracies in many of the tasks, for TUAB we have many different models that might perform better: We have models up to 86% (ChronoNet) in Table 1 in here (https://www.sciencedirect.com/science/article/pii/S2213158223001730) and above 90% in here (https://arxiv.org/pdf/2401.10283). I know there are certain differences: accuracy vs balanced accuracy and the second paper has a different evaluation for the highest accuracy. However, the TUAB dataset is quite balanced to my knowledge and still in the second paper in Table 1, there are multiple models close to 90%. Why are these models not included in the comparison, and since we have this discrepancy in this task, do other models exist also for other tasks as well?\n5. Why is the pretraining set very small? In terms of participants, the downstream set is much bigger than the pretraining set. In addition, the TUEG dataset is bigger than all the datasets used here (combined). Is there any reason why the authors chose these datasets in particular? It is not clear from the paper why this is the case. Usually, during pretraining the dataset is much bigger compared to the downstream task. In this paper this does not seem to be the case. In fact, many of these datasets (all?) used during pretraining do have labels (i.e. why self-supervised).\n6. In some sections it was not clear which evaluation dataset has been used to produce the results. For example in Section 3.4 there is Table 5 but it is not clear which model size and which dataset has been used to produce the results. Because of the accuracy number 58.46 I could infer that it was BCIC2A, but it is not mentioned anywhere. For scaling laws it is mentioned in the text, but it is better to have it on the table description. \n7. Many minor annoyances, which make me wonder how much care the authors took to write this paper:\n-  Misspellings, for example: Accuarcy vs accuracy, Appandix vs Appendix, etc.\n- The references are messed up. Just look at the first two references [1] and [2], where every author is listed at least 2 times, sometimes even 3 times!\n- You need to put a space in front of a reference, like this [1]. You almost always have no space, but not even this is consistent, because sometimes you do.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback. We have uploaded a revision and used blue to mark the new changes. Our detailed responses are as follows.\n\n#### **W1**\nWe added more test results for more datasets in the ablation experiment and scale law experiment, see Revision Section 3.4 & Appendix A. Channel embedding relationships are computed based on channel embeddings extracted from the pretrained model and are independent of the downstream task.\n\n#### **W2**\nIn the scale law experiment, we added test results for the TUAB dataset and the BCIC-2B dataset, see Revision Appendix A.4. The results on the TUAB dataset and the BCIC-2B dataset show that they present a scale law for the performance metrics with respect to the model size.\n\nWe added pretraining experiments using 100%, 50%, 25%, and 12.5% of the training data and tested them on the downstream tasks of BCIC-2A and BCIC-2B, see Revision Appendix A.5. The results show that their performance metrics exhibit a scale law with respect to the amount of data.\n\n#### **W3&Q1**\nThe design motivation of the predictor: (1) on the one hand, it predicts the representations of the masked \"50% time patches\" (equivalent to the BERT-style task), which allows the encoder to extract features in each local patches, that contribute to the task of completing masked patches in the time dimension; (2) on the other hand, it avoids the collapse of representation due to the direct alignment of the outputs from the encoder and momentum encoder (i.e., the parameters of the two are difficult to change during pretraining), and in this way, it is similar to the predictor in BYOL [1]. Also due to the design of (1) we can avoid predictor learning to be a identity map.\n\nWe conducted a pretraining experiment after removing the predictor, the variation of $L_R$ loss with the number of iteration steps during training is shown in Figure 6 of the Revision Appendix A.1. In Figure 6, the $L_R$ loss of the model without the predictor does not decrease, which indicates that directly aligning the outputs of the encoder and the momentum encoder does lead to the problem of representation collapse, resulting in the model can not learn meaningful representation.\n\n#### **W4&Q3**\n(1) The difference in accuracy on TUAB is due to the different sample lengths used by the different methods. For a fair comparison, we use the same evaluation approach on TUAB as in the LaBraM and BIOT papers, which both use 10-second samples for classification. ChronoNet [2] uses a window of different sizes of at least 60s for classification and achieves an accuracy of 86%. In the second paper [3] it used 1-minute samples for classification. This shows that different sample lengths can have a huge impact on the experimental results.\n\n(2) Our aim is to develop better pretraining methods so that the models can achieve better performance (more generality) on a wide range of EEG tasks. Therefore on other tasks we only compare with similar pretrained models. On some tasks, there exist complex models that may achieve better results using handcrafted features or dedicated designs, but they are developed for specific tasks and may be more difficult to generalize across tasks compared to pretrained models.\n\n#### **W5**\nTUEG belongs to the clinical dataset and the data are related to patients and brain diseases, the channels of TUEG dataset are mostly 23 channels 10-20 standard electrodes, not diverse enough. We trained our model on a limited number of pretraining datasets, which includes more brain-computer interface related tasks such as motor imagery, error-related potentials, sleep stage detection, and identity recognition. These datasets also contain more channels. The TUEG dataset was chosen as the downstream task dataset in order to better compare with other pretrain models such as BIOT.\n\nDespite the abundance of labeled data for epilepsy and sleep task, many works [4] [5] [6] have used self-supervised methods to achieve performance that exceeds that of supervised methods. \n\n|Methods|Model Size|Balanced Accuracy|AUROC|\n|-|-|-|-|\n|Ours (no pretrained)|25M|0.7553±0.0014|0.8260±0.0018|\n|Ours|25M|0.7983±0.0030|0.8718±0.0050|\n\nWe added test results using random initialization parameters on TUAB (see Appendix A.7), and according to our experimental results, even with a small amount of pretraining data, the pretrained model works better on the TUAB dataset compared to the unpretrained model.\n\n#### **W6**\nWe added more detailed descriptions for the tables, see Revision PDF Section 3.4. In Table 5, we changed the headers to BCIC-2A-BAC, BCIC-2B-AUROC, and KaggleERN-AUROC. In Table 6, we update the header to BCIC-2A-BAC.\n\n#### **W7**\nWe have fixed all spelling errors and regularized the citation reference format, see Revision PDF.\n\n#### **Q2**\nIn Revision Appendix C.2.6 Table 12&13, we show the architecture of our model used in the TUAB and TUEV. \n\n**[ Model architecture for TUAB & TUEV ]**\n\n|In Size|Opt|kernel|stride|groups|padding|\n|-|-|-|-|-|-|\n|23xT|conv1d,norm,gelu|1|1|1|0|\n|20xT|conv1d,norm,gelu,dropout|K|1|20|K/2|\n|20xT|encoder|64|64|||\n|(T/64)x4x512|flatten,linear|||||\n\nThe 23-channel input is first to reduce the number of channels to 20 by the convolution (K=15 for TUAB and 55 for TUEV). Then, the eegpt-encoder uses the 20 channel embeddings and maps 64-length window segments of the input signals to 4 (number of summary tokens)x512-dimensional features. Finally, the flatten and linear layers are used to output the final classification score.\n\n#### **L1**\nSee author rebuttal ablation experiment.\n\n[1] Bootstrap your own latent-a new approach to self-supervised learning.\n\n[2] ChronoNet: A deep recurrent neural network for abnormal EEG identification.\n\n[3] Window Stacking Meta-Models for Clinical EEG Classification.\n\n[4] Channel-Aware Self-Supervised Learning for EEG-based BCI.\n\n[5] MAEEG: masked auto-encoder for EEG representation learning (2022).\n\n[6] BIOT: Cross-data biosignal learning in the wild.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6eojgCn4j9", "reviewer": "Reviewer_23GR", "summary": "The authors propose a novel pretraining strategy, EEGPT, that is essentially a multi-task self-supervision loss consisting of a masked autoencoder-style reconstruction objective, and an alignment loss that is reminiscent of knowledge distillation approaches such as data2vec. EEGPT is applied to representation learning of electroencephalography (EEG) data, and the authors demonstrate the pretrained model flexibility across a variety of downstream EEG decoding tasks (motor imagery, sleep staging, ERP detection).", "questions": "1. Line 59: The authors claim that the low SNR of EEG “...make it challenging to learn abstract features using masked autoencoders”. However, there have been multiple works that use MAE-style pretraining to successfully learn features from EEG, such as: [1] Chien, et al. MAEEG: Masked Autoencoder for EEG Representation Learning. arXiv:2211.02625. [2] Wu, et al. Neuro-BERT: Rethinking Masked Autoencoding for Self-supervised Neurological Pretraining. IEEE JBHI. \n2. Eqn 9: Did the authors try using a weighted loss with weight $\\lambda$? Or does each individual loss take on similar values? $\\mathcal{L} = \\mathcal{L}_A + \\lambda \\mathcal{L}_R$\n3. Section 2.4: Can the authors clarify which “Features” are passed to the linear classification head? Are these the summary tokens (passed to the Predictor during pretraining)? Or are they the non-summary tokens?\n4. Section 3.1: Were any notch filters used to suppress electrical noise in datasets used for pretraining or downstream tasks?\n5. Section 3.2: Why were 58 electrodes used? Is it because these 58 electrodes were common across the pretraining datasets? Also, what happens at inference time if a new electrode not belonging to the original set of 58 locations appears?\n6. Section 3.2: Is there any normalization applied to the input signal? I saw in the appendix that each dataset has a different normalization (EA on entire session, z-score on each input, etc.). Does that mean that the normalization is dataset-dependent?\n7. Line 197: Missing reference for OneCycle learning rate\n8. Tables 2&3: The proposed EEGPT model has ~8X as many parameters as the second largest model (BIOT). It would be a fairer comparison if the authors also reported BAC for a similarly sized EEGPT model, such as tiny3 or little (from Table 6).\n9. Table 5: Can more description be added to this table? For example, BAC stands for Balanced Accuracy I assume? And which dataset is BAC being reported for here? Or is it an average across datasets?\n10. (Minor comment) The title “EEGPT” is slightly misleading, since it makes the reader think that the model is being pretrained using an autoregressive reconstruction objective such as in the popular GPT language models (e.g. predict the next token given the previous tokens). I would recommend changing to a different name to avoid confusion.\n11. (Minor comment) Line 186: Spelling error “Appendix”\n12. (Minor comment) Line 499: What does EA in “EA normalization” stand for? I tried to find it elsewhere in the paper, but could not find it..", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "a. Originality:  The individual components of EEGPT are not original. The masking strategy and reconstruction loss are based on Masked Autoencoder which was originally developed for vision, and has since been applied to time-series data (including EEG) previously. Similarly, the alignment loss is highly similar to that of data2vec, except that data2vec uses a smoothed L1 loss. However, the combination of these two objectives into a multi-task self-supervision loss is novel (to my knowledge). According to the ablation results in Table 5, it does seem that the two objectives compete against each other, but in a way that is constructive for the downstream decoding tasks.\n\nb. Quality: The conclusions made by this paper are mostly sound. The authors demonstrate the improvements of EEGPT over other SOTA approaches across a comprehensive list of downstream EEG decoding tasks. Also of note, the authors made an effort to demonstrate the design choices of EEGPT in their ablation analysis.\n\nc. Clarity: The paper is overall clear in its writing. There are some missing details, which I have detailed below in the Questions section.\n\nd. Significance: The pretraining approach is likely to be reused in other works. The model generalizability across a diverse set of EEG decoding tasks is especially compelling, since it suggests that this model may be used as a generalized foundation model for EEG.", "weaknesses": "b. Quality: One weakness of this work is that the proposed model has significantly more parameters than the SOTA approaches it is comparing against. It remains unclear if the authors have developed a better pretraining approach, or if the improvements in the downstream tasks are due to higher expressivity in the base model architecture. Could the authors add additional comparisons where the number of parameters EEGPT matches the number of parameters in the other SOTA approaches?\n\nAnother weakness is that the comparison to pretraining approaches such as BIOT is unfair. The authors state in Appendix E: “the BIOT model applies an FFT on 1-second length patches, which is overly extensive and results in significant information loss\". However, this does not mean that the pretraining recipe in this paper is better than BIOT. It could be the case that BIOT performs similarly, it just needs to be retrained with different patch length and/or patch stride. I understand that retraining BIOT with new parameters may not feasible in the rebuttal timeframe, but I think this should be clarified in the text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your appreciation and constructive comments. We have uploaded a revision and used blue to mark the new changes.\n\n---\n\n#### **W1&Q8**\n\nSee Revision Section 3.3 for test results of model variants with comparable number of parameters.\n\n**[Table: Results of experiments on TUAB]**\n\n|Methods|Model Size|Balanced Accuracy|AUROC|\n|-|-|-|-|\n|BIOT|3.2M|0.7959±0.0057|0.8815±0.0043|\n|Ours-Tiny|4.7M|0.7959±0.0021|0.8716±0.0041|\n|Ours|25M|0.7983±0.0030|0.8718±0.0050|\n\n**[Table: Results of experiments on TUEV]**\n\n|Methods|Model Size|Balanced Accuracy|Weighted F1|Cohen's Kappa|\n|-|-|-|-|-|\n|BIOT|3.2M|0.5281±0.0225|0.7492±0.0082|0.5273±0.0249|\n|Ours-Tiny|4.7M|0.5670±0.0066|0.7535±0.0097|0.5085±0.0173|\n|Ours|25M|0.6232±0.0114|0.8187±0.0063|0.6351±0.0134|\n\nThe number of parameters in the table for Ours-Tiny is 4.7M comparable to the BIOT model. Compared to BIOT, our Tiny model improves about 4% on the TUEV dataset and achieves comparable performance with it on the TUAB dataset. Compared to the BIOT method, we used a smaller pretraining dataset and did not use more datasets containing epilepsy samples, but still achieved good results on both datasets. \n\n#### **W2**\n\nWe agree with you that on some tasks it is possible that the pretraining scheme in this paper performs similarly to the BIOT. The statement in Revision Appendix F aims to explain the possible reasons for the poor performance of the BIOT on the BCIC-2A and BCIC-2B motor imagery tasks (see Section 3.3, Table 4). The \"significant information loss\" is mainly reflected in the fact that the BIOT model only retains the spectral energy information for 1s of each patch after the FFT, and discards the phase information. We have provided additional explanations for this, see Revision Appendix F.\n\n#### **Q1**\n\nWhile other work has validated the ability of MAE pretraining methods to learn features in corresponding EEG tasks, our work would like to emphasize that the low signal-to-noise ratio of EEG prevents the model from learning \"high-quality\" representations through MAE pretraining methods. Our ablation experiments verify this, see Revision Section 3.4.\nSee author rebuttal Table 1 for more details. In the absence of $L_A$ loss, the model's performance degradation on all datasets is significant (6% to 9%).\n\n#### **Q2**\n\nIn our experiments, despite the presence of two loss functions, no weights were explicitly introduced during the optimization process to balance the two losses. This approach also simplifies the training process of the model. \n\n#### **Q3**\n\n\"The summary tokens (passed to the Predictor during pretraining)\" are passed to the linear classification head. We provide additional clarification, see Revision Section 2.4.\n\n#### **Q4**\n\nIn Revision Appendix C we describe in detail how we preprocessed each dataset. For all datasets we did not target the use of notch filters to suppress electrical noise.\n\n#### **Q5**\n\nThis is because these 58 electrodes are the electrodes that are present in all pretraining datasets and are the intersection of their electrode sets. These 58 electrodes cover as much as possible the electrodes in the international 10-10 EEG system standard [1]. During the pretraining phase, we removed channel data from the data sample that were not these 58 electrodes.\n\nIf a new electrode that does not belong to the 58 pretrained electrodes appears in the downstream task, the signal from the new electrode can be mapped to the neighbouring/similar electrode input of the encoder model by using the adaptive spatial filter (see Revision Section 2.4), which is done by training the adaptive spatial filter on the downstream task dataset.\n\n#### **Q6**\n\nIn Revision Appendix B, we describe in detail the normalization and preprocessing approach for each dataset. The normalization and preprocessing approach is dependent on different EEG paradigms.\n\n#### **Q7**\n\nWe have added references to OneCycle Learning Rate [2], see Revision Section 3.2 & Appendix C.2.\n\n#### **Q9**\n\nWe added more detailed descriptions for the tables, and in Table 5 of Revision Section 3.4, we changed the headers to BCIC-2A-BAC, BCIC-2B-AUROC, and KaggleERN-AUROC.\n\n#### **Q10**\n\nThanks for commenting on the potential confusion with \"EEGPT\". We chose this to highlight the application of pretrained Transformers on EEG signals, despite not using autoregressive task pretraining. It showcases the ability of to learn universal pattern from massive data. However, we've clarified that our model isn't pretrained using autoregressive objectives in our paper's introduction (Revision Section 1) to prevent misunderstandings, while maintaining our original title intent.\n\n#### **Q11**\n\nWe have fixed spelling errors, see Revision PDF.\n\n#### **Q12**\n\nWe have added a citation and description of EA normalisation [3], see Revision Appendix C.2. EA normalization is a common normalization method applied to EEG data from motor imagery tasks. This approach align EEG trials from different subjects in the Euclidean space to make them more similar, and hence improve the learning performance for a new subject.\n\n[1] The five percent electrode system for high-resolution EEG and ERP measurements.\n\n[2] Super-convergence: Very fast training of neural networks using large learning rates.\n\n[3] Transfer learning for brain–computer interfaces: A Euclidean space data alignment approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BCCGHJokPD", "reviewer": "Reviewer_izki", "summary": "The paper presents EEGPT, a 10-million-parameter pretrained transformer model designed for universal EEG feature extraction. The model employs a dual self-supervised learning method for efficient feature extraction and demonstrates state-of-the-art performance on various downstream tasks with linear probing.", "questions": "1. It is unclear whether the baselines in Table 3 are based on linear probing or fully fine-tuned models. Clarification is needed.", "limitations": "The authors have discussed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed EEGPT model introduces a dual self-supervised learning method combining spatio-temporal representation alignment and mask-based reconstruction, which enhances representation quality and model robustness.\n2. EEGPT achieves state-of-the-art performance on a range of downstream tasks such as motor imagery classification, ERP detection, and sleep stage detection.\n3. The hierarchical structure for decoupled processing of spatial and temporal information reduces computational complexity and enhances flexibility for BCI applications.\n4. Comprehensive experiments demonstrate the superior performance of EEGPT across multiple EEG tasks.", "weaknesses": "1. The function of the adaptive spatial filter is unclear. Why is this module not included during pre-training? An ablation study is needed to evaluate the effectiveness of this component.\n2. In Table 2, the authors use different metrics than in Table 3. Cohen's Kappa should be used consistently across both tables. Additionally, the results of LaBraM on these two datasets should be reported to provide a more comprehensive comparison, even if EEGPT, which uses linear probing, might not surpass LaBraM.\n3. The technical contribution of the paper appears incremental. The framework is largely based on the context autoencoder (CAE) with modifications such as the adaptive spatial filter, multiple summary tokens, and rotary temporal embeddings.\n4. During pre-training, EEGPT uses EEG data of the same configuration. In contrast, LaBraM leverages various datasets with different configurations, which might make EEGPT less universal.\n5. The ablation study is conducted on a single dataset, which is not persuasive enough to demonstrate the effectiveness of the proposed modules and their scalability.\n6. EEGPT uses different model settings for different downstream datasets, which may limit its generalizability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful feedback. We have uploaded a revision with the changes marked as blue. Our detailed responses are as follows:\n\n---\n\n#### **W1: The function of the adaptive spatial filter is unclear. Why is this module not included during  pretraining? An ablation study is needed to evaluate the effectiveness of this component.**\n\nThe adaptive spatial filter is designed for the downstream task, adapting the EEG signal for each task/data distribution to the input of the encoder. In the downstream task, the model learns to adjust the parameter weights of the adaptive spatial filter, which maps (scale and project) the downstream task data to the same distributions as the pretrain data. For example, it can adapt to the different EEG references (e.g., for the epilepsy classification task, the double banana bipolar montage reference [2] with the features are more significant, and adaptive spatial filters allow for such a reference.).\n\nThe adaptive spatial filter is designed only for downstream tasks. It can be of limited use during  pretraining, since we have uniformly pre-processed the  pretraining dataset and assumed that these data are in the same distribution. On the other hand, during  pretraining, since the strategy of masking in both the spatial and temporal dimensions is used, the channels are not consistent in each input, and consequently leads to distributional drift and reduces training efficiency [3].\n\nIn the ablation experiments, we added the results of the ablation experiments that exclude the adaptive spatial filter in the downstream task, as shown in Revision Appendix A.3 Table 8 & author rebuttal Table 2.\n\n#### **W2: In Table 2, the authors use different metrics than in Table 3. Cohen's Kappa should be used consistently across both tables. Additionally, the results of LaBraM on these two datasets should be reported to provide a more comprehensive comparison, even if EEGPT, which uses linear probing, might not surpass LaBraM.**\n\nWe added the Cohen's Kappa metrics to Table 3 and added the LaBraM-Base results, see Revision Appendix A.6.\n\n**[Table: Results of experiments on TUAB]**\n\n|Methods|Model Size|Balanced Accuracy|AUROC|\n|-|-|-|-|\n|BIOT|3.2M|0.7959±0.0057|0.8815±0.0043|\n|LaBraM|5.8M|0.8140±0.0019|0.9022±0.0009|\n|Ours-Tiny|4.7M|0.7959±0.0021|0.8716±0.0041|\n|Ours|25M|0.7983±0.0030|0.8718±0.0050|\n\n**[Table: Results of experiments on TUEV]**\n\n|Methods|Model Size|Balanced Accuracy|Weighted F1|Cohen's Kappa|\n|-|-|-|-|-|\n|BIOT|3.2M|0.5281±0.0225|0.7492±0.0082|0.5273±0.0249|\n|LaBraM|5.8M|0.6409±0.0065|0.8312±0.0052|0.6637±0.0093|\n|Ours-Tiny|4.7M|0.5670±0.0066|0.7535±0.0097|0.5085±0.0173|\n|Ours|25M|0.6232±0.0114|0.8187±0.0063|0.6351±0.0134|\n\nLaBraM employs a larger  pretraining dataset than ours (reference to [1] Appendix D), which also contains the TUEG dataset that is similar to the TUAB and TUEV distributions, and whose paper illustrates the scale effect of the amount of  pretraining data, which may have facilitated the learning of a richer and more downstream task-adaptable representation for LaBraM.\n\n#### **W4: During  pretraining, EEGPT uses EEG data of the same configuration. In contrast, LaBraM leverages various datasets with different configurations, which might make EEGPT less universal.**\n\nIn this paper, we unified the  pretraining dataset into 58 channels, each channel has the same amount of data, making each channel of equal importance. In addition to this, LaBraM uses more eeg reference configuration (the double banana bipolar montage reference [2]) for epilepsy data, which may lead to stronger generalization (for epilepsy tasks). We conducted initial experiments using a limited dataset and will gradually expand to larger datasets (e.g., epilepsy dataset) to enhance the generalizability of EEGPT.\n\n#### **W5: The ablation study is conducted on a single dataset, which is not persuasive enough to demonstrate the effectiveness of the proposed modules and their scalability.**\n\nIn the ablation experiment, we added test results for the BCIC-2B and KaggleERN datasets, as shown in the table below, and see Revision Section 3.4. See author rebuttal Table 1 for more details.\n\n#### **Q1: It is unclear whether the baselines in Table 3 are based on linear probing or fully fine-tuned models. Clarification is needed.**\n\nThe baseline in Table 3 uses the full fine-tuning method, which we have supplemented, see Revision Section 3.2.\n\n[1] Jiang, W. B., Zhao, L. M., & Lu, B. L. (2024). Large brain model for learning generic representations with tremendous EEG data in BCI. *arXiv preprint arXiv:2405.18765*.\n\n[2] Rosenzweig, I., Fogarasi, A., Johnsen, B., Alving, J., Fabricius, M. E., Scherg, M., ... & Beniczky, S. (2014). Beyond the double banana: improved recognition of temporal lobe seizures in long-term EEG. *Journal of Clinical Neurophysiology*, *31*(1), 1-9.\n\n[3] Tian, K., Jiang, Y., Diao, Q., Lin, C., Wang, L., & Yuan, Z. (2023). Designing bert for convolutional networks: Sparse and hierarchical masked modeling. arXiv preprint arXiv:2301.03580.\n\n---\n\nThanks again for your comments. Hope our rebuttal has addressed all your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for your time and constructive feedback. During the rebuttal, we have prepared a revision and used blue to mark the new changes. Below are the results of the added experiments as the response.

**[Table 1: Ablation study for pretraining methods (Appendix A.2)]**

In the ablation experiment, we added test results for the BCIC-2B and KaggleERN datasets, as shown in the table below. The 'BAC' is the Balancing Accuracy, and more detailed descriptions of the metrics can be found in Revision Appendix D.

| Variants                 | $\mathcal{L}_{A}$ | $\mathcal{L}_{R}$ |       BCIC-2A-BAC |     BCIC-2B-AUROC |   KaggleERN-AUROC |
| ------------------------ | ----------------: | ----------------: | ----------------: | ----------------: | ----------------: |
| A: w/o $\mathcal{L}_{A}$ |             37.13 |              0.57 |     0.5287±0.0086 |     0.7264±0.0381 |     0.5752±0.0164 |
| B: w/o $\mathrm{LN}$     |              0.15 |             0.002 |     0.5567±0.0088 |     0.7920±0.0012 |     0.5891±0.0227 |
| C: w/o skip              |              0.12 |              0.56 |     0.5796±0.0011 |     0.7702±0.0122 |     0.6356±0.0296 |
| D: with all              |              0.24 |              0.56 | **0.5846±0.0070** | **0.8059±0.0032** | **0.6621±0.0096** |

The results of the ablation experiments with the addition of the dataset are still in line with the conclusions in the original paper:(1) the model performance decreases significantly on all datasets without the $L_A$ loss (6% to 9%); (2) the performance of the variant B model without layer normalization of the target patches decreases by 3%, 1%, and 7% on BCIC-2A, BCIC-2B and KaggleERN, respectively; (3) the performance of the variant C model with the removal of skip connection was reduced by 1%, 3%, and 3% on the three datasets, respectively.

**[Table 2: Ablation experiments of fine-tuning methods (Appendix A.3)]**

In the ablation experiments, we added experiments comparing the linear probing method with the full fine-tuning method, see Revision Appendix A.3, as shown in the table below.

| Variants | ASF  | L-P  | BCIC-2A-BAC       | BCIC-2B-AUROC     | KaggleERN-AUROC   |
| -------- | ---- | ---- | ----------------- | ----------------- | ----------------- |
| A        |      |      | 0.5774±0.0072     | 0.7871±0.0054     | 0.6078±0.0101     |
| B        | ☑️    |      | 0.5183±0.0155     | 0.7541±0.0083     | 0.6110±0.0019     |
| C        |      | ☑️    | 0.5586±0.0089     | 0.7974±0.0030     | 0.6463±0.0081     |
| D        | ☑️    | ☑️    | **0.5846±0.0070** | **0.8059±0.0032** | **0.6621±0.0096** |

In the above table, ASF denotes with adaptive spatial filter, instead of directly feeding the signal into the model; L-P denotes using linear probing, instead of using the full fine-tuning of the model. Model variants A and C are models with full fine-tuning and linear probing, respectively, after excluding the adaptive spatial filter. Model variants B and D are models with full fine-tuning and linear probing after using adaptive spatial filter, respectively. The results show that on the BCIC-2B and KaggleERN datasets, variants C and D tested with linear probing achieve better results than variants A and B using full fine-tuning; on the BCIC-2A dataset, variant A with full fine-tuning and no adaptive spatial filter is close to variant D with linear probing and adaptive spatial filter. Overall, the variants C and D (using linear probing) outperform A and B. Also, the results show that variants B and D with adaptive spatial filter tested on the BCIC-2A, BCIC-2B and KaggleERN datasets achieve better results compared to variants A and C without adaptive spatial filter.

Meta-review: This paper develops a 10M pre-trained transformer model	for EEG
processing.  The transformer is trained with masked self-supervised
training with spatio-temporal representation alignment.  Results on
several down-stream tasks are improved relative to comparison
approaches.  All reviewers scored this with a 5 or 6 recognizing that
it is worth presenting to the NeurIPS conference.
Decision: Accept (poster)