EEGPT: Pretrained Transformer for Universal and Reliable Representation of EEG Signals

Authors: guagnyu  wang
wenchao  liu
yuhong  he
cong  xu
lin  ma
haifeng  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 11:29:07

Abstract: Electroencephalography (EEG) is crucial for recording brain activity, 
  with applications in medicine, neuroscience, and brain-computer interfaces (BCI). 
  However, challenges such as low signal-to-noise ratio (SNR), high inter-subject variability, and channel mismatch complicate the extraction of robust, 
  universal EEG representations. 
  We propose EEGPT, a novel 10-million-parameter pretrained transformer model designed for universal EEG feature extraction. 
  In EEGPT, a mask-based dual self-supervised learning method for efficient feature extraction is designed. 
  Compared to other mask-based self-supervised learning methods, 
  EEGPT introduces spatio-temporal representation alignment. 
  This involves constructing a self-supervised task based on 
  EEG representations that possess high SNR and rich semantic information, 
  rather than on raw signals. 
  Consequently, this approach mitigates the issue of poor feature quality typically 
  extracted from low SNR signals.
  Additionally, EEGPT's hierarchical structure processes spatial and temporal information separately, 
  reducing computational complexity while increasing flexibility and adaptability for BCI applications. 
  By training on a large mixed multi-task EEG dataset, we fully exploit EEGPT's capabilities.
  The experiment validates the efficacy and scalability of EEGPT, 
  achieving state-of-the-art performance on a range of downstream tasks with linear-probing.
  Our research advances EEG representation learning, offering innovative solutions for bio-signal processing and AI applications.
  The code for this paper is available at: https://github.com/BINE022/EEGPT

Main paper: 
 introduction: 
 Electroencephalography (EEG) dynamically reflects the brain's functional state by recording electrical signals from the cerebral cortex [1] . EEG is essential for studying brain activity and is pivotal in brain-computer interface (BCI) applications due to its non-invasive and portable nature [2] . Despite its potential, EEG-based methods face practical challenges due to low signal-to-noise ratio (SNR) [3] , high inter-subject variability, and significant task-dependent variations in EEG signals [4] . Self-supervised learning, as described by Yann LeCun in his AAAI 2020 keynote [5] , has shown advantages in natural language processing (NLP) [6] , computer vision (CV) [7, 8] , and speech analysis [9] . More and more state-of-the-art (SOTA) models are pretrained by self-supervised learning on large datasets and fine-tuned for specific applications, effectively reducing the need for extensive labeled data. Masked autoencoders, a type of self-supervised learning method, have been successful in NLP [6] and CV [8] by recovering masked patches based on context.Recent advances in EEG analysis using self-supervised learning techniques have shown promising results. In Falck et al. [10] , a framework for learning EEG representations through contrastive learning was proposed. This framework extends the SimCLR framework to time series data, training a channel feature extractor. The model achieved an accuracy of 85.12% on the Sleep-EDF dataset [11] but was only trained on EEG data collected during 20-second tasks, making it unsuitable for shorter tasks such as motor imagery. BENDR [12] applied self-supervised learning based on masked autoencoders and contrastive learning to EEG data. This approach addresses the challenges of multi-task and multi-paradigm EEG data characterization, enhancing the model's universality. BENDR uses a convolutional encoder to extract features from local time windows, masks some features, and then a transformer decoder predicts the information in the masked parts. EEG2VEC [13] introduced a self-supervised model that learns EEG representations based on contrastive loss and reconstruction loss. The pretrained model is used as a feature extractor for downstream tasks. Both EEG2VEC and BENDR utilize convolutional neural network and transformer networks to learn local and global features. EEG2VEC was validated in EEG match-mismatch and EEG regression tasks of the auditory EEG challenge [14] . The Biosignal Transformer (BIOT) [15] model addresses the challenges of cross-data learning with mismatched channels, variable lengths, and missing values in biosignals such as EEG, ECG, and human activity sensory signals. BIOT tokenizes each channel separately into fixed-length segments containing local signal features and then re-arranges the segments to form a long "sentence". In the CHB-MIT seizure detection task [16] , the pretrained BIOT models achieved a 4% improvement. The Large Brain Model (LaBraM) [17] addresses EEG-based deep learning model limitations by enabling cross-dataset learning. It segments EEG signals into channel patches and uses vector-quantized neural spectrum prediction for training a neural tokenizer. This tokenizer encodes raw EEG patches into neural codes, which pretrain transformers to predict the original neural codes for masked patches. LaBraM outperformed SOTA methods in abnormal detection, event type classification [18] , emotion recognition [19] , and gait prediction [20] .Universal models have clearly made progress in EEG data analysis. However, the extremely low SNR of EEG signals and the complexity of brain activities during tasks make it challenging to learn abstract features using masked autoencoders, which are commonly used in NLP and CV [12] . Additionally, the inconsistent sampling rates of different EEG acquisition devices and variations in electrode channel locations [15] hinder the convolutional encoder's ability to decouple the correlation between electrode channels and EEG signals, resulting in robustness and scalability issues.To address these issues, we propose a dual self-supervised EEG universal representation method based on the spatio-temporal consistency of EEG signals [21] , introducing the EEG Pretrained Transformer (EEGPT) for efficient feature extraction. Our method includes spatio-temporal representation alignment and mask-based reconstruction, enhancing representation quality and model robustness. Our method adopts a BERT-style masked recovery task [22] (not autoregressive task) objectives for pretraining. Beyond the original waveform recovery, we align the predicted EEG signal features of the masked parts with full EEG signal features. Additionally, a local spatio-temporal embedding method improves compatibility across different EEG acquisition devices.EEGPT, with over 10 million parameters, is pretrained on a mixed multi-task EEG dataset, including data from PhysioMI [23] , HGD [24] , and M3CV [25] . This pretraining enables EEGPT to extract universal representations for various tasks. For downstream tasks, we employ a linear-probing method, which achieves SOTA performance while also reducing computational resource consumption and preventing overfitting. Our experiments demonstrate EEGPT's superior performance in motor imagery classification [26] , event-related potential (ERP) detection [27] , and sleep stage detection [28] , showcasing its capability to extract high-level abstract features across spatio-temporal dimensions.• Proposal of EEGPT, a 10-million-parameter model for EEG universal feature extraction, leveraging a mixed dataset to enhance performances across tasks and subjects. • Development of a dual self-supervised method for EEG signals, combining spatio-temporal representation alignment and mask-based reconstruction, improving feature quality and convergence. • Design of a hierarchical structure for decoupled processing of spatial and temporal information, reducing computational complexity and enhancing model flexibility for BCI applications. • Implementation of a local spatio-temporal embedding method, increasing robustness and compatibility across different EEG acquisition devices. • Conduct of comprehensive experiments on downstream datasets, demonstrating EEGPT significantly outperforms existing models across multiple EEG tasks and that larger models exhibit improved performance.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
method: 
 Background: According to Kong and Zhang [29] , a masked autoencoder learns features through a form of denoising autoencoder: input signals occluded with random patch masks are fed into the encoder, and the decoder predicts the original embeddings of the masked patches:EQUATIONwhere "⊙" denotes element-wise product; M is the patch mask; f θ (•) and d ϕ (•) are the encoder and decoder, respectively; z is the learned representation; and H(•, •) is the similarity measurement. By minimizing the loss function, the model learns the optimal representation z of the input signal. However, in practice, there is no explicit representation z (no split of encoder and decoder) in the BERT-style model [22] , and the model must be fine-tuned to locate effective representations. In contrast, we add a spatio-temporal representation alignment branch to explicitly represent z, which changes Equation 1 to Equation 2 (dual self-supervised method):EQUATIONThis method encourages the encoded representations to take on a larger extent of semantics, similar to the minimal sufficient representation in the Multi-View Entropy Bottleneck (MVEB) approach [30] , thereby improving the encoding quality and generalization [31] .The structure of the EEGPT model is shown in Figure 1 , which includes operations such as patching, embedding, masking, encoder, predictor and reconstructor. First, the model chunks the input EEG signal x ∈ R M ×T (M channels and T time points) into patches p i,j , and embeds each patch as a token token i,j by local spatio-temporal embedding (Section 2.3), followed by splitting into masked parts M and unmasked parts M, respectively. Then, we pretain the model using a dual self-supervised learning method, including spatio-temporal representation alignment (Section 2.1) and mask-based reconstruction (Section 2.2). Finally, the linear-probing method is used in downstream tasks (Section 2.4).The spatio-temporal representation alignment method aligns the predicted features with the momentum encoder's output, enhancing the encoder's ability to extract robust features and ensuring that the encoder's output contains high-quality global features. We employ the encoder and the predictor to decouple spatial and temporal features, significantly reducing computational complexity.Encoder: The encoder integrates spatial information from masked patches. Equation 3 describes how the encoder (ENC) processes all masked token i,j at time j as input and produces the corresponding output feature enc j : The EEGPT structure involves patching the input EEG signal as p i,j through masking (50% time and 80% channel patches), creating masked part M and unmasked part M and then embedding as token i,j by local spatio-temporal embedding. The encoder processes the masked part, extracting features (enc j ) consisting of {e i } S i=1 for each time segment in the M part with summary tokens {s i } S i=1 . The predictor predicts features (pred j ) for all time segments, aligning with the Momentum Encoder output (menc j ). Based on features extracted by the predictor and encoder, the reconstructor generates rec i,j to reconstruct the EEG signal of the M part.EQUATIONPredictor: As in Equation 4, the predictor (PRED) utilizes features enc j of the masked part from the encoder, combined with temporal position information pos j , to predict the complete encoded feature. Embedding and unembedding [32] are performed linearly on the input and output. We adopt the rotary position embedding method [33] to generate pos j , introducing relative positional and temporal information. To generate prediction features belonging to M, a learnable vector query is used as the query token. Through self-supervised training, the encoder is encouraged to extract more information about the correlation among tokens:{pred t } t∈{1,2,...,N } = PRED {enc j + pos j } ∃i,(i,j)∈M (4)Momentum Encoder: The structure of the momentum encoder is identical to that of the encoder. Equation 5 outlines how the momentum encoder (MENC) processes all token i,j at time j as input and produces the corresponding output menc j . After each training iteration, the parameters of the encoder are accumulated into the momentum encoder with a factor of τ = 0.01.menc j = MENC {token i,j } (i,j)∈M∪M (5)We employ an alignment loss based on Mean Square Error (MSE) [34] to achieve spatio-temporal representation alignment:EQUATIONIn Equation 6, LN denotes layer normalization [35] , which helps to mitigate the effects of extreme values and covariate shift, allowing the model to focus on more important features.The mask-based reconstruction method aligns the reconstructed patches generated by the reconstructor with the raw patches p i,j in the M part.Reconstructor: As shown in Equation 7, the reconstructor (REC) utilizes features enc j from the M part encoded by the encoder and features pred j of the M part predicted by the predictor, along with temporal position pos j , to generate the reconstructed patch rec u,t . We establish a "skip connection" between the encoder and the reconstructor to help maintain the features and accelerate convergence.EQUATIONMask-based reconstruction is achieved using a reconstruction loss based on the Mean Square Error (MSE):L R = - 1 |M| (i,j)∈M ||rec i,j , LN(p i,j )|| 2 2 (8)The complete pretraining loss L is constructed by summing both L A and L R : The local spatio-temporal embedding method first patches and embeds the EEG signal in the spatiotemporal dimension before feeding it into the encoder, as shown in Figure 2 . We denote the set of EEG signal channels as {c i } M i=1 , where c i is the name of each channel. Firstly, the EEG signal is divided into equally sized patches in the spatio-temporal dimensions, denoted as p i,j , i ∈ {1, 2, ..., M }, j ∈ {1, 2, ..., N }: p i,j = x i,(j-1)d:jd (10) where d represents patch's time length, and N = T /d is the number of time patches. Next, the patches are linearly embedded, combining the channel embedding information. We construct a Codex book [36] L = L A + L R (9){ς i ∈ R de } M i=1(d e is the embedding dimension) containing all learnable channel embedding vectors and a mapping from channel names to channel embedding vectors ℜ :{c i } M i=1 → {ς i } M i=1 .This mapping flexibly corresponds the channels of the EEG data to the channels of the model inputs, allowing the model to adapt to multiple datasets and improve channel adaptation. The patch's content is linearly embedded as Embed(p i,j ) = W T p p i,j + b p , where W p ∈ R d×de and b p ∈ R de are learnable parameters. Embedded token denotes as token i,j ∈ R de :EQUATIONBased on self-supervised learning task, the extracted features of patches are mutually predictable and can ignore noise signals at smaller scales. The method aims to extract macroscopic features that span larger scales, which are believed to be more easily recognizable and considered meaningful features. In the downstream tasks, we apply the pretrained encoder and concatenate additional modules to solve classification tasks. As shown in Figure 3 , we introduce the linear-probing method, which freezes the parameters in the pretrained model and only changes the parameters in the additional linear modules. These modules include adaptive spatial filters (1 × 1 convolution) for aligning channels between EEG and the model, and a linear layer to map the features to logits. The encoder passes the output tokens corresponding to summary tokens to the linear classification head. This approach helps avoid the overfitting problem when a large parameter model is fine-tuned using a limited number of samples. Since the additional module is straightforward, the performance is solely determined by the encoder, allowing us to assess the model's capability. We curated EEG public datasets of various paradigms for model pretraining as shown in Table 1 . It contains motor imagery (MI) and execution (ME) datasets PhysioMI [23] , HGD [24] , steady-state visual evoked potential (SSVEP) dataset TSU [37] , emotional classification dataset SEED [38] and multi-subject multi-session multi-paradigm dataset M3CV [25] . To assess the practical utility of the learned representations in downstream tasks, we curated a list of datasets as shown in Table 1 .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We point out the limitations of the work and explain why they are important to consider in H.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
datasets and data processing: 
 It contains MI datasets BCIC-2A [39] , BCIC-2B [40] , sleep stage detection dataset Sleep-EDFx [11] , error related negativity (ERN) dataset KaggleERN [41] and event-related potentials dataset PhysioP300 [23] . We also curated a dataset of abnormal EEG signals dataset TUAB and event type classification dataset TUEV from Temple University EEG Corpus [18] . These diverse datasets enable comprehensive evaluation of the proposed EEGPT model across various tasks. Each dataset underwent similar and distinct preprocessing steps, including cropping (4s), re-referencing (average), channels selecting, scaling (mV) and resampling (256Hz). There are 0-38Hz bandpass filtering in MI datasets for downstream tasks. More details refer to Appendix C.Model implementation. The implementation of encoder, predictor and reconstructor in EEGPT adopts the vision transformer (VIT) [42] and sets S learnable summary tokens (similar to [CLS] token) for summarizing information within the same time patches. We used 58 electrodes (M = 58), as shown in pretraining strategy. In pretraining, for each training dataset, we randomly sampled 10% of the samples as the validation set. As shown in Table 6 , we trained 8 variants with different embedding dims, layers of transformer models (encoder, predictor, reconstructor) and summary tokens S. The AdamW optimizer was employed with the OneCycle learning rate strategy [43] (initial learning rate of 2.5e-4, maximum of 5e-4, minimum of 3.13e-5). The training was conducted for 200 epochs, with a batch size of 64 and 16-bit mixed precision training on 8 Nvidia 3090 GPUs.Evaluation strategy. For the data splitting of TUAB and TUEV, we strictly follow the same strategy as BIOT [15] to compare all methods fairly. In other downstream tasks, we use the same experimental configuration as BENDR [12] , in which Leave-One-Subject-Out (LOSO) validation method are used. Specially, KaggleERN uses 4-fold cross-validation with 10 subjects for testing, Sleep-EDFx uses 10-fold cross-validation and ratio 6:2:2 in training, validation and test splitting. We use linearprobing method for downstream tasks. Particularly, in the sleep stage detection task, we use a 4-layer transformer encoder model as a classifier that integrates the output of our model for every 0.25s for the purpose of processing a long task of 30s. We used the optimal large model in Section 3.5 for testing on all downstream tasks. To ensure the reliability of the experiments, we repeated each experiment three times and calculated the standard deviation.Baselines & Metrics. For the TUAB and TUEV datasets, we use the same baselines from BIOT which are fully fine-tuned models. In other tasks, we use the pretrained BENDR [12] , BIOT [15] and LaBraM [17] as the baselines. The following metrics are used for comparison: 1) Balanced Accuracy (BAC), 2) AUROC, 3) Weighted F1, 4) Cohen's Kappa. We use AUROC only for binary classification tasks and Weighted F1 only for multi-class classification tasks. More details refer to Appendix D.Table 2 : The results of different methods on TUAB.Model Size Balanced Accuracy AUROC SPaRCNet [44] 0.79M 0.7896±0.0018 0.8676±0.0012 ContraWR [45] 1.6M 0.7746±0.0041 0.8456±0.0074 CNN-T [46] 3.2M 0.7777±0.0022 0.8461±0.0013 FFCL [47] 2.4M 0.7848±0.0038 0.8569±0.0051 ST-T [48] 3.5M 0.7966±0.0023 0.8707±0.0019 BIOT [15] 3.2M 0.7959±0.0057 0.8815±0.0043 Ours-Tiny 4.7M 0.7959±0.0021 0.8716±0.0041 Ours 25M 0.7983±0.0030 0.8718±0.0050We conducted a comparative analysis with other large models on the Temple University datasets, using the same configuration as BIOT [15] for experiments on the TUAB and TUEV datasets. The results are presented in Tables 2 and 3 . In the TUAB dataset, the performance of EEGPT is comparable to that of the BIOT model. In the TUEV dataset, EEGPT improves the balanced accuracy by 9.5%, and the weighted F1 score by 6.9% compared to BIOT.To further validate the effectiveness of our model, we conducted comparative experiments with BENDR, BIOT, and LaBraM. The results are shown in Table 4 . On the BCIC-2A and BCIC-2B datasets for motor imagery tasks and the Sleep-EDFx dataset for sleep stage detection, our Cohen's Kappa SPaRCNet [44] 0.79M 0.4161±0.0262 0.7024±0.0104 0.4233±0.0181 ContraWR [45] 1.6M 0.4384±0.0349 0.6893±0.0136 0.3912±0.0237 CNN-T [46] 3.2M 0.4087±0.0161 0.6854±0.0293 0.3815±0.0134 FFCL [47] 2.4M 0.3979±0.0104 0.6783±0.0120 0.3732±0.0188 ST-T [48] 3.5M 0.3984±0.0228 0.6823±0.0190 0.3765±0.0306 BIOT [15] 3 Considering that BENDR used full model fine-tuning while our model only fine-tuned an additional linear layer, this suggests that our model extracts richer and more universal features. We used the linear-probing method for BIOT and LaBraM. EEGPT improved by 2.3%, 3.6%, and 1.4%, respectively, compared to LaBraM, and by 12.5%, and 8.1%, and 2.9%, respectively, compared to BIOT. For the ERP-type task datasets, our model outperforms BENDR by 2.6% and 3.9% on the KaggleERN and PhysioP300 datasets, respectively. The performance of our model on the PhysioP300 dataset is comparable to that of LaBraM, but higher by 3.0% on KaggleERN. Our model also outperforms BIOT on the KaggleERN and PhysioP300 by 7.2% and 10.2%, respectively. On all tasks, our model EEGPT achieves competitive results compared to BENDR, BIOT, and LaBraM. This demonstrates that EEGPT learns consistent representational features over the temporal-spatial dimensions, enabling the model to be more widely applied to multiple paradigm tasks and to achieve better classification performance.Combining the above experimental results, we demonstrate that the method proposed in this paper addresses the issues of poor EEG channel adaptability, poor quality of EEG representations extracted by existing self-supervised learning methods, and the lack of universality of representations across multiple paradigms. Our method effectively extracts high-quality universal EEG representations.We conducted ablation experiments using the large model with four different configurations, and the results are presented in Table 5 . In the absence of alignment loss (L A ), the reconstruction loss [35] , resulting in a 3%,1% and 7% reduction in downstream tasks performance. The version C model, which removes the skip connection and uses all {pred t } N t=1 from the predictor as inputs to the reconstructor, exhibited lower alignment loss and comparable reconstruction loss compared to the version D model, but showed a 1%∼3% lower performance in the downstream task. These results suggest that the dual self-supervised method proposed in this paper is effective, as the spatio-temporal alignment improves the quality of the EEG representations extracted by the model. We designed 8 variants to investigate the effects of model size (embedding size d e / layers) and summary token (S) on pretraining loss and downstream task accuracy. To evaluate the performance of the pretrained models, we used the linear-probing method to test cross-subject classification tasks on the BCIC-2A dataset. The experimental results are presented in Table 6 . As the model size (embedding size d e / layers) and summary tokens (S) increased, the reconstruction loss (L R ) gradually decreased, and the performance on the downstream task improved. For models with the same S, larger models exhibited lower alignment loss L A and higher performance. We also investigated the trends in accuracy and reconstruction loss as the model size increased, as shown in Figure 5 . These trends can be summarized by the scaling laws: ACC = (33.6 * N ) 0.029 and L R = (0.72 * N ) -0.014 , where N is the parameter count of the model. The results of the downstream task experiments indicate that larger models generally achieve higher accuracy, with the large model, featuring an 8-layer, 512-embedding dimension, and 4 summary tokens, exhibiting the highest accuracy. For simplicity, we used the optimal large model for testing on all downstream tasks.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We release all code and data preprocess scripts in the supplemental material and figshareB. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusion: 
 In Compared with BIOT on the TUEV dataset, we achieved a 9.5% performance improvement. These tasks, which have different channel configurations and sampling rates, suggest that EEGPT is scalable.In the future, we plan to further enrich the pretraining EEG dataset and expand the model size and applicability. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite the original paper of existing assets. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: This section provides additional experimental results to support the claims in the main paper.A. We conducted a pretraining experiment after removing the predictor, the variation of L R loss with the number of iteration steps during training is shown in Figure 6 . In this figure, 'baseline' is the standard EEGPT model and 'no_predictor' is the model without the predictor. We can see that the reconstruction loss L R of the 'no_predictor' model after removing the predictor does not decrease, which indicates that directly aligning the outputs of the encoder and the momentum encoder does lead to the problem of representation collapse, resulting in the reconstruction task not being able to learn meaningful representation.A.2 Ablation study for pretraining methods In the ablation experiments, we used the BCIC-2A, BCIC-2B and KaggleERN datasets to test the model with the linear probing method, the results are shown in Table 7 . We can conclude that (1) the model performance degradation on all datasets without L A loss is significant (6% to 9%); (2) without layer normalisation on the reconstruction target, the performance of the version B model degraded by 3%, 1%, and 7% on BCIC-2A, BCIC-2B, and KaggleERN, respectively; and (3) the performance of version C model (removed the skip connection) is reduced by 1%, 3% and 3% on these datasets, respectively.We have added experiments comparing linear probing method and full fine-tuning method, as well as experiments comparing with and without adaptive spatial filter. The experimental results are displayed in Table 8 . In Table 8 , 'ASF' stands for with an adaptive spatial filter, in contrast to feeding the signal directly into the model; 'L-P' stands for using linear probing, in contrast to using full fine-tuning of the model. Model variants A and C are the models with full fine-tuning and linear probing after excluding the adaptive spatial filter, respectively. Model variants B and D are models We added pretraining experiments using 100%, 50%, 25%, and 12.5% of the training data and tested them on the downstream tasks of BCIC-2A and BCIC-2B. The results are presented in Figure 9 LaBraM employs a larger pretraining dataset than ours, which also contains the TUEG dataset that is similar to the TUAB and TUEV distributions, and whose paper illustrates the scale effect of the amount of pretraining data, which may have facilitated the learning of a richer and more downstream task-adaptable representation for LaBraM.A.7 Ablation study of self-supervised learning on TUAB We tested the model with randomly initialized parameters (no pretrained model) on the TUAB dataset and the test results are shown in Table 11 . In Table 11 , 'Ours (no pretrained)' represents the model that is not loaded with pretrained parameters. By comparison, we see that the model without using self-supervised pretraining has the worst performance on TUAB, with a 4% reduction in balance accuracy and a 5% reduction in AUROC compared to the model loaded with pretraining parameters.The code for this paper is available at: https://github.com/BINE022/EEGPT. The PhysioNetMI dataset2 consists of over 1500 one-minute and two-minute EEG recordings obtained from 109 volunteers. Subjects performed different motor execution/imagery tasks while recording 64-channel EEG using the BCI2000 system.Preprocessing: All eight tasks were used during the pretraining of this paper, using a global average reference, intercepting data from 0 to 6s after the start of each trial, and randomly intercepting data from a 4s time window of these during pretraining.The HGD dataset, a 128-electrode dataset, was taken from 14 healthy subjects with approximately 1,000 trials each. The four-second executive movement trials were divided into 13 runs. The four types of movements were left-handed movements, right-handed movements, bipedal movements, and rest. The training set consisted of approximately 880 trials for all but the last two runs, and the test set consisted of approximately 160 trials for the last two runs. We access this dataset by MOABB 3 .Preprocessing: All four tasks were used in the pretraining process of this paper, first downsampled to 256 Hz and then standardized to mV, using a global average reference to intercept the data from 0 to 10s after the start of each trial, and randomly intercepting the data from a 4-s time window of these during pretraining.The TSU dataset4 is a benchmark dataset for brain-computer interfaces based on steady-state visual evoked potentials (TSUBenckmark). This dataset collects SSVEP-BCI recordings from 35 healthy subjects using a brain-computer interface (BCI) speller with 40 characters for experimental data acquisition. SSVEP-BCI recordings were made for 40 characters that flashed at different frequencies (8-15.8 Hz with 0.2 Hz intervals).Stimuli: Each trial began with a visual cue (red square) indicating the target stimulus. The cue appeared on the screen for 0.5 seconds. Subjects were asked to shift their eyes to the target as soon as possible within the cue duration. After the cue was offset, all stimuli began flashing on the screen simultaneously for 5 seconds. After the stimulus offset, the screen was blanked for 0.5 seconds, and then the next trial began, which allowed subjects to have a short rest period between successive trials.Signal: EEG of 35 subjects (64 channels, 250 Hz). Each subject's experiment consisted of 6 blocks. Each block contained 40 trials corresponding to all 40 characters displayed in randomized order. Total: 35 persons x 6 blocks x 40 trials.Preprocessing: All 40 tasks were used in the pretraining process of this paper, first downsampled to 256 Hz, and then standardized to mV, using a globally averaged reference to intercept the data from 0 to 4s after the start of each trial for pretraining.The SEED dataset, Shanghai Jiao Tong University Emotion EEG dataset (SEED) 5 , is an EEG dataset provided by the BCMI lab led by Prof. Baoliang Lu.Stimuli: 15 four-minute long movie clips from six Chinese movies.Signal: EEG (62 channels, 200 Hz) from 15 subjects and eye movement data from 12 subjects. Three experiments were conducted per subject, each approximately one week apart, for a total of 15 subjects x 3 sessions = 45 subjects.Scores: Positive (1), negative (-1), and neutral (0).Preprocessing: All three tasks were used in the pretraining process of this paper, first downsampled to 256 Hz and then standardized to mV, using a global average reference to intercept the data from 0 to 10s after the start of each trial for pretraining.The M3CV dataset6 is a reliable brainprint identification system designed to withstand changes in the mental state of the subjects (cross-paradigm test) and successfully identify individuals even after several days (cross-session test). The Multi-Session Multi-Paradigm EEG database (Multi-Subject Multi-Session Multi-Paradigm Commonality and Variability (M3CV)) contains 106 healthy subjects, two sessions, and six types of EEG paradigms.Experimental paradigms: Resting state, event-related potentials, evoked stimuli, P300, motor execution, and SSVEP.Preprocessing: The full task-state data were used in the pretraining process of this paper, first upsampled to 256 Hz, and then standardized to mV, using a globally averaged reference, intercepting data from 0 to 4s after the start of each trial for pretraining.C.2.1 Motor Imagery Task: BCIC-2A [39] This dataset7 consists of EEG data from 10 subjects and includes four different motor imagery tasks: motor imagery of the left hand (Class 1), right hand (Class 2), feet (Class 3), and tongue (Class 4).Each subject performed two sessions on different days, with a total of 288 trials per session.Preprocessing: Uniform units; 0 to 38 Hz filtering was used; resampling to 256 Hz; EA normalization [49] was used for each session. EA normalization is a common normalization method applied to data from motion imagery tasks such as BCIC2A.Experimental Configuration: Experiments using the LOSO method. In the BCIC2A dataset 4category cross-subject experiment, the accuracy of the model in this paper is lower than that of the task-specific SOTA model. This may be related to the high individual variability of the BCIC2A data.The fine-tuning method of combining the task-specific model with the model in this paper was used for testing. The MSConvNet model, which has a multi-scale convolutional EEGNet-like structure, was used as the motor imagery task-specific model. The model branch first maps the input 22-channel EEG signal into 10-20 system standard channel data using an adaptive spatial filter (convolution); then a pretrained encoder is used to extract the generic representations; and after collocating the generic representations and the in-domain representations of Wenchao's model branch, they are mapped into category logits using a linear layer. Training is performed using the AdamW optimizer, OneCycle learning rate strategy [43] (starting learning rate 5e-4, maximum 1e-3, minimum 4.22e-7), 100 rounds of training, and batch size of 72.This dataset8 consists of EEG data from 10 subjects. Each subject was given five training opportunities, the first two of which were training data without feedback (screening) and the last three with feedback.The three bipolar recordings (C3, Cz, and C4) were sampled at a frequency of 250 Hz. The dataset consisted of motor imagery (MI) for two categories: left-handed (category 1) and right-handed (category 2). Each subject participated in two sessions without feedback recordings on two different days within a two-week period. Each session consisted of 120 category-balanced trials.Preprocessing: Uniform units; 0 to 38 Hz filtering was used; resampling to 256 Hz; EA normalization [49] was used for each session. EA normalization is a common normalization method applied to data from motion imagery tasks such as BCIC2A.Experimental Configuration: Experiment using LOSO validation method. The fine-tuned modeling approach was used to test a cross-subject 2-categorization task on the BCIC2B dataset for the motor imagery task. The input data are 3-channel (C3, Cz, and C4 channels) EEG data with 4s, 256Hz sampling rate; to fully utilize the feature extraction capability of the model, spatial filters (convolution) were used to map the 3-channels of the input data into 7-channels, which corresponds to the C5, C3, C1, CZ, C2, C4, and C6 channels of the pretrained encoder; the features are extracted using the pretrained encoder and then mapped into 7 channels using the linear layer mapped to category logits.The training process is performed using AdamW optimizer, OneCycle learning rate strategy [43] (starting learning rate 1.6e-5, maximum 4e-4, minimum 1.51e-7), 100 rounds of training, and batch size of 64.This dataset 9 gives further insight into the generalizability of the model, as BCI data are typically categorized in the context of a specific trial or event, whereas SSC is a much more continuous problem that requires labeling the specific sleep stage that a subject is in over a long period of time. The Sleep-EDFx dataset contains 197 (78 healthy subjects) all-night sleep recordings, which include EEG, electrooculogram, chin EMG, and event markers.Preprocessing: The preprocessing method of Banville et al. (2020) was referred to, which first converts the data unit to mV, and then uses a 30Hz low-pass filter, intercepts the samples in 30s non-overlapping windows, and uses a channel-wise (channel-independent) z-score normalization for each sample.Experimental Configuration: Subjects were randomly divided according to the ratio of 60 C.2.4 ERN task: KaggleERN [41] In this dataset 10 , each subject is presented with letters and numbers (showing 36 possible items on a matrix) to spell words. Each item of the word is flashed in a random order grouped together and selected one at a time. The selected item is the one that the online algorithm is most likely to recognize as a typical target response. The goal of this challenge was to determine whether the selected items were correct by analyzing the brain signals of subjects after receiving feedback. Two transcription-spelling conditions were used: a fast mode with more errors (4 blinks per item); and a slower, less error-prone condition (8 blinks per item).Preprocessing: Uniform units; downsampling to 256 Hz; global averaging reference; intercepting 2 s of data starting at -0.7 s before the onset of flicker.Experimental Configuration: For comparison with the BENDR paper, 10 of the subject data were used as the test dataset, and the remaining subject data was used for 4-fold cross-validation method training. KaggleERN uses fine-tuning method 1, with a sliding window of 125ms in steps, to firstly map the 56 channels to the 19-channel EEG data of the 10-20 system by adaptive spatial filters (convolution), and then features are extracted using a pretrained encoder, and finally mapped into classification logits using a linear layer. The training process is performed using the AdamW optimizer, the OneCycle learning rate strategy [43] (starting learning rate of 1.6e-5, maximum of 4e-4, and minimum of 1.51e-7), and 100 rounds of training with a batch size of 64.C.2.5 P300 task: PhysioNetP300 [23] In this dataset 11 , each participant was asked to spell a total of 20 characters using a traditional matrix speller (Donchin speller). The target characters were randomly selected before the start of the run. Each row and column of a standard 6x6 character matrix was randomly augmented for 100 ms at 50 ms intervals with approximately 20 flashes. During this time, subjects were asked to focus their attention on the target character and mentally count the number of times the target character was highlighted.Preprocessing: Uniform units; 120Hz low-pass filtering was used; downsampling to 256Hz; and 2s of data starting at -0.7s before the onset of the flicker was intercepted.Experimental Configuration: For comparison with the BENDR paper, subjects 8, 10, and 12 were removed and the data from the remaining 9 subjects were retained. Experiments were performed using the LOSO cross-validation method. The PhysioNetP300 uses a fine-tuning method with a sliding window of 125ms step size, using all channels used in the pretraining phase; the data from each EEG channel is first adaptively scaled by an adaptive spatial filter (learnable scaling factor), then features are extracted using the pretraining encoder, and finally mapped using the linear layer mapping for classification logits. The training process uses AdamW optimizer, OneCycle learning rate strategy [43] (starting learning rate 3.2e-5, maximum 8e-4, minimum 3.02e-7), 100 rounds of training, and batch size of 64.TUAB12 is a corpus of EEGs which are 23-channel and sampled at 256 Hz. All data have been annotated as normal or abnormal. There are total 409,455 10-second samples that we use for binary classification to predict normal/abnormal.TUEV 13 is a subset of TUEG that contains annotations of EEG segments as one of six classes: (1) spike and sharp wave (SPSW), (2) generalized periodic epileptiform discharges (GPED), (3) periodic lateralized epileptiform discharges (PLED), (4) eye movement (EYEM), ( 5) artifact (ARTF) andFor model implementation, the BENDR code is downloaded and modified upon the github 14 , the BENDR code is downloaded and modified upon this github 15 , and the LaBraM code is downloaded and modified upon this github 16 . Other baselines implementations on the TUAB and TUEV datasets are also referred to this repo 17 . The EEGPT structure implementation is mainly modified upon the github 18 , and also references the code of BENDR. The ROPE embedding is implemented using the code of Roformer [50] in github 19 .We mainly use MNE package 20 , braindecode package 21 , and torcheeg package 22 to load and preprocess the data.In the comparative experiments, the baseline model BIOT demonstrated an accuracy of 54% in the binary classification task of EEG P300 signals on PhysioP300 dataset4, suggesting its ineffectiveness.The P300 signal is an event-related potential (ERP) component commonly observed in EEG recordings. It is characterized by a positive deflection in voltage occurring approximately 300 milliseconds after the presentation of a stimulus. The P300 component is primarily concentrated around the 300 ms mark and is crucial for tasks involving attention and stimulus evaluation.One plausible reason for the suboptimal performance of the BIOT model is the inherent nature of the P300 signal classification task, which predominantly relies on time-domain waveform features. The BIOT model, however, applies a fast Fourier transform (FFT) to the input signals, thereby primarily extracting frequency-domain features. This fundamental discrepancy between the feature domain utilized by the model and the domain that is most pertinent to the classification task likely accounts for the observed lack of efficacy. Additionally, while the P300 component is concentrated around the 300 ms mark, the BIOT model applies an FFT on 1-second length patches, which is overly extensive and results in significant information loss. The "significant information loss" is mainly reflected in the fact that the BIOT model only retains the spectral energy information for 1s of each patch after the FFT, and discards the phase information. A similar situation also occurs in the binary classification task for ERN signals [27] on the KaggleERN dataset.To further demonstrate the effectiveness of the model implementation, this paper visualizes the trained model. During the pretraining process, channel information is encoded through learnable channel embeddings. This section visualizes the similarity between positional encodings to show the relationship information learned by the model from the data. Figure 12 shows the cosine similarity relationships between channel embeddings, with electrode positions placed according to their actual locations. Solid lines indicate relationships with similarity greater than 0.3, while dashed lines indicate relationships with similarity between 0.1 and 0.3. It can be seen that channels in close proximity have higher similarity, and there is also significant similarity between distant electrodes on opposite sides. Figure 13 shows the correlation between channels and motor imagery classes detected using the channel perturbation method after training on the BCIC2A dataset. Gaussian multiplicative random noise is randomly added to the signal amplitude of each channel, and the Pearson correlation between the noise intensity and the changes in the corresponding class logits is calculated and presented as a heatmap. Symmetric relationships are observed for electrodes related to left and right hand movements, with bilateral electrodes corresponding to foot movements, and distinct channels corresponding to the four different classes. Figure 15 shows the confusion matrix, indicating that the recognition performance for foot movements is the best, while samples from other classes are often misclassified as foot movements. The confusion matrix shows that the performance for the non-target class is better.Firstly, although we have collected a large-scale multi-task mixed EEG dataset and utilized a model structure with extensive parameters during the pretraining phase, there remains a significant disparity compared to today's large vision and language models. Our work is still in the exploratory phase of training large EEG models to learn general representations. Encouragingly, our experiments have shown that large EEG models can continue to be optimized, achieving considerable performance improvements compared to existing methods developed for specific BCI tasks and general large EEG models. Secondly, while EEGPT requires only linear-probing to adapt to small-scale downstream tasks, it may still result in high memory costs. Finally, EEGPT was pretrained using 4s EEG data. This may strict model's capability. It is worth exploring the training of large EEG models with long recordings.NeurIPS Paper Checklisthttps://physionet.org/content/eegmmidb/1.0.0/https://neurotechx.github.io/moabb/dataset_summary.htmlhttp://bci.med.tsinghua.edu.cn/download.htmlhttps://bcmi.sjtu.edu.cn/home/seed/https://aistudio.baidu.com/competition/detail/315/0/related-materialhttps://www.bbci.de/competition/iv/#datasetshttps://www.bbci.de/competition/iv/#datasetshttps://physionet.org/content/sleep-edfx/1.0.0/https://www.kaggle.com/c/inria-bci-challenge/datahttps://physionet.org/content/erpbci/1.0.0/https://isip.piconepress.com/projects/tuh_eeg/html/downloads.shtmlhttps://isip.piconepress.com/projects/tuh_eeg/html/downloads.shtmlhttps://github.com/SPOClab-ca/BENDRhttps://github.com/ycq091044/BIOThttps://github.com/935963004/LaBraMhttps://github.com/ycq091044/BIOThttps://github.com/google-research/vision_transformerhttps://github.com/ZhuiyiTechnology/roformerhttps://github.com/mne-tools/mne-pythonhttps://github.com/braindecode/braindecodehttps://github.com/torcheeg/torcheeg 


Decision: This paper develops a 10M pre-trained transformer model	for EEG
processing.  The transformer is trained with masked self-supervised
training with spatio-temporal representation alignment.  Results on
several down-stream tasks are improved relative to comparison
approaches.  All reviewers scored this with a 5 or 6 recognizing that
it is worth presenting to the NeurIPS conference.

Reviews: review: paper_summary: paper_summary: The paper presents a new pretrained model called EEGPT (EEG pretrained transformer), designed to improve the analysis of EEG (electroencephalography) signals. EEGPT is a model with over 10 million parameters (up to 100M for the largest model) that aims to solve common problems in EEG analysis, where the challenges are low signal quality and high variability between individuals. The model introduces a special method called "dual self-supervised learning" to extract better features from EEG data. This method includes two main techniques: the usual masked reconstruction loss, and additionally the alignment loss. The alignment loss is a special loss introduced inside the masked auto-encoder to “force” the learned features of the masked patches inside the encoder (and predictor) to be aligned with spatial and temporal patches (all masked and unmasked). This is achieved by adding a momentum encoder and computing the loss between the predictor (global integration of summary patches of each time j) and momentum encoder (spatial integration of all unmasked patches for each time j).

The model was trained on several large EEG datasets and tested on various tasks like classifying motor imagery, detecting event-related potentials, and identifying sleep stages. EEGPT performed better than existing models in these tasks, showing it is effective and scalable.
strength_weakness: strength_weakness: 1. The experiments done for the model design and behaviour such as scaling laws, ablation studies, channel embedding relationships, etc, are done only on one downstream task: namely the BCIC2A dataset. How does the model perform on the other tasks, what is the reason for not presenting the results on the other datasets?
2. The paper presents interesting scaling law experiments, showing potential in the method to scale to larger model size. However the experiments are too limited to be able to state the scaling laws as given in the paper (e.g. the ACC = (33.6*N)^0.029). I would argue one has to test it either to more tasks, or have a unified metric that uses many datasets for the scaling law.  Also, how does the law depend on the amount of training data used?
3. The design of the model was at first confusing and I still have trouble understanding the main motivation from some of the decisions. The ablation studies are a great help here, but I think a few more ablation studies or explanations on the design of the model would clarify the paper better (see Questions section).
4. I have a few doubts on the evaluation of the baselines. Although I am not certain what are the best achievable accuracies in many of the tasks, for TUAB we have many different models that might perform better: We have models up to 86% (ChronoNet) in Table 1 in here (https://www.sciencedirect.com/science/article/pii/S2213158223001730) and above 90% in here (https://arxiv.org/pdf/2401.10283). I know there are certain differences: accuracy vs balanced accuracy and the second paper has a different evaluation for the highest accuracy. However, the TUAB dataset is quite balanced to my knowledge and still in the second paper in Table 1, there are multiple models close to 90%. Why are these models not included in the comparison, and since we have this discrepancy in this task, do other models exist also for other tasks as well?
5. Why is the pretraining set very small? In terms of participants, the downstream set is much bigger than the pretraining set. In addition, the TUEG dataset is bigger than all the datasets used here (combined). Is there any reason why the authors chose these datasets in particular? It is not clear from the paper why this is the case. Usually, during pretraining the dataset is much bigger compared to the downstream task. In this paper this does not seem to be the case. In fact, many of these datasets (all?) used during pretraining do have labels (i.e. why self-supervised).
6. In some sections it was not clear which evaluation dataset has been used to produce the results. For example in Section 3.4 there is Table 5 but it is not clear which model size and which dataset has been used to produce the results. Because of the accuracy number 58.46 I could infer that it was BCIC2A, but it is not mentioned anywhere. For scaling laws it is mentioned in the text, but it is better to have it on the table description. 
7. Many minor annoyances, which make me wonder how much care the authors took to write this paper:
-  Misspellings, for example: Accuarcy vs accuracy, Appandix vs Appendix, etc.
- The references are messed up. Just look at the first two references [1] and [2], where every author is listed at least 2 times, sometimes even 3 times!
- You need to put a space in front of a reference, like this [1]. You almost always have no space, but not even this is consistent, because sometimes you do.
questions: questions: Even though there is the ablation study for the alignment loss, it is not clear from the paper what is the main motivation behind it, and why this particular design is used. To be more concrete, we have the summary tokens from the encoder which are only created from patches at specific time point j. By design we have now an encoder that integrates all the patches spatially (for each timepoint). The confusing part for me is the predictor: on the one hand it tries to capture global features between all tokens in the time domain, on the other hand the alignment loss forces the tokens to only capture the local features at specific timepoint. What is the main motivation behind this? Note that from the experiments it is shown that the method works quite well, but it was confusing for me in the beginning to understand how did you come up with this design choice. Stated differently: how does the model perform if we do the alignment loss before the predictor, or if we remove the predictor. 

How did you evaluate the TUAB dataset since your model can only work 4s windows. The TUAB dataset can have many minutes to hours of EEG data. Same goes for TUEV.

The evaluation for TUAB seems to not include in the comparison the best performing models in the literature (see limitations). Are there also other models for the other tasks that perform better and are not included in the paper?
limitations: limitations: The model is only trained with 4s of EEG data and the pretraining set is very small. Although the initial experiments are promising, It is not clear how the model generalises in other datasets and downstream tasks. The model is not evaluated how it performs if we finetune the model (instead of linear probing).
paper_summary: paper_summary: The authors propose a novel pretraining strategy, EEGPT, that is essentially a multi-task self-supervision loss consisting of a masked autoencoder-style reconstruction objective, and an alignment loss that is reminiscent of knowledge distillation approaches such as data2vec. EEGPT is applied to representation learning of electroencephalography (EEG) data, and the authors demonstrate the pretrained model flexibility across a variety of downstream EEG decoding tasks (motor imagery, sleep staging, ERP detection).
strength_weakness: strength_weakness: b. Quality: One weakness of this work is that the proposed model has significantly more parameters than the SOTA approaches it is comparing against. It remains unclear if the authors have developed a better pretraining approach, or if the improvements in the downstream tasks are due to higher expressivity in the base model architecture. Could the authors add additional comparisons where the number of parameters EEGPT matches the number of parameters in the other SOTA approaches?

Another weakness is that the comparison to pretraining approaches such as BIOT is unfair. The authors state in Appendix E: “the BIOT model applies an FFT on 1-second length patches, which is overly extensive and results in significant information loss". However, this does not mean that the pretraining recipe in this paper is better than BIOT. It could be the case that BIOT performs similarly, it just needs to be retrained with different patch length and/or patch stride. I understand that retraining BIOT with new parameters may not feasible in the rebuttal timeframe, but I think this should be clarified in the text.
questions: questions: 1. Line 59: The authors claim that the low SNR of EEG “...make it challenging to learn abstract features using masked autoencoders”. However, there have been multiple works that use MAE-style pretraining to successfully learn features from EEG, such as: [1] Chien, et al. MAEEG: Masked Autoencoder for EEG Representation Learning. arXiv:2211.02625. [2] Wu, et al. Neuro-BERT: Rethinking Masked Autoencoding for Self-supervised Neurological Pretraining. IEEE JBHI. 
2. Eqn 9: Did the authors try using a weighted loss with weight $\lambda$? Or does each individual loss take on similar values? $\mathcal{L} = \mathcal{L}_A + \lambda \mathcal{L}_R$
3. Section 2.4: Can the authors clarify which “Features” are passed to the linear classification head? Are these the summary tokens (passed to the Predictor during pretraining)? Or are they the non-summary tokens?
4. Section 3.1: Were any notch filters used to suppress electrical noise in datasets used for pretraining or downstream tasks?
5. Section 3.2: Why were 58 electrodes used? Is it because these 58 electrodes were common across the pretraining datasets? Also, what happens at inference time if a new electrode not belonging to the original set of 58 locations appears?
6. Section 3.2: Is there any normalization applied to the input signal? I saw in the appendix that each dataset has a different normalization (EA on entire session, z-score on each input, etc.). Does that mean that the normalization is dataset-dependent?
7. Line 197: Missing reference for OneCycle learning rate
8. Tables 2&3: The proposed EEGPT model has ~8X as many parameters as the second largest model (BIOT). It would be a fairer comparison if the authors also reported BAC for a similarly sized EEGPT model, such as tiny3 or little (from Table 6).
9. Table 5: Can more description be added to this table? For example, BAC stands for Balanced Accuracy I assume? And which dataset is BAC being reported for here? Or is it an average across datasets?
10. (Minor comment) The title “EEGPT” is slightly misleading, since it makes the reader think that the model is being pretrained using an autoregressive reconstruction objective such as in the popular GPT language models (e.g. predict the next token given the previous tokens). I would recommend changing to a different name to avoid confusion.
11. (Minor comment) Line 186: Spelling error “Appendix”
12. (Minor comment) Line 499: What does EA in “EA normalization” stand for? I tried to find it elsewhere in the paper, but could not find it..
limitations: limitations: Yes.
paper_summary: paper_summary: The paper presents EEGPT, a 10-million-parameter pretrained transformer model designed for universal EEG feature extraction. The model employs a dual self-supervised learning method for efficient feature extraction and demonstrates state-of-the-art performance on various downstream tasks with linear probing.
strength_weakness: strength_weakness: 1. The function of the adaptive spatial filter is unclear. Why is this module not included during pre-training? An ablation study is needed to evaluate the effectiveness of this component.
2. In Table 2, the authors use different metrics than in Table 3. Cohen's Kappa should be used consistently across both tables. Additionally, the results of LaBraM on these two datasets should be reported to provide a more comprehensive comparison, even if EEGPT, which uses linear probing, might not surpass LaBraM.
3. The technical contribution of the paper appears incremental. The framework is largely based on the context autoencoder (CAE) with modifications such as the adaptive spatial filter, multiple summary tokens, and rotary temporal embeddings.
4. During pre-training, EEGPT uses EEG data of the same configuration. In contrast, LaBraM leverages various datasets with different configurations, which might make EEGPT less universal.
5. The ablation study is conducted on a single dataset, which is not persuasive enough to demonstrate the effectiveness of the proposed modules and their scalability.
6. EEGPT uses different model settings for different downstream datasets, which may limit its generalizability.
questions: questions: 1. It is unclear whether the baselines in Table 3 are based on linear probing or fully fine-tuned models. Clarification is needed.
limitations: limitations: The authors have discussed the limitations.

Comments: comment: I appreciate the comments by the authors. Some of my concerns are addressed by the clarification and additional experiments. However, regarding the technical contribution and universibility (W3 and W4), I maintain my standpoints.
#### **New comment** 
Our dual self-supervised method improves the quality of EEG representations, therefore it performs better on other tasks.

On the TUAB and TUEV tasks, LaBraM may achieve better performance for two reasons: firstly, LaBraM uses 8-second pre-training data segments (while EEGPT uses 4 seconds), and secondly, LaBraM employs TUEG datasets (which are similar to the data distribution of TUAB and TUEV) as pretraining datasets, thus it is able to better capture the long-term features in the EEG data.

We use a 4-second pretraining dataset because all the pre-training data we use are task-state data segments, which are almost 4 seconds in length to ensure data quality.

#### **W2** 
In this paper, our work mainly focuses on showing that, under the same conditions, our proposed dual self-supervised method demonstrates significant performance improvement compared to the MAE self-supervised method. Moreover, our model exhibits comparable or better performance on a wider range of tasks/datasets compared to other SOTA general EEG pretrained models (not pretraining methods), suggesting that our model may have stronger generalizability.

The main improvement of BIOT lies in the Biosignal Tokenization method, which is an improvement in the tokenizer, such as the introduction of normalization and FFT to retain more task-related features; while LaBraM's main improvement is the proposal of the Neural Tokenizer inspired by VQ-VAE, and using the tokens encoded by the Neural Tokenizer as the prediction target, which is an improvement in the prediction target, making the prediction target contain more task-related features (such as spectral features); in contrast, our work focuses on an enhancement to the self-supervised method itself, using a dual self-supervised method to enhance the quality of the representations.

EEGPT, BIOT, and LaBraM may each have their own design advantages in the pretraining method. More work may be needed in the future for a more fair comparison, and they can learn from and integrate with each other.

#### **Q12**
Thank you for your comments. EA stands for 'Euclidean space EEG data alignment' approach, which we have declared in the new revision.
Thank you for thoroughly addressing my comments & questions. See below for further comments

New comment: I see that the authors have added a comparison to LaBraM in A.6 showing that LaBraM outperforms EEGPT significantly on TUEV and TUAB datasets. However, Table 4 demonstrates that EEGPT is better than LaBraM at other tasks (BCIC, SleepEDF, etc.). Do the authors have any explanation for this?

W1: Thanks for running this! Indeed, it seems like when the number of parameters is matched between EEGPT and BIOT the performance is better (TUEV) or similar (TUAB). 

W2: I appreciate that you added the comment about possible reasons for why BIOT might be worse in the Appendix. I still feel the comparison is slightly unfair, and re-running the BIOT pretraining with smaller token sizes should be subject of future work to determine whether EEGPT is truly a better pretraining strategy.

Q12: Can you define the acronym for EA somewhere in the text? I assume it stands for Euclidean Alignment?