Title: Brant: Foundation Model for Intracranial Neural Signal
Venue: NeurIPS 2023
Accepted: yes
Keywords: Foundation model

Brain signal

Pretraining

Medicine
Abstract: We propose a foundation model named Brant for modeling intracranial recordings, which learns powerful representations of intracranial neural signals by pre-training, providing a large-scale, off-the-shelf model for medicine. Brant is the largest model in the field of brain signals and is pre-trained on a large corpus of intracranial data collected by us. The design of Brant is to capture long-term temporal dependency and spatial correlation from neural signals, combining the information in both time and frequency domains. As a foundation model, Brant achieves SOTA performance on various downstream tasks (i.e. neural signal forecasting, frequency-phase forecasting, imputation and seizure detection), showing the generalization ability to a broad range of tasks. The low-resource label analysis and representation visualization further illustrate the effectiveness of our pre-training strategy. In addition, we explore the effect of model size to show that a larger model with a higher capacity can lead to performance improvements on our dataset. The source code and pre-trained weights are available at: https://zju-brainnet.github.io/Brant.github.io/.

Brant: Foundation Model for Intracranial Neural Signal

Abstract

We propose a foundation model named Brant for modeling intracranial recordings, which learns powerful representations of intracranial neural signals by pre-training, providing a large-scale, off-the-shelf model for medicine. Brant is the largest model in the field of brain signals and is pre-trained on a large corpus of intracranial data collected by us. The design of Brant is to capture long-term temporal dependency and spatial correlation from neural signals, combining the information in both time and frequency domains. As a foundation model, Brant achieves SOTA performance on various downstream tasks (i.e. neural signal forecasting, frequency-phase forecasting, imputation and seizure detection), showing the generalization ability to a broad range of tasks. The low-resource label analysis and representation visualization further illustrate the effectiveness of our pre-training strategy. In addition, we explore the effect of model size to show that a larger model with a higher capacity can lead to performance improvements on our dataset. The source code and pre-trained weights are available at: https://zju-brainnet.github. io/Brant.github.io/ .

1 Introduction

Brain signals are electrical impulses that are generated by brain neurons and transmit through neural networks. These signals provide important information about brain activity and can usually be monitored in two ways, namely scalp electroencephalography (EEG) and intracranial electroencephalography (iEEG). The former records the electrical brain activity through electrodes placed on the scalp, while the latter implants intracranial electrodes into brain tissue directly to derive neural recordings. Compared with EEG, iEEG manifests significant advantages by providing more abundant, stereotactic and detailed information about brain wave patterns from deeper brain structures, which has been the mainstream method to obtain deep brain information, and is essential in the therapies for some brain diseases (e.g., Parkinson's disease [1], epileptic seizure [2]).

Modeling intracranial recordings has drawn much research attention, but several issues still remain unresolved. Currently, studies for modeling intracranial recordings are mainly divided into two research lines, namely handcrafted feature based methods [3-8] and deep learning based methods [9-12]. Handcrafted feature engineering requires lots of domain knowledge and may only work on specific tasks. And most deep learning based methods are fully supervised, which relies heavily on

labeled data. However, labeling data at scale in medical experiments is often infeasible or expensive, which underscores the importance of maximizing the label efficiency. To overcome these limitations, the paradigm of self-supervised pre-training followed by fine-tuning with few samples can greatly reduce the reliance on labels and enable the model to generalize to various downstream tasks.

Moreover, modeling intracranial recordings requires careful consideration of several key factors. (1) Long-term dependency. Since intracranial neural signals are time series and gradual changes in brain activity may only be captured by the long-period analysis [13], long-term temporal dependency is crucial in modeling intracranial data. (2) Spatial correlation. The electrodes implanted in the brain contain many contacts (also called channels), which are distributed across various brain regions. Due to the fact that brain waves propagate through different brain regions [14], signals recorded from different channels can be spatially correlated, reflecting the underlying neural activity. (3) Time and frequency domains. For neural recordings, the time domain provides information about the amplitude and duration while the frequency domain can reveal underlying oscillatory patterns and rhythms [15]. Therefore, modeling neural signals in both domains can provide information more consistent with the neurophysiological mechanisms [16]. To the best of our knowledge, no existing work on intracranial recordings considers all the three key factors simultaneously.

In view of the unresolved issues above, we propose a foundation model for intracranial neural signal named Bra in N eural T ransformer (Brant). The design of our model takes all the three key factors (i.e., long-term dependency, spatial correlation, time and frequency domains) for intracranial signal modeling into account. Moreover, Brant contains more than 500M parameters and is pre-trained on a large intracranial dataset with 1.01 TB data, which can be adapted to accomplish various downstream tasks. Compared to other existing methods for modeling brain signals, Brant can achieve better performance with far fewer labeled samples, showing the great benefit of our work in medical scenarios. As an off-the-shelf model along with the code and weights, Brant can participate in other medical research and experiments, which alleviates the issue of sample and label efficiency.

To sum up, the main contributions of our work comprise:

We propose a foundation model for intracranial neural signals named Brant, which is the largest model on brain signals (shown in Fig. 1) and pre-trained on a large intracranial dataset collected by us, providing a large-scale and off-the-shelf model for medicine.

To our knowledge, Brant is the first to date that attends long-term dependency and captures spatial correlation across channels, while combining the information from both time and frequency domains.

Extensive experiments show that Brant generalizes well to various downstream tasks, showing the great potential in neural recordings modeling. Further analysis illustrates the effectiveness of large-scale pre-trained model, demonstrating the medical value of our work.

2 Method

Model overview. As previously mentioned, we propose Brant to capture long-term dependency and spatial correlation from intracranial recordings, while combining the information from both time and frequency domains. Our model mainly consists of two Transformer encoders, namely, temporal encoder and spatial encoder (shown in Fig. 2). The temporal encoder encodes a sequence of L consecutive patches which focus on the temporal dependency, and the spatial encoder encodes C patches with the same time indices from all channels to capture their underlying spatial correlation.

Input Encoding

Temporal Encoder

Spatial Encoder

Linear Projection

Masked patches

Reconstructed patches

Temporal representations

Final representations

Record

Electrode Positions

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

Channels

Pj:j+L-1

Pj:j+L-1

Pj:j+L-1

2j:j+L-1

h

During pre-training, we randomly mask the input signals recorded from the implanted electrodes, then linearly map the original signals to the latent space and add positional and frequency information through input encoding. Then the input encoding is encoded to latent representations and a linear projection head is added on the top of the spatial encoder which maps the representations to reconstruct the original signals. The details of our model and pre-training task are described as below.

Patching. Since neural recordings are electrical signals with high sampling rates, we aggregate timestamps into patches to (1) enhance the locality and extract semantic information; (2) reduce computation and memory usage; and (3) attend a longer temporal dependency [27]. Specifically, given a neural signal x ∈ R N × C , where N is the number of timestamps and C is the number of electrode channels, we divide x with length M and stride S to generate a set of patches p ∈ R N p × C × M , where N p = ⌊ N -S M ⌋ is the number of patches in each channel.

Frequency encoding. We propose frequency encoding to explicitly inject the information on frequency domain to the observed data. The frequency encoding is mainly based on power spectral density (PSD) which describes the distribution of a signal's total average power over frequency (details about PSD are in App. A) . To be specific, we split the frequency domain into several bands according to the standard description for rhythmic activity [28]: (1) θ (4-8Hz), (2) α (8-13Hz), (3) β (13-30Hz), (4) γ 1 (30-50Hz), (5) γ 2 (50-70Hz), (6) γ 3 (70-90Hz), (7) γ 4 (90-110Hz), (8) γ 5 (110-128Hz). For the i -th frequency band, a learnable encoding f i is set as its representation which is shared across all the patches. Then we compute the absolute spectral power of each patch p j,c ∈ p , j = 1 , ..., N p ; c = 1 , ..., C in the i -th frequency band:

which acts as the weight of f i . The frequency encoding F j,c ∈ R D of patch p j,c is obtained as the weighted sum of the learnable encodings f i :

Encoding process. It contains several steps to encode the input signal to latent representations. Specifically, the input p j : j + L -1 ∈ R L × C × M contains L × C patches and p j : j + L -1 ,c ∈ R L × M

denotes the patches in c -th channel. We first map each sequence of patches p j : j + L -1 ,c , c = 1 , ..., C to the latent space of dimension D by a linear projection W proj ∈ R D × M , and the projected input will be added with a learnable positional encoding W pos ∈ R L × D which monitors the temporal order of patches and the frequency encoding:

where ˜ p j : j + L -1 ,c ∈ R L × D denotes the input encoding of the original signals p j : j + L -1 ,c . The input encoding will be fed into the temporal encoder to obtain temporal hidden representations h j : j + L -1 ,c ∈ R L × D and we denote the temporal representations of the whole input as h j : j + L -1 ∈ R L × C × D . The spatial encoder further captures the spatial correlation across channels, which takes each of the temporal representations h k ∈ R C × D , k = j, ..., j + L -1 as the input and outputs the final representations z k ∈ R C × D , k = j, ..., j + L -1 . Overall, given the input signal p j : j + L -1 , the encoding process returns the corresponding latent representations z j : j + L -1 ∈ R L × C × D .

Self-supervised pre-training. Self-supervised representation learning is a powerful approach to extract high level abstract representation from unlabelled data. Among those methods to learn representation via self-supervised pre-training, masked autoencoder (MAE) has been proved to be a simple but effective way in many fields [27, 29, 30]. We apply this technique to our self-supervised pre-training, in which the model is trained to reconstruct the whole input given its partial observation.

Given the input patches p j : j + L -1 , we mask a subset of patches uniformly at random and encode the masked patches to the latent representations z j : j + L -1 . During the pre-training stage, the representations will be fed into a flatten layer with linear head W rec ∈ R M × D to reconstruct the original patches. Finally we utilize an MSE loss to measure the discrepancy between the reconstructed patches ˆ p j : j + L -1 and the original patches p j : j + L -1 .

3 Experimental Setup

3.1 Dataset

Pre-training dataset. Brant is pre-trained on 1.01 TB neural data, a large clinical intracranial neural signal dataset recorded by stereo-electroencephalography (SEEG) technique from a first-class hospital. The subjects undergo a surgical procedure to implant 4 to 11 invasive electrodes, each with 52 to 153 channels, in their brain. The dataset contains 2528 hours of 1000Hz recordings with more than 1 trillion timestamps. More details are in App. E. We down sample the original signals to 250Hz and generate a set of patches of 6s (1500 timestamps).

Downstream dataset. Another neural dataset collected by us with seizure labels is used to fine-tune and evaluate our model. It contains 29.39 GB data with 43 hours of 1000Hz intracranial recordings and we do the same preprocessing (i.e. down sampling and patching) to it as the pre-training dataset. Professional neurosurgeons participate in the labeling of epileptic seizures and the labels consist of two categories, namely, seizure and normal samples. More details are in App. E. For each downstream task, we sample a small subset from the downstream dataset for fine-tuning and evaluation.

3.2 Pre-training

For the model configurations, the temporal encoder contains a 12-layer Transformer encoder with model dimension 2048, inner dimension (FFN) 3072 and 16 attention heads, and the spatial encoder contains a 5-layer Transformer encoder with model dimension 2048, inner dimension 3072 and 16 attention heads. During the pre-training, 40% patches in each input sample are masked with zero values uniformly at random. We take 16 input samples as a minibatch and each minibatch contains an average of 24k patches. The model is pre-trained on a Linux system with 2 CPUs (AMD EPYC 9654 96-Core Processor) and 4 GPUs (NVIDIA Tesla A100 80G) for about 2.8 days.

We optimize with Adam [31], updating the model parameters every 4 steps, and the model trains for 750k updates in total. A cyclic scheduler that adopts a basic triangular cycle without amplitude scaling is utilized to adjust learning rate during pre-training. Specifically, we set the basic learning rate as 3 × 10 -6 and the maximum learning rate as 1 × 10 -5 , then the learning rate steps up (down)

for every 8k updates. We apply mixed precision training with FP32 and BF16 to reduce the memory usage in GPUs for acceleration.

3.3 Downstream tasks

As Brant is a foundation model for intracranial recordings, we conduct extensive experiments on several downstream tasks, including short- and long-term signal forecasting, frequency-phase forecasting, imputation and seizure detection, to verify the high capacity of our model in modeling intracranial recordings. We conduct each downstream task for Brant on two settings: (1) fine-tune the model with a learning rate of 1 × 10 -7 ; (2) freeze the pre-trained weights. Five random runs were performed to obtain the mean and standard deviation. The detailed setups of these downstream tasks are as follows:

Short- and long-term signal forecasting. Predictive observation of the neural signal values is beneficial for the development of warning systems for patients in need of precautionary measures [32, 33]. Therefore, we adopt short- and long-term signal forecasting, in which the learned representations are fine-tuned to predict future signals with different lengths given a past sequence. The past sequence length is set as 15 patches (90s) and the prediction lengths are set as 2 patches (12s) and 20 patches (120s) in short- and long-term forecasting, respectively. We sample 400 minutes of recordings from the downstream dataset, then randomly split into 320 minutes for fine-tuning and 80 minutes for evaluation. A linear prediction head is used to predict the future signals. We adopt MAE and MSE as the performance metrics.

Frequency-phase forecasting. In medicine, it is essential to predict the physical features like frequency and phase of brain signals to provide guidance for some treatments. For example, in a therapy that modulates brain activity called transcranial alternating current stimulation (tACS), the stimulation control heavily depends on these knowledge about the target brain activity [34]. Therefore, we set up the frequency-phase forecasting task in order for these therapies like tACS to be most effectively used in the treatment of brain disorders [35].

Given a past sequence, this task is to predict the dominant frequency and phase information (see details in App. B) of intracranial signals in the future. The past and prediction lengths are set as 15 patches (90s) and 5 patches (30s), respectively. We use the same sampling and split strategy as that in the short- and long-term signal forecasting to generate the data for the frequency-phase forecasting task. A linear layer is adopted to predict the dominant frequency and phase of the future signals. As for the metrics, following the work by Mansouri et al. [34], we use MAE for dominant frequency forecasting, and phase locking value (PLV) for phase forecasting. The PLV is a value between 0 and 1 calculated by the equation:

where ψ and ˆ ψ are the original signals and the forecasted signals, respectively.

Imputation. During brain signal recordings, measurement problems such as artifact contamination or electrode impairment are not easily corrected, thus the neural recordings will be incomplete. Imputation can fill in these contaminated signals so that other medical devices can keep operating under missing values, making use of available data [36]. For the imputation task, we randomly mask the timestamps in each patch with the ratio of 40% and fine-tune the model to predict the missing values. We adopt the same sampling and split strategy to obtain the imputation data as in the shortand long-term signal forecasting. We add a linear head to make predictions, then apply MAE and MSE as the evaluation metrics to measure the discrepancy between the masked and predicted values.

Seizure detection. As one of the most important applications of intracranial recordings, seizure detection task is to evaluate the model ability to distinguish between epileptic seizures and normal waveforms. We sample 250 minutes of recordings from the downstream dataset with about 10% positive (seizure) samples, where 200 minutes are randomly selected for fine-tuning and the remaining 50 minutes for evaluation. An MLP is adopted to classify the pre-trained representations. The evaluation metrics we use are accuracy, precision, recall, F 1 and F 2 scores. The F-measure is a metric defined as the weighted harmonic mean of precision and recall, with the following equation

(1+

β

2

)

×

precision

×

recall

F

β

=

.

3.4 Baselines

As Brant is a pre-training work on neural signals, we extensively compare our model with the advanced self-supervised or unsupervised pre-training works on all the downstream tasks, including works designed for brain signals: RP [17], TS [17], CPC [17], BENDR [18], MVTS [19] and BrainBERT [20]; and for general time series: CoST [37], TF-C [38], PatchTST [27] and TS-TCC [39]. Considering that seizure detection is an important application scenario of intracranial recordings, we further compare our model with several supervised methods which conduct seizure detection on EEG or iEEG to evaluate the performance of Brant in detecting epilepsy. These baselines contain handcrafted feature based methods, including spectral power [28], rhythmicity spectrogram [3] and amplitude-integrated EEG [40]; and the SOTA deep learning based method on seizure detection, including SEEG-Net [9]. More details of the baselines are shown in App. C.

4 Experimental Results

4.1 Main Results

Fig. 3 summarizes the results of all the downstream tasks. As a foundation model for intracranial recordings, Brant achieves consistent SOTA performance on a variety of tasks compared with other baseline models. We discuss more detailed comparisons of each task in the following paragraphs, where in all the tables we mark values ranking the first ( v ), second (v) and third (*v) in each column.

Long-term Signal

Forecasting

(MAE)

Frequency

Forecasting

(MAE)

Phase Forecasting (PLV)

Imputation

(MAE)

Seizure

Detection

(F2)

Short-term Signal

Forecasting (MAE)

0.40

0.60

0.80

0.55

0.65

0.75

0.00

0.08

0.16

0.61

0.57

0.53

0.15

0.35

0.55

60.00

40.00

20.00

0.40

0.60

0.80

0.55

0.65

0.75

0.00

0.08

0.16

0.61

0.57

0.53

0.15

0.35

0.55

60.00

40.00

20.00

0.40

0.60

0.80

0.55

0.65

0.75

0.00

0.08

0.16

0.61

0.57

0.53

0.15

0.35

0.55

60.00

40.00

20.00

Brant compared with

Pre-training baselines

on brain signals

Brant compared with

Pre-training baselines

on time series

Brant (our model)

Brant

Brant-Freeze

Brant (ours)

RP

TS

CPC

BENDR

MVTS

BrainBERT

Brant (ours)

PatchTST

TS-TCC

TF-C

CoST

Long-term Signal

Forecasting

(MAE)

Frequency

Forecasting

(MAE)

Phase Forecasting (PLV)

Imputation

(MAE)

Seizure

Detection

(F2)

Short-term Signal

Forecasting (MAE)

Long-term Signal

Forecasting

(MAE)

Frequency

Forecasting

(MAE)

Phase Forecasting (PLV)

Imputation

(MAE)

Seizure

Detection

(F2)

Short-term Signal

Forecasting (MAE)

Short- and long-term signal forecasting. The results of short- and long-term signal forecasting are shown on the left and middle of Tab. 1, respectively. Not only does Brant achieve SOTA performance,

but the results of Brant-Freeze are also better than most baselines. Especially on long-term forecasting, Brant-Freeze defeats all the baseline methods, showing the ability to capture long-term dependency of our model. Among the baselines, PatchTST [27] performs well compared to other methods, mainly because it also adopts a patching strategy to attend longer temporal dependency.

Frequency-phase forecasting. Forecasting physical features like frequency and phase of neural signal is essential in some medical techniques (see Sec. 3.3). The results of frequency-phase forecasting are shown on the right of Tab. 1, which shows that our model outperforms all the methods, demonstrating that Brant facilitates effective implementation of therapies for some brain disorders.

Imputation. As a mitigation for measurement problems, imputation can fill in the incomplete neural signals, allowing other medical devices to continue operating even when data is missing (see Sec. 3.3). From Tab. 2, Brant and Brant-Freeze achieve the best and the second best results among all the methods, which demonstrates the ability of our model in capturing underlying temporal patterns with partially observed neural recordings to imputate the contaminated neural signals.

Seizure detection. As seizure detection is an important medical application of intracranial recordings, we not only compare our model with the pre-training based baselines, but also further select 4 supervised method designed for seizure detection. From Tab. 3, Brant outperforms all the other methods and Brant-Freeze achieves second best on most of the metrics. BrainBERT [20] achieves the best accuracy, precision and F2 score among all the baseline methods, primarily due to it provides contextualized neural embeddings and combines the information from time and frequency domains like our model. However, Brant still improves the F2 score by 29.59% over BrainBERT, because our temporal encoder obtains a wider receptive field and the spatial encoder captures the spatial correlation across channels, which are both critical in modeling intracranial recordings.

4.2 Model Analysis.

Low-resource labeled data evaluation. In medical scenarios, collecting labeled data for even small experiments is a huge investment. To demonstrate the practical value of our work, we evaluate Brant on seizure detection where the amount of labeled data is limited. Specifically, the pre-trained model is fine-tuned on 200 minutes, 60 minutes and 20 minutes of labeled data that sampled from the downstream dataset (Sec. 3.1), respectively. After fine-tuning, models are evaluated on the same 50 minutes of labeled data which is also sampled from the downstream dataset but non-overlapped with the fine-tuning data.

From Tab. 4, the performances of supervised methods decrease rapidly compared to the selfsupervised or unsupervised methods, showing that the representations learned on unlabeled data can improve low-resource settings. Among the pre-training works, our model maintains the most stable performance on 20-minute labeled data. Note that the F2 score of our model on 20-minute labeled data (51.03%) is even higher than the F2 score of the best baseline on 200-minute labeled data (43.60%), demonstrating that Brant fully captures the patterns and semantic information from the intracranial data during pre-training and adapts to downstream tasks more easily.

Representation analysis. As classification task can verify the model capacity in high-level representation learning [41], we visualize the pre-trained representations of Brant and 3 best pretraining based methods on seizure detection task using t-SNE (shown in Fig. 4). Compared to other methods, the representations of seizure and normal signals learned from Brant are separated more clearly during pre-training, which intuitively illustrates our SOTA performance on low-resource label settings. Furthermore, the results explain the good performance of Brant-Freeze on seizure detection.

Brant (our model)

CPC

BrainBERT

CoST

Model scale analysis. To explore the effect of model size on performance, we additionally pretrain three variants of Brant with smaller size: Brant-tiny, Brant-small and Brant-medium (detailed configurations are in App. F) on the same dataset as Brant. All models are pre-trained with an Adam optimizer and a cyclic scheduler. Brant-medium adopts the same learning rate as Brant (Sec. 3.2); for Brant-tiny and Brant-small, the basic and maximum learning rates are 6 × 10 -6 and 3 × 10 -5 .

We evaluate these smaller size variants on all the downstream tasks (results shown in Fig. 5). As the model size increases, the performances on the downstream tasks show an overall upward trend, where Brant is ahead of the other three variants in all metrics. Also, the decrease in the standard deviation indicates more stable performance for larger models. On such a huge neural dataset, a larger model with a higher capacity results in better generalization ability to a wide range of downstream tasks.

Short- & long-term Signal Forecasting

Frequency-phase Forecasting

Imputation

Seizure Detection

Ablation study. We perform ablation experiments to assess the effectiveness of each individual component in our model. Specifically, we remove the following components from our model to examine their respective effects on performance: the temporal encoder (Brant-temporal), the spatial encoder (Brant-spatial), and the frequency encoding (Brant-frequency).

The experimental results of ablation study are given in Tab. 5 and Tab. 6. The results indicate that Brant outperforms other model variants across all metrics, providing evidence for the contribution of each component in our model. Among the different variants, Brant-temporal exhibits the most substantial decrease in performance compared to the full Brant model, emphasizing the significance of long-term dependency in modeling brain signals. The performance degradation observed in Brantspatial demonstrates the role of considering channel spatial correlation in learning representations of intracranial recordings. Furthermore, the decline of Brant-frequency suggests that simultaneously extracting information from both the time and frequency domains proves beneficial for effectively modeling brain signals.

Generalization ability analysis. To further verify the generalization ability of Brant on more subjects with more heterogeneity, we evaluated the model on data of 31 unseen subjects from two public datasets named MAYO and FNUSA [42]. More details about this study are shown in App. G.

5 Related Work

Intracranial recordings modeling. With the development of iEEG technique to obtain deep brain information from both cortical and subcortical structures, modeling intracranial recordings has attracted the attention of many researchers. Wang et al. [9] extract time domain features of iEEG signals from multiple receptive fields by utilizing multiscale CNN and LSTM. Guo et al. [10] introduce a hypergraph learning approach to model iEEG signals by detecting high frequency oscillations. Yu and Hu [11] propose EDANN to learn domain-invariant representations of iEEG data from multiple subjects by domain adversarial training. Jiang et al. [8] develop a novel method using short-time resting-state connectivity to identify the seizure onset zone (SOZ) from interictal EEG signals. Wang et al. [12] conduct cross-subject iEEG seizure detection based on adaptive feature fusion of brain network features and single-channel features. However, these works are all supervised that relies heavily on labeled data, which is often difficult and expensive to obtain at scale. Furthermore, most works for intracranial data modeling process each channel independently, ignoring the spatial correlation. Although Chen et al. [23] propose BrainNet which contains a graph diffusion component which measures the brain wave diffusion among channels, their work only focus on the information in time domain and is limited to an individual subject.

Pre-training on brain signals. Pre-training on time series [37, 38, 27, 39] has shown good performance in many scenarios (e.g., weather, traffic flow, exchange rates), including some works designed for brain signals. Banville et al. [17] learn representations from EEG signals by a selfsupervised temporal context prediction task, revealing clear latent structures related to physiological and clinical phenomena. Kostas et al. [18] address the problem of limited labeled data on EEG by using a contrastive self-supervised learning task to pre-train a model named BENDR, which is then fine-tuned for downstream tasks. Potter et al. [19] propose an unsupervised approach to model EEG signals using a transformer-based model with a signal reconstruction task. Cai et al. [43] propose to study the self-supervised learning framework for brain signals that can be applied to pre-train either scalp or intracranial EEG data. Wang et al. [20] propose a pre-training work named BrainBERT for intracranial recordings and conduct experiments on iEEG data. They adopt time-frequency representations but ignore the spatial correlation between channels, which is critial in modeling intracranial recordings (additional commentary on differences between our work and BrainBERT is in App. D).

6 Conclusion

We propose a task-agnostic foundation model, Brant, which learns powerful representations of intracranial recordings. Brant is the largest pre-training model on brain signals, whose design (1) attends a long temporal dependency; (2) captures the spatial correlation between channels; and (3) extracts information from both time and frequency domains. Experimentally, Brant achieves consistent SOTA performance on various downstream tasks w.r.t. medical scenarios. Further analysis shows the effectiveness and benefit of a large-scale pre-trained model in the field of medicine. Brant is an off-the-shelf model with its code and weights, which significantly alleviates the issue of sample and label efficiency and can directly participate in other medical research and treatment.

Limitations and future works. By pre-training on a large amount of intracranial data, Brant contains over 500M parameters, far more than other existing works on brain signals. However, compared to other fields such as CV and NLP in which the models can reach billions of parameters and achieve good performance on a variety of tasks by zero-shot learning, there is still potential for further improvement of our work. In the future, by scaling up our dataset, the scale of our model can be further expanded to capture higher-level semantic information from neural data, revealing more complicated brain activities and dynamics, to provide assistance for more healthcare applications.

Acknowledgments

This work is supported by NSFC (No.62176233), the National Key Research and Development Project of China (No.2018AAA0101900) and the Fundamental Research Funds for the Central Universities.

References

Alessandra Del Felice, Leonora Castiglia, Emanuela Formaggio, Manuela Cattelan, Bruno Scarpa, Paolo Manganotti, Elena Tenconi, and Stefano Masiero. Personalized transcranial alternating current stimulation (tacs) and physical therapy to treat motor and cognitive symptoms in parkinson's disease: a randomized cross-over trial. NeuroImage: Clinical , 22:101768, 2019.

Florian Mormann, Ralph G Andrzejak, Christian E Elger, and Klaus Lehnertz. Seizure prediction: the long and winding road. Brain , 130:314-333, 2007.

Palak Handa and Nidhi Goel. Epileptic seizure detection using rhythmicity spectrogram and cross-patient test set. In 2021 8th International Conference on Signal Processing and Integrated Networks (SPIN) , pages 898-902, 2021.

Scott B. Wilson, Mark L. Scheuer, Ronald G. Emerson, and Andrew J. Gabor. Seizure detection: evaluation of the Reveal algorithm. Clinical Neurophysiology , 115:2280-2291, 2004.

Zisheng Zhang and Keshab K. Parhi. Low-Complexity Seizure Prediction From iEEG/sEEG Using Spectral Power and Ratios of Spectral Power. IEEE transactions on biomedical circuits and systems , 10:693-706, 2016.

Badreddine Mandhouj, Mohamed Ali Cherni, and Mounir Sayadi. An automated classification of EEG signals based on spectrogram and CNN for epilepsy diagnosis. Analog Integrated Circuits and Signal Processing , 108:101-110, 2021.

Yudhajit Das, Hanli Liu, Fenghua Tian, Srinivas Kota, Rong Zhang, and Lina F. Chalak. Rigor of Neurovascular Coupling (NVC) Assessment in Newborns Using Different Amplitude EEG Algorithms. Scientific Reports , 10:9183, 2020.

Haiteng Jiang, Vasileios Kokkinos, Shuai Ye, Alexandra Urban, Anto Bagi´ c, Mark Richardson, and Bin He. Interictal seeg resting-state connectivity localizes the seizure onset zone and predicts seizure outcome. Advanced Science , 9:2200887, 2022.

Yiping Wang, Yanfeng Yang, Gongpeng Cao, Jinjie Guo, Penghu Wei, Tao Feng, Yang Dai, Jinguo Huang, Guixia Kang, and Guoguang Zhao. Seeg-net: An explainable and deep learningbased cross-subject pathological activity detection method for drug-resistant epilepsy. Computers in Biology and Medicine , 148:105703, 2022.

Jiayang Guo, Hailong Li, Xiaoshuai Sun, Lei Qi, Hui Qiao, Yijie Pan, Jing Xiang, and Rongrong Ji. Detecting high frequency oscillations for stereoelectroencephalography in epilepsy via hypergraph learning. IEEE Transactions on Neural Systems and Rehabilitation Engineering , 29:587-596, 2021.

Hao Yu and Mengqi Hu. Epilepsy seeg data classification based on domain adversarial learning. IEEE Access , 9:82000-82009, 2021.

Ying Wang, Zhengfeng Bao, Yiping Wang, Yanfeng Yang, Tao Feng, and Guixia Kang. A cross-patient seeg epileptic signal detection method based on adaptive feature fusion of brain network features and single-channel features. In Proceedings of the 2022 9th International Conference on Biomedical and Bioinformatics Engineering , pages 165-172, 2022.

J. Bancaud. Surgery of Epilepsy Based on Stereotactic Investigations-the Plan of the SEEG Investigation. In F. John Gillingham, Jan Gybels, Edward Hitchcock, Gian Franco Rossi, and Gábor Szikla, editors, Advances in Stereotactic and Functional Neurosurgery 4 , pages 25-34, 1980.

Greg A. Worrell, Andrew B. Gardner, S. Matt Stead, Sanqing Hu, Steve Goerss, Gregory J. Cascino, Fredric B. Meyer, Richard Marsh, and Brian Litt. High-frequency oscillations in human temporal lobe: simultaneous microwire and clinical macroelectrode recordings. Brain , 131:928-937, 2008.

MKalaivani, V Kalaivani, and V Anusuya Devi. Analysis of eeg signal for the detection of brain abnormalities. at International Journal of Computer Applications® year , 2014.

Santiago Morales and Maureen Bowers. Time-frequency analysis methods and their application in developmental eeg data. Developmental Cognitive Neuroscience , 54:101067, 2022.

Hubert Banville, Omar Chehab, Aapo Hyvärinen, Denis-Alexander Engemann, and Alexandre Gramfort. Uncovering the structure of clinical eeg signals with self-supervised learning. Journal of Neural Engineering , 18:046020, 2021.

Demetres Kostas, Stephane Aroca-Ouellette, and Frank Rudzicz. Bendr: using transformers and a contrastive self-supervised learning task to learn from massive amounts of eeg data. Frontiers in Human Neuroscience , 15:653659, 2021.

˙ Ilkay Yıldız Potter, George Zerveas, Carsten Eickhoff, and Dominique Duncan. Unsupervised multivariate time-series transformers for seizure identification on eeg. arXiv preprint arXiv:2301.03470 , 2023.

Christopher Wang, Vighnesh Subramaniam, Adam Uri Yaari, Gabriel Kreiman, Boris Katz, Ignacio Cases, and Andrei Barbu. Brainbert: Self-supervised representation learning for intracranial recordings, 2023.

Yi Ding, Neethu Robinson, Qiuhao Zeng, and Cuntai Guan. Lggnet: Learning from localglobal-graph representations for brain-computer interface. CoRR , abs/2105.02786, 2021.

Neeraj Wagh and Yogatheesan Varatharajah. Eeg-gcnn: Augmenting electroencephalogrambased neurological disease diagnosis using a domain-guided graph convolutional neural network, 2020.

Junru Chen, Yang Yang, Tao Yu, Yingying Fan, Xiaolong Mo, and Carl Yang. Brainnet: Epileptic wave detection from seeg with hierarchical graph diffusion learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , page 2741-2751, 2022.

Eunjin Jeon, Wonjun Ko, Jee Seok Yoon, and Heung-Il Suk. Mutual information-driven subjectinvariant and class-relevant deep representation learning in bci. IEEE Transactions on Neural Networks and Learning Systems , 2021.

Hubert J. Banville, Isabela Albuquerque, Aapo Hyvärinen, Graeme Moffat, Denis-Alexander Engemann, and Alexandre Gramfort. Self-supervised representation learning from electroencephalography signals. CoRR , abs/1911.05419, 2019.

Siyi Tang, Jared Dunnmon, Khaled Kamal Saab, Xuan Zhang, Qianying Huang, Florian Dubost, Daniel Rubin, and Christopher Lee-Messer. Self-supervised graph neural networks for improved electroencephalographic seizure analysis. In International Conference on Learning Representations , 2022.

Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers, 2023.

Zisheng Zhang and Keshab K. Parhi. Low-complexity seizure prediction from ieeg/seeg using spectral power and ratios of spectral power. IEEE Transactions on Biomedical Circuits and Systems , 10:693-706, 2016.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.

Raluca Maria Aileni, Sever Pasca, and Adriana Florescu. Eeg-brain activity monitoring and predictive analysis of signals using artificial neural networks. Sensors , 20(12):3346, 2020.

DK Thara, BG Premasudha, TV Murthy, and Syed Ahmad Chan Bukhari. Eeg forecasting with univariate and multivariate time series using windowing and baseline method. International Journal of E-Health and Medical Communications (IJEHMC) , 13:1-13, 2022.

Farrokh Mansouri, Katharine Dunlop, Peter Giacobbe, Jonathan Downar, and José Zariffa. A fast eeg forecasting algorithm for phase-locked transcranial electrical stimulation of the human brain. Frontiers in Neuroscience , 11, 2017.

Anke Karabanov, Axel Thielscher, and Hartwig Roman Siebner. Transcranial brain stimulation: closing the loop between brain and stimulation. Current Opinion in Neurology , 29(4):397-404, 2016.

Atsunori Kanemura, Yuhsen Cheng, Takumi Kaneko, Kento Nozawa, and Shuichi Fukunaga. Imputing missing values in eeg with multivariate autoregressive models. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) , pages 2639-2642, 2018.

Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. CoRR , abs/2202.01575, 2022.

Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency, 2022.

Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. CoRR , abs/2106.14112, 2021.

Maliheh Kadivar, Elahe Movahedi Moghadam, Reza Shervin Badv, Raziye Sangsari, and Maryam Saeedy. A comparison of conventional electroencephalography with amplitudeintegrated eeg in detection of neonatal seizures. Medical Devices: Evidence and Research , pages 489-496, 2019.

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis, 2023.

Petr Nejedly, Vaclav Kremen, Vladimir Sladky, Jan Cimbalnik, Petr Klimes, Filip Plesinger, Filip Mivalt, Vojtech Travnicek, Ivo Viscor, Martin Pail, et al. Multicenter intracranial eeg dataset for classification of graphoelements and artifactual signals. Scientific data , 7:179, 2020.

Donghong Cai, Junru Chen, Yang Yang, Teng Liu, and Yafeng Li. Mbrain: A multi-channel self-supervised learning framework for brain signals, 2023.

Ethics Statement

The data collection and experiments conducted in our work have been approved by the Institutional Review Board (IRB) and passed ethical review. All participants have signed informed consent forms.

A Intuition of Using PSD as Weights

PSD displays the spectal power distribution in different frequencies of brain signals. The power distribution in different frequency bands is associated with different brain functional states. For example, during wakefulness, α and β waves are more active; during sleep, δ and θ waves are more prominent; and γ waves will become significant during epileptic seizures. Therefore, in medical research, the power distribution in different frequency bands is often used as a feature reflecting the functional state of the brain.

In our work, we treat the power distribution as the weights of learnable representations f i , i = 1 , ..., 8 in 8 standard frequency bands. The intuition is that the frequency bands with higher power are more representative of the brain's physiological state, and should be given more weight in their representations. In this way, the model can focus more on the frequency bands where the main power of the brain signals resides.

B Details of the Dominant Frequency and Phase

Dominant frequency. Dominant frequency is defined as the frequency related with the greatest power or amplitude in the brain signal. Dominant frequency can be calculated from Fourier Transform spectrum. Mathematically, given a brain signal series s ( t ) , t ∈ ( t 0 , t 1 ) , the amplitude A ( ω ) on frequency ω is:

and the dominant frequency ω D is:

Phase. The instantaneous phase of a complex-valued function signal s ( t ) is the real-valued function:

where angle ( · ) is the angle between the positive real axis and the direction of complex number s ( t ) .

C Details of Baselines

Firstly, we compare our model to the existing self-supervised or unsupervised pre-training works on brain signals. The Details of these baseline models are given here:

RP, TS, CPC: Three self-supervised learning approaches with three self-supervised learning pretext tasks of relative positioning (RP), temporal shuffling (TS) and contrastive predictive coding (CPC) to learn representations of EEG signals based on StagerNet.

BENDR: A self-supervised training model that learn compressed representations to model completely novel raw EEG sequences recorded with differing hardware, and different subjects performing different tasks.

MVTS: An unsupervised transformer-based anomaly detection approach to train an autoencoder with a masking strategy on non-seizure signals.

BrainBERT: A reusable transformer for intracranial field potential recordings enables classifying complex concepts and decoding neural data.

Secondly, we compare our model with the advanced self-supervised or unsupervised pre-training works designed for general time series, including:

CoST: A time series representation learning framework for long sequence time series forecasting, which applies contrastive learning methods to learn disentangled seasonal-trend representations.

TF-C: A decomposable pre-training model, where the self-supervised signal is provided by the distance between time and frequency components, each individually trained by contrastive estimation.

PatchTST: An efficient design of Transformer-based models for multivariate time series forecasting and self-supervised representation learning, based on the segmentation of time series into subseries-level patches.

TS-TCC: An unsupervised time series representation learning framework to learn robust temporal and discriminative representations by designing a tough cross-view prediction task and a contextual contrasting module.

Furthermore, as seizure detection is the most important application scenario of intracranial recordings, to evaluate Brant's ability in seizure detection, we compare our model with some seizure detection methods on EEG or iEEG. These baselines include handcrafted feature based methods and a deep learning based method:

Spectral Power: The spectral power in a particular frequency band means the power of a signal in that frequency band. The rhythmic activity on brain signals is typically described in terms of the spectral power or its ratio on some standard frequency bands.

Rhythmicity Spectrogram: This feature displays a density spectral array of frequency and power characteristics of brain signals, providing a graphical depiction of the amplitude of primary rhythmic EEG components present in four frequency bands.

Amplitude-integrated EEG: A trace of electrical activity displayed on a semilogarithmic graph of peak-to-peak amplitude over time, which can be used to monitor and diagnose seizure activity.

SEEG-Net: A model that can address the problems of sample imbalance, cross-subject domain shift, and poor interpretability and realizes high-sensitivity SEEG pathological activity detection. Since the source code of SEEG-Net is not released, we re-implement it to conduct experiments.

D Detailed Differences between Brant and BrainBERT

BrainBERT is also a pre-training work conducted on intracranial recordings, but our work is different with them in the following aspects.

Data and model scale. Pre-training often requires a large amount of data by which the model fully learns the distributions, and the pre-trained model usually contains a huge quantity of parameters to fit the data. According to the paper of BrainBERT and the model weights released by its authors [20], the total amount of data and parameters in BrainBERT is 43.7 hours and 43.18M, respectively. Brant is pre-trained on 2528 hours of data and the scale of Brant reaches to 505.69M parameters.

Method. As discussed in the main paper, Brant considers long-term dependency, channel correlation, time and frequency domains when modeling intracranial recordings. BrainBERT provides contextualized neural embeddings and adopts time-frequency representations, but ignores the channel correlation which reflects the propagation of brain waves.

Downstream tasks. BrainBERT is evaluated on four classification tasks related with the audio listened from the subjects, which mainly focus on a particular stimulus, i.e., passive movie viewing. Brant is evaluated on various downstream tasks, including signal forecasting, frequency-phase forecasting, imputation and seizure detection, all of which are medically valuable (described in Sec. 3.3).

E Details of the Datasets

Pre-training dataset. The pre-training dataset contains 1.01TB data with a sampling rate of 1000Hz. The details of the pre-training dataset are given in Tab. 7.

Downstream dataset. The downstream dataset contains 29.39 GB data with a sampling rate of 1000Hz. The details of the downstream dataset with seizure labels are given in Tab. 8. For each downstream task, we sample a small subset from the downstream dataset for fine-tuning and evaluation. The sampling and split strategies are described in the setup of downstream tasks in Sec. 3.3.

F Details of Brant with Different Scales

To explore the effect of model scales on performance, we additionally pre-train three variants of Brant with smaller size: Brant-tiny, Brant-small and Brant-medium on the same dataset. Detailed configurations of Brant and its variants are shown in Tab. 9.

G Details of the Generalization Ability Analysis

As a supplement to the experiments on downstream dataset in the main text, to further verify the generalization ability of Brant on more subjects with more heterogeneity, we evaluated the model on data of 31 unseen subjects from two public datasets named MAYO and FNUSA [42]. The MAYO dataset includes intracranial neural signals of 18 subjects collected from Mayo Clinic (Rochester, Minnesota, United States of America). The FNUSA dataset comprises intracranial data of 13 subjects recorded from St. Anne's University Hospital (Brno, Czech Republic).

The results of generalization study are given in Tab. 10, Tab. 11 and Tab. 12. Compared to other baseline models, Brant holds the consistent SOTA performance on these downstream tasks on the unseen 31 heterogeneous subject data collected from different hospitals in different countries, highlighting the generalization ability of our model.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a foundation model (called BPT) for modeling intracranial brain signals. BPT consists of a Transformer encoder for temporal encoding, and a separate Transformer encoder for spatial encoding. BPT is pre-trained for reconstructing masked input signals, and is fine-tuned for a variety of downstream tasks (short-term and long-term forecasting, seizure detection etc.). Experimental results suggest that BPT outperforms other self-supervised/unsupervised pre-training models, as well as supervised models on downstream tasks. Model scaling analysis further suggests the benefit of having a large model size.\n\nSTRENGTHS:\n1. While the ideas of self-supervised pre-training using a reconstruction task and modeling brain signals using Transformer encoders are not new, the originality of this work comes from a large model pre-trained (largest model for brain signals to date) on a large dataset, as well as its analyses on a variety of downstream tasks.\n2. The methods are technically sound, and the experiments are carefully designed to support the authors’ claims.\n3. The paper is relatively easy to understand.\n\nWEAKNESSES:\n1. Baselines seem to be chosen selectively. Some existing works are shown in Figure 1 but not included as baselines (e.g., GNN-based methods, including BrainNet, EEG-GCNN, Tang et al. [23]).\n2. While the pre-training dataset is large (1.1TB), it only consists of 9 subjects (Appendix E), which could limit the representativeness of the learned latent representations.\n3. There is an overload of notations in the Method section.\n4. The authors claim that BPT is a foundation model for brain signals. However, as I noted above, the pre-training dataset only comprises iEEG data from 9 subjects. The downstream experiments are also done on iEEG data from a small number of subjects (8 subjects). In my opinion, more diverse pre-training data and more downstream experiments on larger datasets are needed to demonstrate that BPT can be served as a foundation model.\n\nQUESTIONS:\n1. Notations: In “Model overview” section, $L$ is used to denote the number of patches. But later in “Patching” section, $N_p$ is used to denote the number of patches. Please be consistent.\n2. What’s the intuition of frequency encoding using PSD as weights? Please justify in “Frequency Encoding” section.\n3. The patch length P is projected to dimension D (“Encoding Process” section) using a linear layer. Doesn’t this ignore temporal dependency within the patch? In my opinion, this is a limitation and should be addressed (e.g., using a small CNN layer) or discussed.   \n4. The patch length is selected to be 6s. Please justify this choice.\n5. Please include some ablation studies to demonstrate the effectiveness of each component in BPT. The ablation studies can be centered around the 3 claimed advantages: 1) long-range dependency, 2) spatial correlation, and 3) time and frequency domains.\n6. Is the data split patient-wise? If not, the downstream tasks may be relatively easy as the same patient’s signals are seen during fine-tuning.\n7. For the imputation task, it is unsurprising that BPT would perform well given that it’s pre-trained for a masked reconstruction task. Instead of imputing masked timestamps, a more challenging imputation task would be to mask an entire channel and impute the values in the masked channel.\n8. Please also include GNN-based approaches listed in Figure 1 as baselines. e.g., BrainNet, EEG-GCNN, Tang et al. [23].\n\nLIMITATIONS:\nLimitations about the dataset size are discussed. In my opinion, the following two points are also limitations and should be discussed:\n- Data come from a small number of subjects.\n- Projecting the temporal dimension of a patch using a linear layer.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

comments: SUMMARY:
This paper presents Brain Pre-trained Transformer (BPT) for intracranial neural signal modeling. The model was trained on a large-scale dataset and achieved SOTA performance on multiple downstream tasks.

STRENGTHS:
- originality: the authors developed a transformer model for brain signal modeling, which can learn rich representations from intracranial neural recordings.
- quality: this paper presents the largest pre-training model on brain signals, which can capture long temporal dependency and spatial correlations.
- clarity: the paper is well-written and organized.
- significance: the model achieved consistent SOTA performance on various downstream tasks.

WEAKNESSES:
I'm not an expert in this field. So I'm not able to give constructive comments on the weakness.

As a foundation model for medical scenarios, it seems that the interpretability analysis is missed.

QUESTIONS:
Will the datasets be publicly available?

LIMITATIONS:
It would be great to provide the interpretability analysis of the learned features.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a foundation model named BPT for modeling intracranial neural signals. BPT is pre-trained on a large corpus (1.01 TB) of intracranial recordings collected by the authors. Animportant feature is that BPT is designed to capture long-term temporal dependency and spatial correlation from neural signals, combining information in both time and frequency domains. BPT is pre-trained using a masked autoencoder objective. The results claim that  BPT achieves state-of-the-art performance on various downstream tasks.\n\nSTRENGTHS:\n- The paper proposes a novel self-supervised foundation model for modeling intracranial neural recordings.\n- The authors claim that the size of the model is of unparalleled precedent.\n- The experiments are thorough, well-designed, and provide strong empirical evidence for the effectiveness of the proposed method. \n- The paper is clearly written, with a well-structured flow of ideas. \n- The work has high potential impact and significance for the medical field and perhaps a step towards future neural interfaces.\n- The model will be made publicly available.\n\nWEAKNESSES:\n- The work is scaling up previous architectures. No \n- if possible it would be great to see model preformance on more clinically meaningful tasks\n- The model analysis is a bit superficial. Would be useful for the reader to have a more in-depth analysis, e.g. how different frequency bands contribute to the model performance, how the learned representations capture spatial or temporal relationships in iEEG signals, etc.\n- The results are not compared with other SOTA models on the same datasets. Despite this may not be feasible, specially on medical domain.\n\nQUESTIONS:\n- The pre-training task adopted in this work is masked autoencoder. Why did the authors choose this task over other self-supervised tasks (e.g. contrastive learning, denoising autoencoder)? A comparison and analysis on different pre-training tasks would be helpful.\n\n- Patients demographics: The dataset seems to contain data from many subjects. The analysis of the dataset does not provide much details on the subjects information (e.g. age, gender, diagnoses). More details on this would help to assess generability of the model.\n\n- Is it possible to compare with humans for the same tasks?\n\nLIMITATIONS:\n- Lack of external validation, despite not being feasible due to the lack of dataset.\n- The authors should discuss any potential privacy concerns or negative societal impacts as neural data is highly sensitive and building large models from such data raises privacy concerns.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript trains a foundation model for intracranial signals based upon a very large electrophysiological dataset.  The methods are based upon a transformer architecture with adaptations to account for the electrode structure and important frequency properties in electrophysiological systems.  The manuscript present experiments compared to many baselines on the tasks, include forecasting, imputation, and seizure detection.\n\nUpdate after rebuttal:\nMost of my concerns have been addressed on the extensive rebuttal, and I have therefore adjusted my score.  Some of the weaknesses remain, such as the limited demonstration of downstream tasks, which is limited to seizure detection, and the original method for seizure detection not using split subjects.  In the final version, these limitations should be noted.  Additionally, the method for calculating uncertainty should be set to report inter-subject variability, and the approach used should be included in the revised manuscript.\n\nSTRENGTHS:\nThis type of electrophysiological signals are of high importance in many fields, including several types of electrophysiological signals.  This manuscript could contribution to an exciting line of work building foundation models that are capable of enhancing neuroscience, neurology, neurosurgery, and psychiatry.  The need for foundation models in this area is huge, as studies are frequently limited by relatively small sample sizes, usually insufficient to capture complex patterns appearing in the brain.\n\nThe ideas are presented in a straightforward way, and the application is clear.  \n\nIn my view, the primary contribution is simply to run these algorithms at such a large scale, which is critically necessary to start building real foundation models. \n\nI was impressed by the number of comparison algorithms that were run.\n\nWEAKNESSES:\nWhile I am enthused by the ideas put forth in this manuscript, I view several claims in this manuscript as exaggerated or unsupported.  First, it is necessary to note that while the data scale is huge, the number of individuals is not.  The training data is made up of $N_{train}=9$ individuals, and the test set is made up of $N_{test}=8$ individuals.  As the primary source of heterogeneity in many tasks is the differences between individuals, it is important to assess the variance and performance over individuals rather than windows (e.g., see [1]).  The reported uncertainties seem entirely too small to be over 8 individuals, and those metrics need to be recalculated to more accurately reflect the performance and evidence for the method.\n\nIn a similar vein, it is typical in seizure detection to separate out individuals for the training and test set, whereas this manuscript uses data from the same individuals, as described in lines 198-200.  This overestimates performance.  Notably, the mentioned methods do cross-subject validation (e.g. your reference 9).\n\nOne of the claims is that this method is much improved over competing methods, but the training data for methods was different.  As mentioned in Supplemental Section C, BrainBERT was pretrained on a tiny fraction of the data that BPT was trained on.  As such, it is difficult to assess whether the performance improvement was due to greater amount of training data (as well as better matched training data, as inter-device comparisons can be challenging).  This should be clarified by training methods on all the same data and with similar schemes.  Likewise, the other comparisons were not trained on the same data, making it difficult to assess whether the performance improvement is due to methodological improvement or simply better matched training data (e.g., EEG-> intracranial recordings is non-trivial).\n\nThe claims in the manuscript is that this method is shown to generalize well to several medical scenarios, but I do not see evidence to support that.  The only medical scenario that I see is on seizure, whereas the other tasks are not directly medical scenarios.  Additionally, there are several limitations of this claim: (1) trained on only a single hospital, whereas hospital and device differences can significantly shift signals; (2) total number of individuals in the data set is very small; and (3) the comparisons are not especially clear since they were trained on alternative datasets.  I would encourage the authors to precisely evaluate what evidence of generalization they have provided.\n\n\n[1] Vu, Mai-Anh T., et al. \"A shared vision for machine learning in neuroscience.\" Journal of Neuroscience 38.7 (2018): 1601-1607.\n\nQUESTIONS:\nIs the performance improved due to larger and better matched training data, or because the method is inherently better?\n\nOnce you account for the fact that there are only 8 individuals in the test set, how strong is the evidence of improvement?\n\nLIMITATIONS:\nThe limitations were not properly addressed; please see weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}