Brant: Foundation Model for Intracranial Neural Signal
Authors: Daoze Zhang
Zhizhang Yuan
Yang Yang
Junru Chen
Jingjing Wang
Yafeng Li
Venue: NeurIPS 2023 poster (2023)
Keywords: Foundation model
Brain signal
Pretraining
Medicine
TL;DR: We propose a foundation model for modeling intracranial recordings, providing a large-scale, off-the-shelf model for medicine.

Abstract: We propose a foundation model named Brant for modeling intracranial recordings, which learns powerful representations of intracranial neural signals by pre-training, providing a large-scale, off-the-shelf model for medicine. Brant is the largest model in the field of brain signals and is pre-trained on a large corpus of intracranial data collected by us. The design of Brant is to capture long-term temporal dependency and spatial correlation from neural signals, combining the information in both time and frequency domains. As a foundation model, Brant achieves SOTA performance on various downstream tasks (i.e. neural signal forecasting, frequency-phase forecasting, imputation and seizure detection), showing the generalization ability to a broad range of tasks. The low-resource label analysis and representation visualization further illustrate the effectiveness of our pre-training strategy. In addition, we explore the effect of model size to show that a larger model with a higher capacity can lead to performance improvements on our dataset. The source code and pre-trained weights are available at: https://zju-brainnet.github.io/Brant.github.io/.

Brant: Foundation Model for
Intracranial Neural Signal
Daoze Zhang" Zhejiang University zhangdzōzju. edu.
Zhizhang Yuan" Zhejiang University zhizhangyuan02juu edu. edu en
Yang Yang! Zhejiang University yangyačzju. edu. cn
Junru Chen
Jingjing Wang Zhejiang University wj wjjxjezzju edu. en
Yafeng
Zhejiang University j bechen_calizzjj edw..u.edu.co edu. en
Nuozhu Technology Co., Ltd. yafeng, wwwneerox..ccommm cn
Abstract
We propose foundation model named Brant for modeling intracranial recordings, which learns powerful representations of intracranial neural signals by pre-training, providing lcgge-scale, off-the-shelf model for medicine. Brant the largest model in the field of brain signals and is pre-trained collected by us. The design of Brant to capture long-term temporal dependency and spatial correlation from neural signals, combining the information in both time and frequency domains. As a foundation model, Brant achieves SOTA performance various downstream tasks (i.e. neural signal forecasting, frequency siguency-ppasee forecasting, imputation and seizure detection), showing the generalization ability broad range of tasks. The low-resoures label analysis and representation visualization further illustrate the effectiveness of our pre-training strategy. In addition, explore the effect of model size to show that larger model with higher capacity can lead to performance improvements on our dataset. The source code and pre-trained weights available at: hntps://zju--raimmet. werainnet.gitine io/Brant. B//Brant.gituu.... io/
large corpus of intracranial data
Introduction
Brain signals are electrical impulses that are generated by brain neurons and transmit through neural networks. These signals provide important information about brain activity and can usually be monitored in two ways, namely scalp electroencephalography (EEG) and intracranial electroen- cephalography (iEEG). The former records the electrical brain activity through electrodes placed on the scalp, while the latter implants intracranial electrodes into brain tissue directly to derive neural recordings. Compared with EEG, iEEG manifests significant advantages by providing more abundant, stereotactic and detailed information about brain wave patterns from deeper brain structures, which has been the mainstream method to obtain deep brain information, and is essential in the therapics for some brain discases (e.g., Parkinson's discase [ epileptic seizure 2
Modeling intracranial recordings has drawn much research attention, but several issues still remain unresolved. Currently, studies for modeling intracranial recordings are mainly divided into two research lines, namely handcrafted feature based methods (3/S] and deep learning based methods [9H2). Handcrafted feature engineering requires lots of domain knowledge and may only work on specific tasks. And most deep learning based methods are fully supervised, which relies heavily on
Equal contribution. Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

labeled data. However, labeling data at scale in medical experiments is often infeasible or expensive, which underscores the importance of maximizing the label efficiency. To overcome these limitations, the paradigm of self-suuer pre-training followed by fine-tuning with few samples can greatly reduce the reliance on labels and enable the model to generalize to various downstream tasks.
Moreover, modeling intracranial recordings requires careful consideration of several key factors. (I) Long-term dependency. Since intracranial neural signals are time series and gradual changes in brain activity may only be captured by the long-period analysis [3] long-term temporal dependency is crucial in modeling intracranial data. (2) Spatial correlation. The electrodes implanted in the brain contain many contacts (also called channels), which are distributed across various brain regions. Due to the fact that brain waves propagate through different brain regions [14, signals recorded from different channels can be spatially correlated, reflecting the underlying neural activity. (3) Time and frequency domains. For neural recordings, the time domain provides information about the amplitude and duration while the frequency domain can reveal underlying oscillatory patterns and rhythms [15]. Therefore, modeling neural signals both domains can provide information more consistent with the neurophysiological mechanisms [6.. To the best of our knowledge, existing work intracranial recordings considers all the three key factors simultaneously.
In view of the unresolved issues above, we pro- pose a foundation model for intracranial neural signal named Brain Neural Transformer (Brant). The design of our model takes all the three key
505.69
Parameters IM
factors (i.e., long-term dependency, spatial corre- lation, time and frequency domains) for intracra- nial signal modeling into account. Moreover, Brant contains more than 500M parameters and is pre-trained large intracranial dataset with 1.01 TB data, which can be adapted to accom- plish various downstream tasks, Compared to other existing methods for modeling brain sig- nals, Brant can achieve better performance with far fewer labeled samples, showing the great benefit of work in medical scenarios. As an off-the-shelf model along with the code and weights, Brant can participate in other medical research and experiments, which alleviates the issue of sample and label efficiency.
43.18
030.030.260.3900
0.230.550.720.01
1.295 044 044.49
8P
15
BENDR
MVTS BrainBeeT Banville et Tang an al Looter
EEG-CCHI
Brainher
SIC
SBEG-N
Brant
Figure 1: The model scale of existing brain signal models, including RP [7]. TS [7], CPC BENDR 18 . MVTS
[7]
19 BrainBERT
20
LGGNet 21). EEG-GCNN (22), BrainNet
23
SICR 24). SEEG-Net 9 and two works from
Banville et al.
25 and Tang et al. [26).
To sum up, the main contributions of our work comprise:
We propose a foundation model for intracranial neural signals named Brant, which is the largest model on brain signals (shown in Fig. I and pre-trained on a large intracranial dataset collected by us, providing a large-scale and off-the-shelf model for medicine.
To our knowledge, Brant is the first to date that attends long-term dependency and captures spatial correlation across channels, while combining the information from both time and frequency domains.
Extensive experiments show that Brant generalizes well to various downstream tasks, show- ing the great potential in neural recordings modeling. Further analysis illustrates the effectiveness of large-scale pre-trained model, demonstrating the medical value of our work.
2 Method
Model overview. As previously mentioned, we propose Brant to capture long-term dependency and spatial correlation from intracranial recordings, while combining the information from both time and frequency domains. Our model mainly consists of two Transformer encoders, namely, temporal encoder and spatial encoder (shown in Fig. 2). The temporal encoder encodes a sequence of I, consecutive patches which focus on the temporal dependency, and the spatial encoder encodes C patches with the same time indices from all channels to capture their underlying spatial correlation.

---

Record
Channels
P---L-1 P
Iaput Encoding
Linear Projection
Electrode Positions
P
Temporal Encoder
kygt-1
Masked patches
Reconstructed patches
Temporal representations
h T--
Spatial Encoder
Final representations
The pre-training framework of Brant. The collected intracranial neural recordings first processed to set of patches E N as the input. Then each time we mask a subset of patches P and map the input patches the hidden space while adding positional and frequency information to derive the input encoding Pj:+--11- The temporal encoder encodes the input encoding in each channel to derive the temporal representations hj The spatial encoder encodes the temporal representations with the same time indices from all channels to obtain the final L-1-
Figure 2:
representations 2 L-1- Then linear head is used obtain the reconstructed patches Pj:++L-1 from the final representations.
During pre-training, we randomly mask the input signals recorded from the implanted electrodes, then linearly map the original signals to the latent space and add positional and frequency information through input encoding. Then the input encoding is encoded to latent representations and linear projection head added on the top of the spatial encoder which maps the representations to reconstruct the original signals. The details of our model and pre-training task are described as below.
electrical signals with high sampling rates, we aggregate timestamps into patches to (1) enhance the locality and extract semantic information; (2) reduce computation and memory usage; and (3) attend longer temporal dependency [27]. Specifically, given neural signal æ E RRV NCC where N is the number of timestamps and C' is the number of electrode channels, we divide æ with length M and stride to generate set of patches p N2 where N, = N-M is the number of patches in each channel.
Patching. Since neural recordings
Frequency encoding. We propose frequency encoding explicitly inject the information frequency domain to the observed data. The frequency encoding is mainly based on power spectral density (PSD) which describes the distribution of signal's total average power over frequency (details about PSD are in App. A A. To be specific, we split the frequency domain into several bands according to the standard description for rhythmic activity 28 (1) (4-8Hz), (2) a (8- (8-13Hz), (3) 8 (13-30Hz), (4) 71 (30-50Hz), (5) 72 (50-70Hz), (6) 33 (70-90Hz), (7) 74 (90- 10Hz). (8) 75 1110--28H2) For the i-th frequency band, learnable encoding f. which is shared across all the patches. Then we compute the absolute spectral power of each patch Pj.c E p.j = 1, 1.... Np;c I,.... C. in the i-th frequency band:
its representation
(t)(()= log
PSDps. (w). iE (1,).....88 (bbnan()))
Σ
(1)
which acts as the weight of f. The frequency encoding F,,c E IR of patch py.c is obtained as the weighted sum of the learnable encodings fü
8
exp exp(Pr.(())
Fj, Σ 1=1
E exp(PP,c(i")) fr:
(2)
Encoding process. contains several steps encode the input signal to latent representations. Specifically, the input Pjs+L-1 E gLxXxM contains L x C patches and Pis+L-1ee E B L

---

denotes the patches in c-th channel. We first map each sequence of patches P.:itL-l,ei ..., to the latent space of dimension D by linear projection W Wooj poo E RDxM and the projected input
will be added with learnable positional encoding W pos RR 4x which monitors the temporal order of patches and the frequency encoding:
Pyjt--1,c- (W (WmnjP) T
pwnpijstt---- j-+L-11,) + W pus + F
(3)
where Pj.j-L-L.c E R LLXD denotes the input encoding of the original signals PAJ+L-1e- The imput encoding will be fed into the temporal encoder to obtain temporal hidden representations his+t-lle E IR Lx and we denote the temporal representations of the whole input as h E RLxCxD, The spatial encoder further captures the spatial correlation across channels, which takes each of the temporal representations h2 E R RCDD k = j. j...... + L 1 as the input and outputs the final representations 2 E RCx X k j. + 1. Overall, given the input signal p +. the encoding process returns the corresponding latent representations 3j:++L-1 E R LLXXx
Self-supervised pre-training. Self-supervised representation learning is powerful approach to extract high level abstract representation from unlabelled data. Among those methods to learn representation via self-supervised pre-training, masked autoencoder (MAE) has been proved to be simple but effective way in many fields [27 29 30. We apply this technique to our self-supervised pre-training, in which the model is trained reconstruct the whole input given its partial observation.
Given the input patches P) we mask a subset of patches uniformly at random and encode the masked patches to the latent representations L- During the pre-training stage, the repre- sentations will be fed into flatten layer with linear head W E RR to reconstruct the original patches. Finally we utilize an MSE loss to measure the discrepancy between the reconstructed patches PJ:++-1 and the original patches P
3 Experimental Setup
3.1 Dataset
Pre-training dataset. Brant is pre-trained on 1.01 TB neural data, a large clinical intracranial neural signal dataset recorded by stereneeletroonneehhallgaaa (SEEG) technique from first-class hospital. The subjects undergo a surgical procedure to implant 4 to 11 invasive electrodes, each with 52 153 channels, in their brain. The dataset contains 2528 hours of 1000Hz recordings with more than trillion timestamps. More details are in App. E We down sample the original signals to 250Hz and generate set of patches of 6s (1500 timestamps).
Downstream dataset. Another neural dataset collected by us with seizure labels is used to fine-tume and evaluate model. It contains 29.39 GB data with 43 hours of 1000Hz intracranial recordings and we do the same preprocessing (i.e. down sampling and patching) to it as the pre-training dataset. Professional neurosurgeons participate in the labeling of epileptic seizures and the labels consist of two categories, namely, seizure and normal samples. More details are in App. E For each downstream task, we sample small subset from the downstream dataset for fine-tuning and evaluation.
3.2 Pre-training
For the model configurations, the temporal encoder contains a 12-layer Transformer encoder with model dimension 2048, inner dimension (FFN) 3072 and 16 attention heads, and the spatial encoder contains a 5-layer Transformer encoder with model dimension 2048, inner dimension 3072 and 16 attention heads. During the pre-training, 40% patches in each input sample are masked with zero values uniformly at random. We take 16 input samples an average of 24k patches. The model is pre-trained on a Linux system with 2 CPUs (AMD EPYC 9654 96- Co-e Processor) and 4 GPUs (NVIDIA Tesla A100 80G) for about 2.8 days.
minibatch and each minibatch contains
We optimize with Adam 31,, updating the model parameters every steps, and the model trains for 750k updates in total. A cyclic scheduler that adopts basic triangular cycle without amplitude scaling utilized to adjust learning rate during pre-training. Specifically, we set the basic learning rate 3 x 10 and the maximum learning rate x 10 then the learning rate steps up (down)

---

for every 8k updates. We apply mixed precision training with FP32 and BF16 to reduce the memory usage in GPUs for acceleration.
3.3 Downstream tasks
As Brant is a foundation model for intracranial recordings, conduct extensive experiments on several downstream tasks, including short- and long-term signal forecasting, frequency- phassee forecasting, imputation and seizure detection, verify the high capacity of our model in modeling intracranial recordings. We conduct each downstream task for Brant on two settings: (I) fine-tune the model with learning rate of 1 10-7; (2) freeze the pre-trained weights. Five random performed to obtain the mean and standard deviation. The detailed setups of these downstream tasks are follows:
Short- and long-term signal forecasting. Predictive observation of the neural signal values is beneficial for the development of warning systems for patients in need of precautionary measures [32 331. Therefore, we adopt short- and long-term term signal forecasting, in which the learned representations fine-tuned to predict future signals with different lengths given a past sequence. The past sequence length is set as 15 patches (90s) and the prediction lengths are set as 2 patches (12s) and 20 patches (120s) in short- and long-term forecasting, respectively. We sample 400 minutes of recordings from the downstream dataset, then randomly split into 320 minutes for fine-tuning and 80 minutes for evaluation. A linear prediction head is used predict the future signals. We adopt MAE and MSE as the performance metrics.
Frequency-phase forecasting, In medicine, it is essential to predict the physical features like frequency and phase of brain signals to provide guidance for some treatments. For example, in therapy that modulates brain activity called transcranial alternating current stimulation (tACS), the stimulation control heavily depends on these knowledge about the target brain activity [4). up the frequency-phan forecasting task in order for these therapies like tACS to be most effectively used in the treatment of brain disorders
Therefore,
35
Given a past sequence, this task is to predict the dominant frequency and phase information (see details in App. B of intracranial signals the future. The past and prediction lengths are set as 15 patches (90s) and 5 patches (30s), respectively. We use the same sampling and split strategy that in the short- and long-term signal forecasting to generate the data for the frequency-phan task. A linear layer is adopted to predict the dominant frequency and phase of the future signals. As for the metrics, following the work by Mansouri et al. (34), we use MAE for dominant frequency forecasting, and phase locking value (PLV) for phase forecasting. The PLV is value between 0 and 1 calculated by the equation:
forecasting
PLV = | Σ (pl(i((()-00)) .(m)|
(4)
where sa and è are the original signals and the forecasted signals, respectively.
Imputation. During brain signal recordings, measurement problems such as artifact contamination or electrode impairment are not easily corrected, thus the neural recordings will be incomplete. Imputation can fill in these contaminated signals so that other medical devices can keep operating under missing values, making use of available data [36]. For the imputation task, we randomly mask the timestamps in each patch with the ratio of 40% and fine-tune the model to predict the missing values. We adopt the same sampling and split strategy to obtain the imputation data as in the short- and long-term signal forecasting. We add linear head make predictions, then apply MAE and MSE as the evaluation metrics to measure the discrepancy between the masked and predicted values.
Seizure detection. As of the most important applications of intracranial recordings, seizure detection task is to evaluate the model ability to distinguish between epileptic seizures and normal waveforms. We sample 250 minutes of recordings from the downstream dataset with about 10% positive (seizure) samples, where 200 minutes are randomly selected for fine-tuning and the remaining 50 minutes for evaluation. An MLP adopted classify the pre-trained representations. The evaluation metrics we use are accuracy, precision, recall, F1 and F, scores. The F-measure is metric defined as the weighted harmonic mean of precision and recall, with the following equation Fg (1+ xprecision xreealt xxxprcision+treaa
5

---

3.4 Baselines
As Brant a pre-training work on neural signals, we extensively compare our model with the advanced self supervisdd or unsupervised pre- vre-training works on all the downstream tasks, including works designed for brain signals: RP [7]. TS
T CPC
[7. BENDR
[8]. MVTS [9] and
BrainBERT [20]: and for general time series: CoST
37 TF-C
38 PatchTST 27 and TS-TCC [39]
Considering that seizure detection an important application scenario of intracranial recordings, we further compare our model with several supervised methods which conduct seizure detection on EEG or iEEG to evaluate the performance of Brant detecting epilepsy. These baselines contain handcrafted feature based methods, including spectral power [28), rhythmicity spectrogram [3] and amplitudeiintegrated EEG [40]; and the SOTA deep learning based method on seizure detection, including SEEG-Net 99]
More details of the baselines are shown in App.
4
Experimental Results
4.1
Main Results
Fig. 3 summarizes the results of all the downstream tasks. As a foundation model for intracranial recordings, Brant achieves consistent SOTA performance on a variety of tasks compared with other baseline models. We discuss more detailed comparisons of each task in the following paragraphs, where in all the tables we mark values ranking the first (v), second (v) and third (*v) in each column.
Brane
Brant compared with Pre-training baselines signals
Sturt. (bura)
Brant compared with Pre-training baselines
Braee (sund
(our model)
Signel Forecasting
SExion NNTS
Signal Forecasting
C:ST
Forecasting (MAE)
Long-term Signal Forecasting (MAE)
Seizure Detaction
Long ferm Signal Forecasting (MAE)
Seizure Detection (F2)
Long-term Signal
Seizure Detaction (F2)
Forecasting (MAEL
3.5
20,00
3:25
0.22
3.25
3.30 Forecasting (MAE)
Frequency Forecasting
Imoutation (MAE)
Frequency Forecasting (MAL)
(MAE)
(MAE)
(MAEL
Phase Forecasting (PLY)
Phase Forecasting (PU/)
Phase Forecasting (PI/)
Figure 3: Performance comparison of our model and other baseline models on all downstream tasks. Model performances are plotted on three radar subfigures for clarity with the same coordinate range.
Table 1: Performance on forecasting tasks.
Task
Short-term signal forecasting
Long-term signal forecasting Frequency-phase forecasting
Model
MAE
MSE
MAE
MSE
ph. PLV
freq. MAE
RP 17
0.7569 20.0016
1.1769 +000003
0.6774
1.0579
0.4929
0.4197 +000022
TS 17
0.6142+a.001
0,877 +:0.076
0.6286
0,977 +
0.5609#0.00m
0.0245 +00000
CPC 17
0.6318 00000012
0,8886+40.008
0..024
0.9604-10000
0.5608
0.1036+00004
BENDR 18
0.6002.20.0114 0.8720200008
0.5948=0.0020
0.8535
0.5685
0.0604100017
MVTS 19
0.6868=20.009 1.1859
0.5867
0.838
1:000014
"0.5843.20.004
0.0320
BrainBERT
20
0..061 0.88571:00099
0.7214
1.4727 1:0.0012
0.5712
0.1065
PatchTST 27
0.5050=o.m10 0.6482-40.022
#0.5792-0.m014
#0.8235 0000006
0.5905=2..mw
0.0267
TS- TCC 39
0.6042:00.012 1.02444100.096
0.5938
0.8506
0.5743
0.00002
0.0061 1.00003
TF-C 388
o..035
2.9441 1:0.0146
1.0683310.005
2.2491 1:0.0016
0.5493
1.549931..000 0.0000
0.1593
CoST 37
0.553 1100011
0.5955
0.8546
0.5728
0.0003
0.1152
Brant-Freeze
#0.5332 *0.7161 1::0111
0.5780=0.0044
0.78699 0007
0.5777=2n.mor:
#0.01 000
Brant
0.4007=2a.m14
0.4626=:00048
0.5693
0.7676:20.005
0.6004
0.0041
Short- and long-term signal forecasting. The results short- and long-term signal forecasting are shown on the left and middle of Tab. respectively. Not only does Brant achieve SOTA performance,

---

but the results of Brant--Frece Freeze are also better than most baselines. Especially on long term forecasting. Brant-Freeze defeats all the baseline methods, showing the ability to capture long-tern dependency of our model. Among the baselines, PStchTST 27 performs well compared to other methods, mainly because also adopts a patching strategy attend longer temporal dependency.
Frequency-phare forecasting. Forecasting physical features like frequency and phase of neural signal essential in some medical techniques (see Sec. (.3). The results of frequency-phan forecasting are shown on the right of Tab. which shows that our model outperforms all the methods, demonstrating that Brant facilitates effective implementation of therapies for some brain disorders.
Imputation. As a mitigation for measurement problems, imputation can fill the incomplete neural signals, allowing other medical devices to continue operating Brant and Brant-Fread achieve the best and the second best results among all the methods, which demonstrates the ability of model in capturing underlying temporal patterns with partially observed neural recordings to imputate the contaminated neural signals.
when data is missing (see
Sec. 3.3). From Tab. 2
Seizure detection. As seizure detection is an important medical application intracranial recordings, we not only compare our model with the pre-tr training based baselines, but also fur- ther select 4 supervised method designed for seizure detection. From Tab. 3 Brant outper- forms all the other methods and Brant-Freeze achieves second best on most of the metrics. BrainBERT (00 achieves the best accuracy, pre- cision and F2 score among all the baseline meth- ods, primarily due to it provides contextualized neural embeddings and combines the informa- tion from time and frequency domains like our model. However, Brant still improves the F2 score by 29.59% over BrainBERT, because our temporal encoder obtains a wider receptive field and the spatial encoder captures the spatial cor- relation across channels, which are both critical in modeling intracranial recordings.
Table Performance on the imputation task.
Task
Imputation
Model
MAE
MSE
RP 17
0.5953-20.008
0.9158200000
TS 17
0,5225
0.7476:00003
CPC 17
0.5663
0.827 =00004
BENDR 18
0.4849=0.000 0.6492.=0.0ss
MVTS
0.5946=0.0009
0.8903
BrainBERT
20
0.6083 0.9790
PatchTST 27
0.4282 0.5506=2n.0022
TS-TCC 39
0.6144 0.000s 1.558820.000
TF-C 38
1.1876
3.0179
CoST 37
#0.2652
#0.1638
Fant-Frecze
0.1963=00.00
0.0865
Brant
0.1912:2000m:
0.0814:0.000
Table 3: Performance the seizure detection task.
Task
Seizure detection
Model
Accuracy
Precision
Recall
F1
F2
Spectral Power
28
89.01 +0.12
72.77 1:98
3..07=1.23
48.2 08.37
40. 1.02
Rhythmicity Spectrogram
88.58
70.31 1:00
3..07
48.55
39.1011.0P
cmplitude-integnann EEG
88. 1:0.9
71.21 1:201
35.73
47.58
39.68 1.112
SEEG-Net
88.97 4:014
70.45 #:2.0
38.44
*49.74-10.00
42,28
RP 17
67.65
18.624244
34.71
24.24-1187
19.59
TS
17
85.90
54.68 23.5
31.66 1.97
40.10=081
34.57=1166
CPC 17
84.72++0.40
48.3 +:280
36.03++0.p
41.28
37.96++144
BENDR 18
88. 14-448
71.49 #3.42
29. +:203
42,10
33.77++181
MVTS
88.35 +-222
69.43 +3319
32.03
43.84++196
35, 15.90+194
BrainBERT
20
889,5940.12
77.86
39 39.28
52,21 3:039
#43.6010..
PatchTST
27
$1.1344022
38.71 +3.64
21.16 2
27.37++022
23,27++122
TS-TCC
88. +::29
89.61 #:208
23.81 1.50
37.62++040
27. 11.11
TF-C 38
75.05 4366
18.75
19.09 2.002
18.92+041
19.02
CoST 37
81.05 $1.05
30.98
#43.19
36.08
4..0
Brant-Freeze
90.53
#77.26
48.92:10.72
59. 59.87.10.
52.87.1944
Brant
91.17+4015
79.25
52.74:11.43
63.29
56.50+1.08

---

4.2 Model Analysis.
Low-resource labeled data evaluation. In medical scenarios, collecting labeled data for even small experiments huge investment. To demonstrate the practical value of work, evaluate Brant on seizure detection where the amount of labeled data is limited. Specifically, the pre-trained model is fine-tuned on 200 minutes, 60 minutes and 20 minutes of labeled data that sampled from the downstream dataset (Sec. (.... respectively. After fine-tuning, models are evaluated on the same 50 minutes of labeled data which also sampled from the downstream dataset but aon--oelappeed with the fine-tuning data.
From Tab. the performances of supervised methods decrease rapidly compared to the self- supervised or unsupervised methods, showing that the representations learned on unlabeled data improve low-resource settings. Among the pre-training works, model maintains the most stable performance on 20-minute labeled data. Note that the F2 score of our model on 20-minute labeled data (51, 1.03%) is even higher than the F2 score of the best baseline on 200-minute labeled data (43.60% demonstrating that Brant fully captures the patterns and semantic information from the intracranial data during pre-training and adapts downstream tasks more easily.
Table 4: Low-resource labeled data evaluation on the seizure detection task and the relative decrease of the F2 score on 60-minute and 20-minute labeled data versus 200-minute labeled data.
200 minutes
minutes
minutes
Model
F2
F2
Decrease
F2
Decrease
Spectral Power 28
40.12
25,95
35.32%
5,06
87.39%
Rhythmicity Spectrogram B
39.10=1.0
21.87+233
44.07%
2.81 4:037
92.81%
Amplitude-interrate EEG 40
39.68 1.12
18.42
53.58%
2.05
94.84%
SEEG-Net
(
*42.28
35.54
15.94%
12.7612.15
69.82%
RP 17
29.59
27.62:1200
*6.66%
25.05
15.34%
TS
[17
34.57
30.15
12.79%
29.61
*14.35%
CPC 17
37.96
30.55
19.52%
29.57 1:3.4
22.10%
BENDR 18
33.77 1.81
25.37+3.12
24.87%
22.18
34.32%
MVTS 19
35.90
26.6221.11
25.85%
24.39
32.06%
BrainBERT
20
43.60
41.93:12.0
3.84%
36.354322
16.63%
PatchTST
23.27 1.22
18.02:12.23
22.55%
17.07
26.64%
TS TSC 39
1..11 1:11
25.35
9.17%
20.36
27.05%
TF-C 38
19,02
15,97 1.23
16.04%
13.66
28. 18%
CoST 37
40.03
#39.18
2.12%
36. 10
9.82%
Brant
56.50
52.30+2204
7.43%
51.03++2.4
9.68%
Representation analysis. As classification task can verify the model capacity in high-level Brant and best pre-
representation learning [41] we visualize the pre-trained representations training based methods seizure detection task using t-SNE (shown in Fig. 4). Compared to other methods, the representations of seizure and normal signals learned from Brant are separated more clearly during pre- training, which intuitively illustrates our SOTA performance low-resource label settings. Furthermore, the results explain the good performance of Frent-Freeze on seizure detection.
Brant tour modr)
CPC
Bra BBEIT
CoST
Figure 4: Pre-trained representation visualization of Brant and 3 best baselines seizure detection task. The representations of seizure and normal samples are plotted in red and yellow (+).

---

Model scale analysis. To explore the effect of model size on performance, we additionally pre- train three variants of Brant with smaller size: Brant-tiny, Brant-small and Brant-medium (detailed configurations are in App. F on the same dataset as Brant. All models are pre-trained with an Adam optimizer and cyelic scheduler. Brant-medium adopts the same learning rate as Brant (Sec. 3.2); for Brant-tiny and Brant-small, the basic and maximum learning rates are 6 x 10-6 and 3 x 10
We evaluate these smaller size variants on all the downstream tasks (results shown in Fig. 5 As the model size increases, the performances on the downstream tasks show an overall upward trend, where Brant ahead of the other three variants in all metrics. Also, the decrease in the standard deviation indicates more stable performance for larger models. On such a huge neural dataset, larger model with higher capacity results in better generalization ability wide range of downstream tasks.
Short- & long term Signal Forecasting
Frequency phase forecasting
Imputation
Seicure Detection
0.80
0.800
2.75
-0.009
0.70
00000 0.225
2.65
0.5
3.00
0.590
0.007
0.55
0.585
0.150
0.4
0,005
0.125
0.580
0.100
0,054
0.075
100
250
500
70
100
250
500
70
100
250
500
1000
250
500
Parameters M
Parameters M
Parameters/ M
Parametors/ M
Figure 5: Performance on all the downstream tasks across Brants with different model size: Brant- tiny (70M), Brant-small (100M), Brant-medium (250M) and Brant (500M).
Ablation study. We perform ablation experiments to assess the effectiveness of each individual component in our model. Specifically, we remove the following components from our model to examine their respective effects on performance: the temporal encoder (Brant-temporal), the spatial encoder (Brant-spatial), and the frequency encoding (Brant-frequency))
The experimental results of ablation study are given in Tab. [nd Tab. 6 The results indicate that Brant outperforms other model variants across all metrics, providing evidence for the contribution of each component in our model. Among the different variants, Brant-temporal exhibits the most substantial decrease in performance compared to the full Brant model, emphasizing the significance of long-term dependency in modeling brain signals. The performance degradation observed in Brant- spatial demonstrates the role of considering channel spatial correlation in learning representations of intracranial recordings. Furthermore, the decline extracting information from both the time and frequency domains proves beneficial for effectively modeling brain signals.
freuuncyy suggests that simultaneously
Table 5: Results of ablation experiments on forecasting tasks.
Task
Short-term signal forecasting Long term signal forecasting Frequency phase forecating
Model
MAE
MSE
MAE
MSE
ph. PLV
freq. MAE
Brant-temporal
0,5356
0.7021
0,5777
0.7876
0,5800
0.0094
Brant-spatial
0.5403
0.7521
0.5827
0.7990
0.5780
0.0092
Brant-frequency
0.4762
0.5676
0.5726
0.7741
0.5645
0.0143
Brant
0.4007
0.4626
0.5693
0.7676
0.6004
0.0041
Table 6: Results of ablation experiments on the imputation and seizure detection tasks.
Task
Imputation
Seizure detection
Model
MAE
MSE
Acc.
Prec.
Rec.
F1
F2
Brant-temporal
0.2798
0.1797
88.56
67.39
40.25
50.40
43.78
spatial
0.2770
0.1741
89.93
84.77
36.88
51.40
41.57
Brant- frequenyy
0.2495
0.1406
90.16
78.41
44.03
56.39
48.26
Brant
0.1912
0.0814
91.17
79.25
52.74
63.29
56.50

---

Generalization ability analysis. To further verify the generalization ability of Brant on more subjects with more heterogeneity. we evaluated the model on data of 31 unseen subjects from two public datasets named MAYO and FNUSA [2]. More details about this study are shown in App. G
5 Related Work
Intracranial recordings modeling, With the development of iEEG technique to obtain deep brain information from both cortical and subcortical structures, modeling intracranial recordings has attracted the attention of many researchers. Wang ct al. [9] extract time domain features of iEEG signals from multiple receptive fields by utilizing multiscale CNN and LSTM. Guo al. 10 introduce hypergraph learning approach to model iEEG signals by detecting high frequency oscillations. Yu and Hu [1]] propose EDANN to learn domain-invariam representations of iEEG data from multiple subjects by domain adversarial training. Jiang et al. [8] develop a novel method using short-time resting-state connectivity identify the seizure onset zone (SOZ) from interictal EEG signals. Wang et al. [12 conduct cross-subject iEEG seizure detection based on adaptive feature fusion of brain network features and single-channel features. However, these works are all supervised that relies heavily on labeled data, which is often difficult and expensive to obtain at scale. Furthermore, most works for intracranial data modeling process each channel independently, ignoring the spatial correlation. Although Chen et al. (23) propose BrainNet which contains a graph diffusion component which measures the brain wave diffusion among channels, their work only focus on the information in time domain and is limited to an individual subject.
has shown good performance in many scenarios (e.g., weather, traffic flow, exchange rates), including some works designed for brain signals. Banville al. (17 learn representations from EEG signals by self- supervised temporal context prediction task, revealing clear latent structures related to physiological and clinical phenomena. Kostas et al. (18] address the problem of limited labeled data on EEG by using a contrastive self-supervised learning task to pre-train model named BENDR, which then fine-tuned for downstream tasks. Potter et al. [19] propose an unsupervised approach to model EEG signals using a transformer- based model with a signal reconstruction task. Cai et al. 43 propose to study the self-supervised learning framework for brain signals that can be applied to pre-train either scalp or intracranial EEG data. Wang et al. (20 propose traiirain work named BrainBERT for intracranial recordings and conduct experiments on iEEG data. They adopt time-frequency representations but ignore the spatial correlation between channels, which is critial in modeling intracranial recordings (additional commentary on differences between our work and BrainBERT is in App. D
Pre-training on brain signals. Pre-training on time series
37
38
27
39
6 Conclusion
We propose a task-agnostic foundation model, Brant, which learns powerful representations of intracranial recordings. Brant is the largest pre-training model on brain signals, whose design (1) attends a long temporal dependency; (2) captures the spatial correlation between channels; and (3) extracts information from both time and frequency domains. Experimentally, Brant achieves consistent SOTA performance on various downstream tasks w.r.t. medical scenarios. Further analysis shows the effectiveness and benefit of a large-scale pre-trained model in the field of medicine. Brant is an off-the-shelf model with its code and weights, which significantly alleviates the issue of sample and label efficiency and can directly participate in other medical research and treatment.
Limitations and future works. By pre- pre-training on a large amount of intracranial data, Brant contains over 500M parameters, far more than other existing works on brain signals. However, compared to other fields such as CV and NLP in which the models can reach billions of parameters and achieve good performance further improvement of our work. In the future, by scaling up our dataset, the scale of our model can be further expanded capture higher-level semantic information from neural data, revealing more complicated brain activities and dynamics, to provide assistance for more healthcare applications.
variety of tasks by zero-shot learning, there is still potential for
10

---

Acknowledgments
This work is supported by NSFC (No.62176233), the National Key Research and Development Project of China (N0.2018AA401019000 and the Fundamental Research Funds for the Central Universities.
References
[1] Alessandra Del Felice, Leonora Castiglia, Emanuela Formaggio, Manuela Cattelan, Bruno Scarpa, Paolo Manganotti, Elena Tenconi, and Stefano Masiero. Personalized transcranial alternating current stimulation (tacs) and physical therapy to treat motor and cognitive symptoms parkinson's disease: a randomized cross-over trial. Neurolmage: Clinical, 22:101768, 2019.
[2] Florian Mormann, Ralph G Andrzejak, Christian E Elger, and Klaus Lehnertz. Seizure predic- tion: the long and winding road. Brain, 130:314-333, 2007.
[3] Palak Handa and Nidhi Goel. Epileptic seizure detection using rhythmicity spectrogram and cross-patient
In 2021 8th International Conference on Signal Processing and Integrated Networks (SPIN), pages 898-902, 2021.
[4] Scott B. Wilson, Mark L. Scheuer, Ronald G. Emerson, and Andrew J. Gabor. Seizure detection: evaluation of the Reveal algorithm. Clinical Neurophysiology, 115:2280-2298 2291,, 2004.
[5] Zisheng Zhang and Keshab K. Parhi. Low-Complexity Seizure Prediction From iEEG/SEEG Using Spectral Power and Ratios of Spectral Power. IEEE transactions on biomedical circuits and systems, 10:693-706, 2016.
[6] Badreddine Mandhouj. Mohamed Ali Cherni, and Mounir Sayadi. An automated classification of EEG signals based on spectrogram and CNN for epilepsy diagnosis. Analog Integrated Circuits and Signal Processing, 108:101-110, 2021.
[7] Yudhajit Das. Hanli Liu, Fenghua Tian, Srinivas Kota, Rong Zhang, and Lina F. Chalak. Rigor of Neurovascular Coupling (NVC) Assessment in Newborns Using Different Amplitude EEG Algorithms. Scientific Reports, 10:9183, 2020.
[8] Haiteng Jiang, Vasileios Kokkinos, Shuai Ye, Alexandra Urban, Anto Bagić, Mark Richardson, and Bin He. Interictal seeg resting-state connectivity localizes the seizure onset zone and predicts seizure outcome. Advanced Science, 9:2200887, 2022.
[9] Yiping Wang, Yanfeng Yang, Gongpeng Cao, Jinjie Guo, Penghu Wei, Tao Feng, Yang Dai, Jinguo Huang, Guixia Kang, and Guoguang Zhao. Seeg-net: An explainable and deep learning- based subject pathological activity detection method for drug-resistant epilepsy. Comput- ers in Biology and Medicine, 148:105705 105703, 2022.
[10] Jiayang Guo, Hailong Li, Xiaoshuai Sun, Lei Qi, Hui Qiao, Yijie Pan, Jing Xiang, and Rongrong Ji. Detecting high frequency oscillations for steroellecteenephhllggaaa in epilepsy via hypergraph learning. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 29:587-596, 2021.
[11] Hao Yu and Menggi Hu. Epilepsy seeg data classification based on domain adversarial learning. 2021.
IEEE Access,
[12] Ying Wang. Zhengfeng Bao, Yiping Wang, Yanfeng Yang, Tao Feng, and Guixia Kang. A cross-patient seeg epileptic signal detection method based on adaptive feature fusion of brain network features and single-channel features. Proceedings of the 2022 9th International Conference Biomedical and Bioinformatics Engineering, pages 165-172, 2022.
[13] Bancaud. Surgery of Epilepsy Based on Stereotactic Investigations- the Plan of the SEEG Investigation. In F. John Gillingham, Jan Gybels, Edward Hitchcock, Gian Franco Rossi, and Gábor Szikla, editors, Advances in Stereotactic and Functional Neurosurgery 4, pages 25-34, 1980.
11

---

[14] Greg A. Worrell, Andrew B. Gardner, S. Matt Stead, Sanqing Hu, Steve Goerss, Gregory J. Cascino, Fredric B. Meyer, Richard Marsh, and Brian Litt. frequenyy oscillations in human temporal lobe: simultaneous microwire and clinical macroelectrode recordings. Brain, 131:928 937, 2008.
[15] M Kalaivani, V Kalaivani, and V Anusuya Devi. Analysis of eeg signal for the detection of brain abnormalities. International Journal Computer Applications@ year, 2014.
[16] Santiago Morales and Maureen Bowers. Time-frequency analysis methods and their application in developmental ceg data. Developmental Cognitive Neuroscience, 54:101067, 2022.
[17] Hubert Banville, Omar Chehab, Aapo Hyvärinen, Denis-Alexander Engemann, and Alexandre Gramfort. Uncovering the structure of clinical eeg signals with self-supervise learning. Journal of Neural Engineering, 18:046020, 2021.
[18] Demetres Kostas, Stephane Aroca-Ouellett and Frank Rudzicz. Bendr: contrastive self-supervised learning task to learn from massive amounts of eeg data. Frontiers in Human Neuroscience, 15:653659, 2021.
using transformers and
[19] ilkay Yildız Potter, George Zerveas, Carsten Eickhoff, and Dominique Duncan. Unsuper- vised multivariate time-series transformers for seizure identification on eeg. arXiv preprint urXiv:2301.03470 2023.
[20] Christopher Wang, Vighnesh Subramaniam, Adam Uri Yaari, Gabriel Kreiman, Boris Katz, Ignacio Cases, and Andrei Barbu. Brainbert: Self-supervised representation learning for intracranial recordings, 2023.
[21] Yi Ding, Neethu Robinson, Qiuhao Zeng, and Cuntai Guan. Lggnet: Learning from local- global-graph representations for brain-computer interface. CoRR, abs/2105.02786, 2021.
[22] Neeraj Wagh and Yogatheesan Varatharajah. Beg-genn: Augmenting electroenceehallgramm based neurological disease diagnosis using domain-guidee guided graph convolutional neural network, 2020.
[23] Junru Chen, Yang Yang. Tao Yu, Yingying Fan, Xiaolong Mo, and Carl Yang. Brainnet: Epileptic wave detection from seeg with hierarchical graph diffusion learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, page 2741-2751, 2022.
[24] Eunjin Jeon, Wonjun Ko, Jee Seok Yoon, and Heung-Il Suk. Mutual information-driven subject- invariant and class-relevant deep representation learning in bei IEEE Transactions on Neural Networks and Learning Systems, 2021.
[25] Hubert J. Banville, Isabela Albuquerque, Aapo Hyvärinen, Graeme Moffat, Denis-Alexander Engemann, and Alexandre Gramfort. Self-supervised representation learning from electroen- cephalography signals. CoRR, htt///911055199 2019.
[26] Siyi Tang, Jared Dunnmon, Khaled Kamal Saab, Xuan Zhang, Qianying Huang, Florian Dubost, Daniel Rubin, and Christopher Lee-Messer. Self-supervised graph neural networks for improved edettreencephalograpiie seizure analysis. In International Conference on Learning Representations, 2022.
[27] Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers, 2023.
[28] Zisheng Zhang and Keshab K. Parhi. Low-complexity seizure prediction from iceg/seeg using spectral power and ratios of spectral power. IEEE Transactions on Biomedical Circuits and Systems, 10:693-706. 2016.
[29] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021.
[30] Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
12

---

[31] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
[32] Raluca Maria Aileni, Sever Pasca, and Adriana Florescu. Eeg-brain activity monitoring and predictive analysis signals using artificial neural networks. Sensors, 20(12))3346, 2020.
[33] DK Thara, BG Premasudha, TV Murthy, and Syed Ahmad Chan Bukhari. Eeg forecasting with univariate and multivariate time series using windowing and baseline method. International Journal of E-Health and Medical Communications (LJEHMC), 13:1-13, 2022.
[34] Farrokh Mansouri, Katharine Dunlop, Peter Giacobbe, Jonathan Downar, and José 7ariffa. A fast eeg forecasting algorithm for phase-locked transcranial electrical stimulation of the human brain. Frontiers in Neuroscience, 11, 2017.
[35] Anke Karabanov, Axel Thielscher, and Hartwig Roman Siebner. Transcranial brain stimulation: closing the loop between brain and stimulation. Current Opinion in Neurology, 29(4):997-4) 2016.
[36] Atsunori Kanemura, Yuhsen Cheng, Takumi Kaneko, Kento Nozawa, and Shuichi Fukunaga. Imputing missing values in eeg with multivariate autoregressive models. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 2639-2642, 2018.
[37] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. CoRR, abs/2202.01575. 2022.
[38] Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre- pre-training for time series via freeufeey consistency, 2022.
[39] Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. CoRR, abs/2106. 14111 12, 2021.
[40] Maliheh Kadivar, Elahe Movahedi Moghadam, Reza Shervin Badv, Raziye Sangsari, and Maryam Saeedy. A comparison of conventional electroencephalography with amplitude- integrated eeg in detection of neonatal seizures. Medical Devices: Evidence and Research, pages 489-496, 2019.
[41] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long, Timesnet: Temporal 2d-variation modeling for general time series analysis, 2023.
[42] Petr Nejedly, Vaclav Kremen, Vladimir Sladky, Jan Cimbalnik, Petr Klimes, Filip Plesinger, Filip Mivalt, Vojtech Travnicek, Ivo Viscor, Martin Pail, al. Multicenter intracranial eeg dataset for classification of graphoelements and artifactual signals. Scientific data, 7:179, 2020.
[43] Donghong Cai, Junru Chen, Yang Yang. Teng Liu, and Yafeng Li. Mbrain: A multi cuati- self-supervised learning framework for brain signals, 2023.
13

---

Ethics Statement
The data collection and experiments conducted in our work have been approved by the Institutional Review Board (IRB) and passed ethical review. All participants have signed informed consent forms.
A Intuition of Using PSD as Weights
PSD displays the spectal power distribution in different frequencies of brain signals. The power distribution in different frequency bands is associated with different brain functional states. For example, during wakefulness, and / waves are more active; during sleep, 8 and 0 waves are more prominent; and ? waves will become significant during epileptic seizures. Therefore, in medical research, the power distribution in different frequency bands is often used as feature reflecting the functional state of the brain.
In our work, we treat the power distribution as the weights of learnable representations fú, - 1,.., in 8 standard frequency bands. The intuition is that the frequency bands with higher power more representative of the brain's physiological state, and should be given more weight in their representations. In this way, the model can focus more the frequency bands where the main power of the brain signals resides.
B Details of the Dominant Frequency and Phase
Dominant frequency. Dominant frequency defined as the frequency related with the greatest power amplitude in the brain signal. Dominant frequency be calculated from Fourier Transform spectrum. Mathematically, given brain signal series s(t), t E (to. t1). the amplitude A(w) on frequency w is:
-hte-wut A(w) = [F(w) = =If s(t)c nut
(5)
and the dominant frequency wp is:
WD arg max A(w)
(6)
Phase. The instantaneous phase of a complex-valued function signal s(1) is the real-valued function:
(t()) angle(s(t)).
(7)
where angle(-) is the angle between the positive real axis and the direction of complex number s(t).
C Details of Baselines
Firstly, we compare model to the existing self-supervised or unsupervised pre-training works on brain signals. The Details of these baseline models are given here:
RP, TS, CPC: Three self-supervised learning approaches with three self-supervised learning pretext tasks of relative positioning (RP), temporal shuffling (TS) and contrastive predictive coding (CPC) to learn representations of EEG signals based on StagerNet.
BENDR: A self-supervised training model that learn compressed representations to model completely novel raw EEG sequences recorded with differing hardware, and different subjects performing different tasks.
MVTS: An unsupervised transformer- anomaly detection approach to train an autoen- coder with masking strategy on non-seizure signals.
BrainBERT: A reusable transformer for intracranial field potential recordings enables classi- fying complex concepts and decoding neural data.
Secondly, we compare our model with the advanced self-supervised or unsupervised pre-training works designed for general time series, including:
14

---

CoST: A time series representation learning framework for long sequence time series forecasting, which applies contrastive learning methods learn disentangled seasonal-trend representations.
TF-C: A decomposable pre-training model, where the self-supervised signal is provided by the distance between time and frequency components, each individually trained by contrastive estimation.
PatchTST: An efficient design of Transformer- models for multivariate time series forecasting and sele-ssue representation learning, based on the segmentation of time series into subseries-level patches.
TS-TCC: An unsupervised time series representation learning framework to learn robust temporal and discriminative representations by designing a tough cross-view prediction task and contextual contrasting module.
Furthermore, as seizure detection is the most important application scenario of intracranial recordings, to evaluate Brant's ability in seizure detection, compare model with some seizure detection methods on EEG or iEEG. These baselines include handcrafted feature based methods and a deep learning based method:
Spectral Power: The spectral power in a particular frequency band means the power of signal in that frequency band. The rhythmic activity brain signals is typically described in terms of the spectral power or its ratio on some standard frequency bands.
Rhythmicity Spectrogram: This feature displays a density spectral array of frequency and power characteristics of brain signals, providing graphical depiction of the amplitude of primary rhythmic EEG components present in four frequency bands.
Amplitude-inteps EEG: A trace of electrical activity displayed semilogarithmic graph of peak-to-peak amplitude over time, which can be used to monitor and diagnose seizure activity.
SEEG-Net: A model that can address the problems of sample imbalance, cross-subject domain shift, and poor interpretability and realizes high-sensitivity SEEG pathological activity detection. Since the source code of SEEG-Net is not released, we re-implement it to conduct experiments.
D Detailed Differences between Brant and BrainBERT
BrainBERT is also pre-training work conducted on intracranial recordings, but our work is different with them in the following aspects.
Data and model scale. Pre-training often requires large amount of data by which the model fully learns the distributions, and the pre-trained model usually contains huge quantity of parameters to fit the data. According to the paper of BrainBERT and the model weights released by its authors (20), the total amount of data and parameters BrainBERT 43.7 hours and 43.18M, respectively. Brant is pre-trained on 2528 hours of data and the scale of Brant reaches to 505. 655.69M parameters.
Method. As discussed in the main paper, Brant considers long-term dependency, channel corre- lation, time and frequency domains when modeling intracranial recordings. BrainBERT provides contextualized neural embeddings and adopts time-frequency representations, but ignores the channel correlation which reflects the propagation of brain waves.
Downstream tasks. BrainBERT is evaluated on four classification tasks related with the audio listened from the subjects, which mainly focus on a particular stimulus, i.e., passive movie viewing. Brant is evaluated on various downstream tasks, including signal forecasting, frequeney-phase forecasting, imputation and seizure detection, all of which are medically valuable (described in Sec. 3.3
E Details of the Datasets
Pre-training dataset. The pre-training dataset contains 1.0ITB data with sampling rate of 1000Hz. The details of the pre-training dataset given in Tab.
15

---

Downstream dataset. The downstream dataset contains 29.39 GB data with sampling rate of 1000Hz. The details of the downstream dataset with seizure labels are given Tab. g For each downstream task, we sample small subset from the downstream dataset for fine-tuning and evaluation. The sampling and split strategies are described in the setup of downstream tasks in Sec. 3.3
Table 7:
Details of the pre-training dataset.
Subject id
Recording hours
#Electrodes
#Channels
#Sample points
235.34
10
124
105,055,776,000
82.39
52
15,423,408. 40,,0000,00
3
393.09
10
120
169, 1199814,88 11,880,0000
137.45
10
133
65,81 ..6,00000
214.22
8
116
89, 89,458.272,00
386.70
8
101
140,604, 00.604,120,0000 120,000
111.55
67
26,905 860,000
8
207.42
5
47
35,095,464,000 464,000
759.80
11
134
366,527,520,000
Table 8:
Details of the downstream dataset.
Subject
Recording hours
#Seizure timestamps
#Normal timestamps
3.46
3,298,000
383,154,000
4.63
539,000
216,288,00
5.23
2,129,000
563,091,000
6.00
3,427,000
622.977. ,0000
E
6.00
1,311,000
544,090,000
F
6.00
496,000
361,282,000
G
5.92
7,278,000 0000
243,255,000 ,000
5.98
3,330,000
717,831,000
F
Details of Brant with Different Scales
To explore the effect of model scales on performance, we additionally pre-train three variants of Brant with smaller size: Brant-tiny, Brant-small and Brant-medium on the same dataset. Detailed configurations of Brant and its variants are shown in Tab. 9
Table 9: Configurations of different scales of Brant.
Config
Temporal/Spatial Encoder Layer
Model Dimension
Inner Dimension
Parameter Number
Model
Brant-tiny
8/4
768
2048
68.49M
Brant-small
8/4
1024
2048
103.90M
Brant-medium
12/5
1280
3072
249.22M
Brant
12/5
2048
3072
505.68M
G Details of the Generalization Ability Analysis
As supplement the experiments on downstream dataset in the main text, further verify the generalization ability of Brant on more subjects with more heterogeneity, we evaluated the model on data of 31 unseen subjects from two public datasets named MAYO and FNUSA 42 The MAYO dataset includes intracranial neural signals of 18 subjects collected from Mayo Clinic (Rochester, Minnesota, United States of America). The FNUSA dataset comprises intracranial data of 13 subjects recorded from St. Anne's 's University Hospital (Brno Czech Republic).
16

---

Table 10: Performance on the short- and long-term forecasting tasks on two public datasets.
Short-term signal forecasting
Long-term signal forecasting
Task & Dataset
MAYO
FNUSA
MAYO
FNUSA
Model
MAE
MSE
MAE
MSE
МAE
MSE
MAE
MSE
RP
1.3881
3.0642
1.3786
3.0317
1,4025
3.1277
1.3839
3.0562
TS
0.9464
1.4572
0.9505
1.4846
0.8318
1.1138
0.8353
1.1213
CPC
0.8009
1.0311
0.7922
1.0306
0.7979
1.0254
0.7898
1.0249
BENDR
0.9673
1.4950
0.9507
1.4585
1.0943
2.4782
1.0921
2.4325
MVTS
1.1027
1.7043
1.0838
1.6627
1.2475
2.8251
1.2450
2.7731
BrainBERT
#0.7971
1.0110
0.7829
0.9971
90.7943
#1.0183
*0.7872
#1.0002
PatchTST
0.7969
*1.0227
*0.7879
..0237
0.7479
0.9815
0.7498
1.1833
TSTCC
1.4434
2.7430
1.4532
3.0083
1.7217
2.7587
1.7246
2.8697
TFC
2.1544
4.0941
2.1689
4.4900
2.5697
4.1174
2.5741
4.2831
CoST
0.8812
1.2783
0.8935
1.2895
0.7996
1.0299
0.8094
1.0582
Brant
0.6860
0.8064
0.6915
0.8249
0.7329
0.8856
0.7374
0.9118
Table 11: Performance on the frequency- -hasee forecasting task and the imputation task on two public datasets.
Frequency phase forecasting
Imputation
Task & Dataset
MAYO
FNUSA
MAYO
FNUSA
Model
ph. PLV
Ireq. MAE
ph. PLV
freq.
MAE
MAE
MSE
MAE
MSE
RP
0.2194
1.0222
0.2232
1.0225
0.7025
0.8340
0.7079
0.8556
TS
0.5064
0.0501
0.5076
0.0517
0.7381
0.8935
0.7427
0.9215
CPC
*0.5601
0.1183
*0.5616
0.1119
0.7725
0.9623
0.7737
0.9796
BENDR
0.5459
0.0638
0.5486
0.0639
0.8906
1.0880
0.9001
1.1510
MVTS
0.5389
0.0672
0.5397
0.0674
0.8097
0.9891
0.8187
1.0463
BrainBERT
0.5075
0.0355
0.5091
0.0356
0.7908
1.0045
0.7812
1.0039
PatchTST
0.5627
*0.0455
0.5640
*0,0470
0.4934
0.4753
0.5073
0.4896
TSTCC
0.1285
0.2756
0.1288
0.2752
0.7411
0.9201
0.7451
0.9488
TFC
0.0714
0.5367
0.0712
0.5366
0.7640
0.9486
0.7681
0.9781
CoST
0.5279
0.0482
0.5292
0.0484
*0.4964
*0.4783
*0.5088
#0,4914
Brant
0.5824
0.0304
0.5814
0.0305
0.4887
0,4622
0.5020
0.4835
Table 12:
Performance on the seizure detection task on two public datasets.
Seizure detection
Task & Dataset
MAYO
FNUSA
Model
Acc.
Prec.
Rec.
Fl
F2
Acc.
Prec.
Rec.
Fl
F2
Spec Power
58.79
38.77
44.27
41.34
43.05
58.08
38.48
46.42
42.08
44.58
Rhythm Spec
58.06
31.23
#45.06
36.89
41.39
56.27
30.53
47.64
37.21
42.84
aEEG
58.36
31.19
43.98
36.50
40.65
57.83
36.52
38.70
37.58
38.25
SEEG-Net
57.92
38.38
46.72
*42.14
44.77
58.21
39.12
"49.29
*43.62
+46.85
RP
86.51
46.19
20.29
28.20
22.86
70.19
59.29
19.88
29.78
22.93
TS
72.55
48.72
17.72
25.98
20.30
69.00
52.88
22.64
31.70
25.56
CPC
*87.29
52.90
24.01
33.03
26.96
68.34
50.40
24.98
33.40
27.78
BENDR
80.95
23.95
21.11
22.44
21.62
62.28
5.70
34.04
9.76
17.06
MVTS
69.19
53,30
24.99
34.03
27.96
67.11
49.66
19.68
28.19
22.38
BrainBERT
85.39
44.13
44.92
44.52
*44.76
68.31
49.50
47.12
48.52
PatchTST
87.39
53.02
29.42
37.84
32.29
*69.94
#55.81
26.08
35.55
29.19
TSTCC
77.26
10.47
9.16
9.77
9,40
63.83
24.40
11.36
15.50
12.71
TFC
82.09
11.41
4.91
6.87
5.54
68.71
29.36
5.05
8.62
6.05
CoST
87.39
"53.02
29.42
37.84
32.29
65.92
36.34
33.65
34.95
34.16
Brant
89,40
58.78
70.80
64.23
68.02
83.51
83.60
54.18
65.75
58.28
17

---

The results of generalization study are given in Tab. TO Tab. and Tab. 12 Compared to other baseline models, Brant holds the consistent SOTA performance on these downstream tasks on the unseen 31 heterogeneous subject data collected from different hospitals in different countries, highlighting the generalization ability of our model.
18


Reviews:
[{"review_id": "SaBZeAcZFI", "reviewer": "Reviewer_5spB", "summary": "This manuscript trains a foundation model for intracranial signals based upon a very large electrophysiological dataset.  The methods are based upon a transformer architecture with adaptations to account for the electrode structure and important frequency properties in electrophysiological systems.  The manuscript present experiments compared to many baselines on the tasks, include forecasting, imputation, and seizure detection.\n\nUpdate after rebuttal:\nMost of my concerns have been addressed on the extensive rebuttal, and I have therefore adjusted my score.  Some of the weaknesses remain, such as the limited demonstration of downstream tasks, which is limited to seizure detection, and the original method for seizure detection not using split subjects.  In the final version, these limitations should be noted.  Additionally, the method for calculating uncertainty should be set to report inter-subject variability, and the approach used should be included in the revised manuscript.", "questions": "Is the performance improved due to larger and better matched training data, or because the method is inherently better?\n\nOnce you account for the fact that there are only 8 individuals in the test set, how strong is the evidence of improvement?\n\n", "limitations": "The limitations were not properly addressed; please see weaknesses.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This type of electrophysiological signals are of high importance in many fields, including several types of electrophysiological signals.  This manuscript could contribution to an exciting line of work building foundation models that are capable of enhancing neuroscience, neurology, neurosurgery, and psychiatry.  The need for foundation models in this area is huge, as studies are frequently limited by relatively small sample sizes, usually insufficient to capture complex patterns appearing in the brain.\n\nThe ideas are presented in a straightforward way, and the application is clear.  \n\nIn my view, the primary contribution is simply to run these algorithms at such a large scale, which is critically necessary to start building real foundation models. \n\nI was impressed by the number of comparison algorithms that were run.", "weaknesses": "While I am enthused by the ideas put forth in this manuscript, I view several claims in this manuscript as exaggerated or unsupported.  First, it is necessary to note that while the data scale is huge, the number of individuals is not.  The training data is made up of $N_{train}=9$ individuals, and the test set is made up of $N_{test}=8$ individuals.  As the primary source of heterogeneity in many tasks is the differences between individuals, it is important to assess the variance and performance over individuals rather than windows (e.g., see [1]).  The reported uncertainties seem entirely too small to be over 8 individuals, and those metrics need to be recalculated to more accurately reflect the performance and evidence for the method.\n\nIn a similar vein, it is typical in seizure detection to separate out individuals for the training and test set, whereas this manuscript uses data from the same individuals, as described in lines 198-200.  This overestimates performance.  Notably, the mentioned methods do cross-subject validation (e.g. your reference 9).\n\nOne of the claims is that this method is much improved over competing methods, but the training data for methods was different.  As mentioned in Supplemental Section C, BrainBERT was pretrained on a tiny fraction of the data that BPT was trained on.  As such, it is difficult to assess whether the performance improvement was due to greater amount of training data (as well as better matched training data, as inter-device comparisons can be challenging).  This should be clarified by training methods on all the same data and with similar schemes.  Likewise, the other comparisons were not trained on the same data, making it difficult to assess whether the performance improvement is due to methodological improvement or simply better matched training data (e.g., EEG-> intracranial recordings is non-trivial).\n\nThe claims in the manuscript is that this method is shown to generalize well to several medical scenarios, but I do not see evidence to support that.  The only medical scenario that I see is on seizure, whereas the other tasks are not directly medical scenarios.  Additionally, there are several limitations of this claim: (1) trained on only a single hospital, whereas hospital and device differences can significantly shift signals; (2) total number of individuals in the data set is very small; and (3) the comparisons are not especially clear since they were trained on alternative datasets.  I would encourage the authors to precisely evaluate what evidence of generalization they have provided.\n\n\n[1] Vu, Mai-Anh T., et al. \"A shared vision for machine learning in neuroscience.\" Journal of Neuroscience 38.7 (2018): 1601-1607.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive and detailed comments. We apologize for the imprecise claims in certain parts of our manuscript and misunderstandings caused by our writing. We will reconsider the claims to be more precise. We hope that the responses below could address your specific comments.\n\n**1. Q1&W3: Clarification for the fairness of training data and comparison**\n\nThank you for pointing out this ambiguity. We regret not clearly explaining the following fact in our manuscript: To ensure the fairness of our experiments, all pretraining based baselines are re-pretrained on our pretraining dataset, instead of utilizing the weights published (if any) by the original work. In other words, all the performance comparison are **conducted on the same data** and with the similar schemes, including the BrainBERT and other baselines designed for EEG. Therefore, the performance gains are due to our better model rather than larger and better matched training data.\n\nOne thing worth noting is, in Supplemental Section C, we describe that \"The total amount of data in BrainBERT is 43.7 hours\". What we mean here is that the model weights released by the authors of BrainBERT are pretrained on data of 43.7 hours.\n\nWe will clarify these facts more clearly in line 214 and 509 of our revised manuscript.\n\n**2. Q2&W1: Generalization experiment addresses the concern of subject number**\n\nThank you for the valuable comment. Our response includes two parts:\n\n(1) Obtaining intracranial recordings is challenging as it requires craniotomy surgery for electrode implantation, which involves extensive protocols and approvals. Therefore, in the field of intracranial neural signals, the scale of the subject number in datasets has not reached that of EEG datasets. For instance, a recent work published on ICLR'23, BrainBERT, also included only 10 subjects, with an average recording duration of 4.37 hours per subject [17].\n\nAlthough the work in the manuscript includes only 9 pretraining subjects, the large data scale is still helpful for the model’s generalization ability. This is because intracranial data is a non-stationary time series that encompasses numerous variations, which is sufficient to cover the relatively most common types of epilepsy seen in clinical practice.\n\n(2) We agree that, with a test set of 8 subjects, the experimental results are not that strong to support the improvements of our model. To illustrate the generalization ability of our model on more subjects with more heterogeneity, we added new experiments on the data of another 31 subjects in Global Response. We hope that it can provide an evidence for the improvement of our model on performance involving individual heterogeneity.\n\n**3. W1: Clarification on the calculation of uncertainty**\n\nThank you for the suggestion. We indicate in lines 162-163 that the reported standard deviations were obtained from five random runs, with each evaluation randomly sampling data from 8 subjects. So the uncertainties are not calculated from the experimental results over 8 individuals.\n\n**4. W2: Generalization experiment addresses the concern of split-subject setting**\n\nThank you for the constructive comment. Our response includes two parts:\n\n(1) We concur that the experimental settings that separate training and test subjects (split-subject setting) are highly appropriate in epilepsy detection. Actually, there also exists some seizure detection works without this separation, such as [20,23,R1]. We apologize for the setting we chose, which has limited our experiments to fully showcase the effectiveness of our model on subjects not involved in training.\n\n(2) Following the comment, we added new experiments on 31 subjects in Global Response. We hope that this experiment will help alleviate your concerns regarding performance overestimation and model generalization.\n\n**5. W4: Clarification on the claims about medical scenarios**\n\nThank you for the valuable comments. As an emerging field, the application of intracranial data in machine learning for healthcare is currently focused on seizure detection. In addition to seizure detection, which is directly relevant to medical scenarios, intracranial data may also involve some indirectly related situations, such as tACS, warning systems development, missing data recovery, etc. Therefore, by adding relevant downstream tasks, we hope to provide potential assistance and contribute to the research in these areas.\n\nAmong these downstream tasks, PatchTST performs well on the forecasting, CoST excels on the imputation, and BrainBERT performs best on the seizure detection. However, only BPT achieves consistent SOTA performance on all these downstream tasks, which demonstrates that our model can generalize well to these downstream tasks.\n\nIn line 73, we describe that \"BPT generalizes well to various downstream tasks w.r.t. several medical scenarios\". Actually, what we intended to claim here is that our model can generalize to these downstream tasks, but there might be some imprecision in our wording. We will clarify the relationship between our work and these medical scenarios more clearly in the manuscript. Thank you again.\n\n**6. W4: Generalization experiment addresses the concern of hospital and device diversity**\n\nThank you for the thoughtful suggestion. We hope that the added experiments in Global Response can convince you that our model can generalize to more subject data collected from different hospitals with different devices.\n\nIf you remain unconvinced that we have sufficiently justified a higher score, we would be eager to know where we can make further improvements or address any concerns you have. We appreciate your feedback!\n\nReference:\n\n[R1] Y. Zhao, et al. Patient-Specific Seizure Prediction via Adder Network and Supervised Contrastive Learning, in *IEEE Transactions on Neural Systems and Rehabilitation Engineering*, vol. 30, pp. 1536-1547, 2022. "}, {"review_id": "68EbLOmmhf", "reviewer": "Reviewer_T6RH", "summary": "The authors propose a foundation model named BPT for modeling intracranial neural signals. BPT is pre-trained on a large corpus (1.01 TB) of intracranial recordings collected by the authors. Animportant feature is that BPT is designed to capture long-term temporal dependency and spatial correlation from neural signals, combining information in both time and frequency domains. BPT is pre-trained using a masked autoencoder objective. The results claim that  BPT achieves state-of-the-art performance on various downstream tasks.\n", "questions": "- The pre-training task adopted in this work is masked autoencoder. Why did the authors choose this task over other self-supervised tasks (e.g. contrastive learning, denoising autoencoder)? A comparison and analysis on different pre-training tasks would be helpful.\n\n- Patients demographics: The dataset seems to contain data from many subjects. The analysis of the dataset does not provide much details on the subjects information (e.g. age, gender, diagnoses). More details on this would help to assess generability of the model.\n\n- Is it possible to compare with humans for the same tasks?", "limitations": "- Lack of external validation, despite not being feasible due to the lack of dataset.\n- The authors should discuss any potential privacy concerns or negative societal impacts as neural data is highly sensitive and building large models from such data raises privacy concerns.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The paper proposes a novel self-supervised foundation model for modeling intracranial neural recordings.\n- The authors claim that the size of the model is of unparalleled precedent.\n- The experiments are thorough, well-designed, and provide strong empirical evidence for the effectiveness of the proposed method. \n- The paper is clearly written, with a well-structured flow of ideas. \n- The work has high potential impact and significance for the medical field and perhaps a step towards future neural interfaces.\n- The model will be made publicly available.", "weaknesses": "- The work is scaling up previous architectures. No \n- if possible it would be great to see model preformance on more clinically meaningful tasks\n- The model analysis is a bit superficial. Would be useful for the reader to have a more in-depth analysis, e.g. how different frequency bands contribute to the model performance, how the learned representations capture spatial or temporal relationships in iEEG signals, etc.\n- The results are not compared with other SOTA models on the same datasets. Despite this may not be feasible, specially on medical domain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive and detailed comments. We are excited that you felt our work has high potential impact and significance for the medical field and perhaps a step towards future neural interfaces. Responses to specific comments are listed below. \n\n**1. Q1: Discussion on why not use other SSL tasks (e.g. contrastive learning, denoising autoencoder)**\n\nThank you for the insightful question. The motivation is mainly driven by the variety of downstream tasks. As noted in a recent review on self-supervised learning [R1], generative self-supervised learning (like autoencoder) has the ability to recover the original data distribution without assumptions for downstream tasks, which enables generative models’ wide applications both classification and generation. In contrast, contrastive learning has assumed the downstream applications to be classifications [R1]. Therefore, as a representative of the generative approach, autoencoders meet our needs to design the architecture of a foundation model, which is expected to generalize to various downstream tasks well. \n\nAs for the denoising aotuencoder, both masked autoencoder and denoising autoencoder share the same goal of learning a robust representation of the input signal. Therefore, we empirically believe that there is no intuitive difference between using masked autoencoder or denoising autoencoder in our model architecture. Following exsiting works such as masked language model (MLM) in natural language processing and PatchTST in time series [27], masked autoencoder is used to build our model. \n\n**2. Q2: Notes on patient demographics & Generalization experiment for generability**\n\nThank you for the thoughtful suggestion. Due to the restrictions from the hospital, we sincerely apologize for the inability to disclose certain subject profile details such as age, gender, and diagnoses due to concerns regarding personal privacy. As a supplement to the individual diversity information of subjects in our dataset, we can only disclose that the age range of subjects spans from infants to adults, including males and females. The dataset comprises a collection of relatively diverse and heterogeneous subjects.\n\nTo further alleviate concerns regarding generability of our model, we conducted additional experiments (in Global Response) on 31 new subjects from two public datasets, MAYO and FNUSA. As shown in Table 1 to Table 3 in the PDF in Global Response, BPT holds the consistent SOTA performance on these downstream tasks, demonstrating the generability of our model. \n\n**3. Q3: Discussion on comparing with humans for the same tasks** \n\nThank you for the valuable suggestion. We agree that comparing models to humans is a good way to evaluate the effectiveness of models in assisting humans. However, it is worth noting that the successful completion of these downstream tasks by humans relies on the expertise of professionally trained and experienced neurologists. Conducting direct comparisons between machines and humans requires recruiting specialized neurologists to perform these tasks, which is challenging for us to carry out currently. We appreciate your valuable suggestion and will consider incorporating human comparisons in our future works. Thank you again. \n\nIf you have any other concerns, please let us know what we can do to address any concerns you may have. Thanks for your thoughtful feedback again! \n\nReference: \n\n[R1] X. Liu et al. Self-Supervised Learning: Generative or Contrastive, in *IEEE Transactions on Knowledge and Data Engineering*, vol. 35, no. 1, pp. 857-876, 1 Jan. 2023."}, {"review_id": "9uNAIMU6sA", "reviewer": "Reviewer_5rAo", "summary": "This paper presents Brain Pre-trained Transformer (BPT) for intracranial neural signal modeling. The model was trained on a large-scale dataset and achieved SOTA performance on multiple downstream tasks.", "questions": "Will the datasets be publicly available?\n", "limitations": "It would be great to provide the interpretability analysis of the learned features. ", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- originality: the authors developed a transformer model for brain signal modeling, which can learn rich representations from intracranial neural recordings.\n- quality: this paper presents the largest pre-training model on brain signals, which can capture long temporal dependency and spatial correlations.\n- clarity: the paper is well-written and organized.\n- significance: the model achieved consistent SOTA performance on various downstream tasks.", "weaknesses": "I'm not an expert in this field. So I'm not able to give constructive comments on the weakness.\n\nAs a foundation model for medical scenarios, it seems that the interpretability analysis is missed.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful reading and comments. We are happy that you acknowledged that our model can learn rich representations from intracranial neural recordings. Responses to specific comments are listed below. \n\n**1. Q1: Our planned steps for the dataset release** \n\nThank you for the valuable suggestion. We apologize for the temporary unavailability of the dataset, because it involves highly sensitive intracranial data and personal privacy concerns. \n\nIn the future, in order to make this dataset a scientific research tool and better serve the research community, we plan to promote the release of our dataset in the following three stages: \n\n- Stage1: We plan to publicly release the hidden representations learned by our model through pretraining, along with their corresponding labels, shortly after our work is accepted. This will enable other researchers to not only replicate our model’s experimental results, but also utilize these representations and labels for other research of interest. \n\n- Stage2: We will actively communicate with the hospital and aim to release the raw data of a portion of subjects by the end of the year, to support further research efforts. The same as the work in our manuscript, these releases will be also conducted in compliance with ethical review requirements.\n\n- Stage3: In the future, we will explore the possibility of releasing the full dataset following approval of the relevant ethical review, to allow researchers to use the large-scale dataset for more research. \n\n**2. Ethics statement**\n\nThe data collection and experiments conducted in our work have been approved by the Institutional Review Board (IRB) and passed ethical review. All participants have signed informed consent forms. \n\nIf you still have any other new concerns, we would be eager to know what we can do to address any questions or concerns you may have. Thanks for your thoughtful feedback again! "}, {"review_id": "XaMMjwnH89", "reviewer": "Reviewer_uVSw", "summary": "This paper proposes a foundation model (called BPT) for modeling intracranial brain signals. BPT consists of a Transformer encoder for temporal encoding, and a separate Transformer encoder for spatial encoding. BPT is pre-trained for reconstructing masked input signals, and is fine-tuned for a variety of downstream tasks (short-term and long-term forecasting, seizure detection etc.). Experimental results suggest that BPT outperforms other self-supervised/unsupervised pre-training models, as well as supervised models on downstream tasks. Model scaling analysis further suggests the benefit of having a large model size. ", "questions": "1. Notations: In “Model overview” section, $L$ is used to denote the number of patches. But later in “Patching” section, $N_p$ is used to denote the number of patches. Please be consistent.\n2. What’s the intuition of frequency encoding using PSD as weights? Please justify in “Frequency Encoding” section.\n3. The patch length P is projected to dimension D (“Encoding Process” section) using a linear layer. Doesn’t this ignore temporal dependency within the patch? In my opinion, this is a limitation and should be addressed (e.g., using a small CNN layer) or discussed.   \n4. The patch length is selected to be 6s. Please justify this choice.\n5. Please include some ablation studies to demonstrate the effectiveness of each component in BPT. The ablation studies can be centered around the 3 claimed advantages: 1) long-range dependency, 2) spatial correlation, and 3) time and frequency domains.\n6. Is the data split patient-wise? If not, the downstream tasks may be relatively easy as the same patient’s signals are seen during fine-tuning.\n7. For the imputation task, it is unsurprising that BPT would perform well given that it’s pre-trained for a masked reconstruction task. Instead of imputing masked timestamps, a more challenging imputation task would be to mask an entire channel and impute the values in the masked channel.\n8. Please also include GNN-based approaches listed in Figure 1 as baselines. e.g., BrainNet, EEG-GCNN, Tang et al. [23].\n", "limitations": "Limitations about the dataset size are discussed. In my opinion, the following two points are also limitations and should be discussed:\n- Data come from a small number of subjects.\n- Projecting the temporal dimension of a patch using a linear layer.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. While the ideas of self-supervised pre-training using a reconstruction task and modeling brain signals using Transformer encoders are not new, the originality of this work comes from a large model pre-trained (largest model for brain signals to date) on a large dataset, as well as its analyses on a variety of downstream tasks.\n2. The methods are technically sound, and the experiments are carefully designed to support the authors’ claims.\n3. The paper is relatively easy to understand.", "weaknesses": "1. Baselines seem to be chosen selectively. Some existing works are shown in Figure 1 but not included as baselines (e.g., GNN-based methods, including BrainNet, EEG-GCNN, Tang et al. [23]).\n2. While the pre-training dataset is large (1.1TB), it only consists of 9 subjects (Appendix E), which could limit the representativeness of the learned latent representations.\n3. There is an overload of notations in the Method section.\n4. The authors claim that BPT is a foundation model for brain signals. However, as I noted above, the pre-training dataset only comprises iEEG data from 9 subjects. The downstream experiments are also done on iEEG data from a small number of subjects (8 subjects). In my opinion, more diverse pre-training data and more downstream experiments on larger datasets are needed to demonstrate that BPT can be served as a foundation model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive and detailed comments. Responses to specific comments are listed below.\n\n**1. Q1: Clarification on notations**\n\nThank you for pointing out this potential ambiguity. Actually, $N_p$ denotes the number of all the patches from an entire channel, while $L$ denotes the number of patches input to the temporal encoder each time. We will provide a clearer clarification in lines 92-94 in the manuscript.\n\n**2. Q2: Intuition for using PSD as weights**\n\nPSD displays the spectal power distribution in different frequencies of brain signals. The power distribution in different frequency bands is associated with different brain functional states. For example, during wakefulness, $\\alpha$ and $\\beta$ waves are more active; during sleep, $\\delta$ and $\\theta$ waves are more prominent; and $\\gamma$ waves will become significant during seizures [R1]. Therefore, in medical research, the power distribution is often used as a feature reflecting the functional state of the brain.\n\nIn our work, we treat the power distribution as the weights of 8 learnable representations in 8 standard frequency bands. The intuition is that the frequency bands with higher power are more representative of the brain’s physiological state, and should be given more weight in their representations. In this way, the model can focus more on the frequency bands where the main power of the brain signals resides. We will explain this intuition clearer in line 104 of the manuscript.\n\n**3. Q3&L2: Discussion on whether to extract temporal dependencies within the patch**\n\nThank you for the insightful suggestion. For high-frequency signals like intracranial data, we aggregate timestamps into patches to enhance the locality and extract semantic information. Here we consider the patch as the smallest unit for representing brain signals, so we ignore the temporal dependency within the patch. Moreover, recent works [27,R2] in the field of time series also project patches to the hidden dimension using a linear layer, and achieves SOTA performance on a variety of time series data. Therefore, we choose to use a linear layer to project patches to dimension D. \n\nInspired by your valuable suggestion, we agree that using CNN to perform convolution on patches is beneficial in extracting the internal dependencies within patches. We will include this discussion in the the limitation of the manuscript.\n\n**4. Q4: Reason for choosing a patch length of 6s**\n\nBased on recent works [R3,17] and the experience of our neurologists, setting a few seconds as the minimum unit is empirically a reasonable choice to capture the underlying physiological states of the brain from neural signals. For instance, in [R3], researchers used window lengths of 10s and 6s, respectively, to learn the spatiotemporal dynamics of the brain. In [17], segment length of 5s was chosen on intracranial signals for their model pretraining. Therefore, we choose 6s as the patch length for training of our model.\n\n**5. Q5: Supplementary ablation experiments**\n\nWe apologize for initially overlooking the detailed justifications for the model architecture design. To demonstrate the effectiveness of each components in BPT, we did the ablation experiments on three model variants you mentioned. The ablation results are given in Table 4 in the PDF in Global Response. We will include the ablation study in the appendix of our manuscript.\n\n**6. Q6: Generalization experiment addresses the concern of split-subject setting**\n\nThank you for the comment. In the experiments in our manuscript, the data for fine-tuning and testing were sampled from the same group of subjects. We hope that the additional experiments on 31 new subjects in the Global Response can provide complementary support for the generalization of our model on split-subject setting.\n\n**7. Q7: Discussion on masking an entire channel in the imputation task**\n\nThank you for the suggestion. For measurement problems, imputation can fill in the incomplete neural signals, allowing other medical devices to continue operating even when data is missing [36] (details in lines 188-191). Based on our experience in data acquisition and cleaning processes, missing records are usually partial sample point missing caused by mechanical or electromagnetic interference, rather than an entire channel lost. Moreover, in the field of time series, existing works [41,R4] on the imputation task were also randomly masking a certain proportion of sample points to simulate malfunctions in real-world systems. Depending on the application and following existing works, we impute the masked sample points as our imputation task.\n\n**8. Q8: The reason why GNN-based methods cannot be baselines**\n\nThank you for the suggestion. These GNN-based methods were not included in the baselines, not because we selectively chose the baselines, but because these graph-based methods require constructing graph nodes using channels (each channel corresponds to a node). However, the number of channels varies among different subjects, which means that graph-based methods can only be applied to individual subjects. Therefore, we apologize for not being able to include these methods as baselines because they cannot fulfill our experimental setting on multiple subjects.\n\nIf you remain unconvinced that we have sufficiently justified a higher score, please inform us where we can further improve our work or any concerns you have. Thank you!\n\nReference: \n\n[R1] P. A. Abhang, et al. Chapter 2: Technological Basics of EEG Recording and Operation of Apparatus, 2016\n\n[R2] Shao, Zezhi, et al. Pre-training enhanced spatial-temporal graph neural network for multivariate time series forecasting. 2022\n\n[R3] Zhang Y, et al. Functional annotation of human cognitive states using deep graph convolution. 2021\n\n[R4] Zhou T, et al. One Fits All: Power General Time Series Analysis by Pretrained LM. 2023"}]

Author rebuttal:
# Global Response to AC and all reviewers  
Thanks to all reviewers for the careful reading and thoughtful feedback. We are grateful for acknowledging "the model size is of unparalleled precedent" and our work "has high potential impact and significance for the medical field and perhaps a step towards future neural interfaces" (Reviewer T6RH). We are also excited that our work "could contribute to an exciting line of work building foundation models that are capable of enhancing neuroscience, neurology, neurosurgery, and psychiatry." (Reviewer 5spB). Further, we are delighted to hear "the originality comes from a large model pretrained on a large dataset, as well as its analysis on a variety of downstream tasks" (Reviewer uVSw) with "consistent SOTA performance on various downstream tasks" (Reviewer 5rAo).  

In response to reviewers' comments, we performed two new experiments: generalization experiment and ablation experiment, summarizing the additional studies. 

## About Generalization Experiment

To supplement the generalization ability of BPT on more subjects with more heterogeneity, we added the new generalization experiment on the data of another 31 previously useen subjects, as a solution to similar concerns raised by some reviewers. These concerns include: 

- the small number of test subjects (Reviewer 5spB, uVSw), 
- the lack of separation between fine-tuning and testing subjects (Reviewer 5spB, uVSw), and 
- the single-source of the hospital and device (Reviewer 5spB). 

We introduce the generalization experiment as follows: 

- Subject number: We evaluated the model on data of 31 unseen subjects from two public datasets named MAYO and FNUSA [R1]. These two public datasets include 18 and 13 subjects, respectively.

- Diversity on countries, hospitals and devices: These 31 subjects were from two distinct hospitals or institutions located in different countries. As for the acquisition equipment, these data were recorded using completely different acquisition systems and electrode types. Details are given in the table below:

  | Dataset | Country & location | Hospital   | Acquisition system   | Electrode type | Electrode material  | Contact diameter | Contact length | Contact distance |
  | ------- | ---------------------------------------------- | ------------------------------ | ---------------------------------------------- | --------------------------------------- | ---------------------------------- | ---------------- | -------------- | ---------------- |
  | MAYO    | Rochester, Minnesota, United States of America | Mayo Clinic                    | Neuralynx Cheetah system (Neuralynx Inc., USA) | depth electrodes (AD-Tech Corp., USA)   | 4 or 8 contact Pt/Ir electrodes   | 1 mm             | 2.3 mm         | 5 or 10 mm       |
  |         |       |        |                 | grids and strips electrodes             | Pt/Ir discs                        | 4.0 mm           | /              | 10 mm            |
  | FNUSA   | Brno, Czech Republic                           | St. Anne’s University Hospital | BrainScope system (M&I, BrainScope, Czech)     | depth electrodes (Temis Health, France) | 5, 10 and 15 contact Pt electrodes | 0.8 mm           | 2 mm           | 1.5 mm           |

- Split-subject setting: To separate out subjects for the training and test set, we re-evaluated the performance of BPT on the 31 unseen subject data. To ensure the fairness of the experiment, for BPT and each baseline method, we followed the same setting, i.e., fine-tuning on our test set of 8 subjects and re-evaluating them on 31 subject data. In this way, we could assess the model’s generalization ability on data of subjects not involved during the training phase.

- Consistency: To maintain consistency with the experiments in our manuscript, we utilized the BPT weights that were fine-tuned on our test set. In other words, the model used for the generalization experiments was identical to the one in our manuscript.

The results of our additional generalization experiment are given in Table 1 to Table 3 in the new PDF uploaded with this Global Response, where we mark values ranking the first (bold), second (underlined) and third (starred) in each column. As shown in Table 1 to Table 3 in the new PDF, compared to other baseline models, BPT holds the consistent SOTA performance on these downstream tasks on the **unseen 31 heterogeneous subject data** collected from **different hospitals in different countries with different devices**, highlighting the generalization ability of our model **on the split-subject setting**. We hope that this additional experiment will help alleviate the reviewer's concerns regarding performance overestimation and model generalization. We will add this new experiment to our revised paper. Its experimental setup and analysis will be added to the Experiment section of the main paper, and its results will be added to Appendix to maintain the 9-page limit.

## About Specific Responses

We have individually addressed all of your comments below, specifically addressing each reviewer’s concerns in the corresponding responses. Please note that in our responses, references in the format "[R1]" indicate citations that are newly added in the rebuttal, while references in the format "[1]" are citations from the original manuscript. 

We have dedicated significant effort to improving our manuscript, and we sincerely hope that our responses will be informative and valuable. We would love to receive your further feedback. 

Reference:

[R1] P. Nejedly et al. Multicenter intracranial EEG dataset for classification of graphoelements and artifactual signals, *Sci Data*, vol. 7, no. 1, Art. no. 1, Jun. 2020.

Meta-review: The paper has convinced the 4 reviewers. Although the methodology employed is not particularly novel (pretraining with masking task) the experiments have been considered overall convincing with extensive comparisons. It also worth pointing out that this work uses a unique dataset for the field. One can just regret that such data cannot be shared, although the code and the trained models are made available. I am endorsing this work for publication at NeurIPS 2023.
Decision: Accept (poster)