SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Imputation. As a mitigation for measurement problems, imputation can fill in the incomplete neural signals, allowing other medical devices to continue operating even when data is missing (see Sec. 3.3). From Tab. 2, Brant and Brant-Freeze achieve the best and the second best results among all the methods, which demonstrates the ability of our model in capturing underlying temporal patterns with partially observed neural recordings to impute the contaminated neural signals.
Seizure detection. As seizure detection is an important medical application of intracranial recordings, we not only compare our model with the pre-training based baselines, but also further select 4 supervised method designed for seizure detection. From Tab. 3, Brant outperforms all the other methods and Brant-Freeze achieves second best on most of the metrics. BrainBERT [20] achieves the best accuracy, precision and F2 score among all the baseline methods, primarily due to it provides contextualized neural embeddings and combines the information from time and frequency domains like our model. However, Brant still improves the F2 score by  $29.59\%$  over BrainBERT, because our temporal encoder obtains a wider receptive field and the spatial encoder captures the spatial correlation across channels, which are both critical in modeling intracranial recordings.
Table 2: Performance on the imputation task.
Task Model	Imputation
MAE	MSE
RP [17]	0.5953±0.0008	0.9158±0.0038
TS [17]	0.5225±0.0008	0.7476±0.0032
CPC [17]	0.5663±0.0006	0.8271±0.0034
BENDR [18]	0.4849±0.0008	0.6492±0.0031
MVTS [19]	0.5946±0.0009	0.8903±0.0032
BrainBERT [20]	0.6083±0.0008	0.9790±0.0026
PatchTST [27]	0.4282±0.0007	0.5506±0.0012
TS-TCC [39]	0.6144±0.0008	1.0558±0.0029
TF-C [38]	1.1876±0.0014	3.0179±0.0042
CoST [37]	*0.2652±0.0005	*0.1638±0.0007
Brant-Freeze	0.1963±0.0005	0.0865±0.0004
Brant	0.1912±0.0003	0.0814±0.0002
Table 3: Performance on the seizure detection task.
Task
Model	Seizure detection
Accuracy	Precision	Recall	F1	F2
Spectral Power [28]	89.01±0.12	72.77±1.98	36.07±1.23	48.23±0.57	40.12±1.02
Rhythmicity Spectrogram [3]	88.58±0.16	70.31±2.09	37.07±1.39	48.55±0.60	39.10±1.09
Amplitude-integrated EEG [40]	88.82±0.19	71.21±2.01	35.73±1.50	47.58±0.61	39.68±1.12
SEEG-Net	88.97±0.14	70.45±2.20	38.44±1.47	*49.74±0.60	42.28±1.10
RP [17]	67.65±1.21	18.62±2.45	34.71±2.11	24.24±1.87	29.59±1.97
TS [17]	85.90±0.36	54.68±3.65	31.66±1.97	40.10±0.81	34.57±1.66
CPC [17]	84.72±0.40	48.31±2.80	36.03±0.92	41.28±0.66	37.96±1.42
BENDR [18]	88.14±0.68	71.49±3.42	29.83±2.03	42.10±1.84	33.77±1.81
MVTS [19]	88.35±0.22	69.43±3.19	32.03±2.08	43.84±1.96	35.90±1.94
BrainBERT [20]	*89.59±0.12	77.86±3.10	39.28±0.88	52.21±0.39	*43.60±0.98
PatchTST [27]	81.13±0.22	38.71±3.64	21.16±2.21	27.37±0.42	23.27±1.26
TS-TCC [39]	88.13±0.29	89.61±2.08	23.81±1.50	37.62±0.40	27.91±1.19
TF-C [38]	75.05±0.61	18.75±2.41	19.09±2.02	18.92±0.41	19.02±1.24
CoST [37]	81.05±0.15	30.98±2.59	*43.19±1.88	36.08±0.79	40.03±1.88
Brant-Freeze	90.53±0.33	*77.26±3.16	48.92±0.72	59.87±0.62	52.87±0.47
Brant	91.17±0.15	79.25±2.32	52.74±1.43	63.29±0.37	56.50±1.08
4.2 Model Analysis.
Low-resource labeled data evaluation. In medical scenarios, collecting labeled data for even small experiments is a huge investment. To demonstrate the practical value of our work, we evaluate Brant on seizure detection where the amount of labeled data is limited. Specifically, the pre-trained model is fine-tuned on 200 minutes, 60 minutes and 20 minutes of labeled data that sampled from the downstream dataset (Sec. 3.1), respectively. After fine-tuning, models are evaluated on the same 50 minutes of labeled data which is also sampled from the downstream dataset but non-overlapped with the fine-tuning data.
From Tab. 4, the performances of supervised methods decrease rapidly compared to the self-supervised or unsupervised methods, showing that the representations learned on unlabeled data can improve low-resource settings. Among the pre-training works, our model maintains the most stable performance on 20-minute labeled data. Note that the F2 score of our model on 20-minute labeled data (51.03%) is even higher than the F2 score of the best baseline on 200-minute labeled data (43.60%), demonstrating that Brant fully captures the patterns and semantic information from the intracranial data during pre-training and adapts to downstream tasks more easily.
Table 4: Low-resource labeled data evaluation on the seizure detection task and the relative decrease of the F2 score on 60-minute and 20-minute labeled data versus 200-minute labeled data.
Model	200 minutes	60 minutes	20 minutes
F2	F2	Decrease	F2	Decrease
Spectral Power [28]	40.12±1.02	25.95±1.92	35.32%	5.06±1.67	87.39%
Rhythmicity Spectrogram [3]	39.10±1.09	21.87±2.33	44.07%	2.81±0.87	92.81%
Amplitude-integrated EEG [40]	39.68±1.12	18.42±2.34	53.58%	2.05±0.95	94.84%
SEEG-Net [9]	*42.28±1.10	35.54±1.90	15.94%	12.76±2.13	69.82%
RP [17]	29.59±1.97	27.62±2.03	*6.66%	25.05±1.98	15.34%
TS [17]	34.57±1.66	30.15±3.05	12.79%	29.61±3.34	*14.35%
CPC [17]	37.96±1.42	30.55±3.01	19.52%	29.57±3.74	22.10%
BENDR [18]	33.77±1.81	25.37±3.12	24.87%	22.18±4.09	34.32%
MVTS [19]	35.90±1.94	26.62±3.11	25.85%	24.39±4.01	32.06%
BrainBERT [20]	43.60±0.98	41.93±2.09	3.84%	36.35±3.23	16.63%
PatchTST [27]	23.27±1.26	18.02±2.23	22.55%	17.07±2.11	26.64%
TS-TCC [39]	27.91±1.19	25.35±2.07	9.17%	20.36±1.90	27.05%
TF-C [38]	19.02±1.24	15.97±1.23	16.04%	13.66±2.10	28.18%
CoST [37]	40.03±1.88	*39.18±3.02	2.12%	36.10±4.12	9.82%
Brant	56.50±1.08	52.30±2.04	7.43%	51.03±2.74	9.68%
Representation analysis. As classification task can verify the model capacity in high-level representation learning [41], we visualize the pre-trained representations of Brant and 3 best pretraining based methods on seizure detection task using t-SNE (shown in Fig. 4). Compared to other methods, the representations of seizure and normal signals learned from Brant are separated more clearly during pre-training, which intuitively illustrates our SOTA performance on low-resource label settings. Furthermore, the results explain the good performance of Brant-Freeze on seizure detection.
Figure 4: Pre-trained representation visualization of Brant and 3 best baselines on seizure detection task. The representations of seizure and normal samples are plotted in red  $(\bullet)$  and yellow  $(\bullet)$ .
Model scale analysis. To explore the effect of model size on performance, we additionally pretrain three variants of Brant with smaller size: Brant-tiny, Brant-small and Brant-medium (detailed configurations are in App. F) on the same dataset as Brant. All models are pre-trained with an Adam optimizer and a cyclic scheduler. Brant-medium adopts the same learning rate as Brant (Sec. 3.2); for Brant-tiny and Brant-small, the basic and maximum learning rates are  $6 \times 10^{-6}$  and  $3 \times 10^{-5}$ .
We evaluate these smaller size variants on all the downstream tasks (results shown in Fig. 5). As the model size increases, the performances on the downstream tasks show an overall upward trend, where Brant is ahead of the other three variants in all metrics. Also, the decrease in the standard deviation indicates more stable performance for larger models. On such a huge neural dataset, a larger model with a higher capacity results in better generalization ability to a wide range of downstream tasks.
Figure 5: Performance on all the downstream tasks across Brants with different model size: Branttiny (70M), Brant-small (100M), Brant-medium (250M) and Brant (500M).
Ablation study. We perform ablation experiments to assess the effectiveness of each individual component in our model. Specifically, we remove the following components from our model to examine their respective effects on performance: the temporal encoder (Brant-temporal), the spatial encoder (Brant-spatial), and the frequency encoding (Brant-frequency).
The experimental results of ablation study are given in Tab. 5 and Tab. 6. The results indicate that Brant outperforms other model variants across all metrics, providing evidence for the contribution of each component in our model. Among the different variants, Brant-temporal exhibits the most substantial decrease in performance compared to the full Brant model, emphasizing the significance of long-term dependency in modeling brain signals. The performance degradation observed in Brant-spatial demonstrates the role of considering channel spatial correlation in learning representations of intracranial recordings. Furthermore, the decline of Brant-frequency suggests that simultaneously extracting information from both the time and frequency domains proves beneficial for effectively modeling brain signals.
Table 5: Results of ablation experiments on forecasting tasks.
Task Model	Short-term signal forecasting	Long-term signal forecasting	Frequency-phase forecasting
MAE	MSE	MAE	MSE	ph. PLV	freq. MAE
Brant-temporal	0.5356	0.7021	0.5777	0.7876	0.5800	0.0094
Brant-spatial	0.5403	0.7521	0.5827	0.7990	0.5780	0.0092
Brant-frequency	0.4762	0.5676	0.5726	0.7741	0.5645	0.0143
Brant	0.4007	0.4626	0.5693	0.7676	0.6004	0.0041
Table 6: Results of ablation experiments on the imputation and seizure detection tasks.
Task Model	Imputation	Seizure detection
MAE	MSE	Acc.	Prec.	Rec.	F1	F2
Brant-temporal	0.2798	0.1797	88.56	67.39	40.25	50.40	43.78
Brant-spatial	0.2770	0.1741	89.93	84.77	36.88	51.40	41.57
Brant-frequency	0.2495	0.1406	90.16	78.41	44.03	56.39	48.26
Brant	0.1912	0.0814	91.17	79.25	52.74	63.29	56.50
Generalization ability analysis. To further verify the generalization ability of Brant on more subjects with more heterogeneity, we evaluated the model on data of 31 unseen subjects from two public datasets named MAYO and FNUSA [42]. More details about this study are shown in App. G.
Intracranial recordings modeling. With the development of iEEG technique to obtain deep brain information from both cortical and subcortical structures, modeling intracranial recordings has attracted the attention of many researchers. Wang et al. [9] extract time domain features of iEEG signals from multiple receptive fields by utilizing multiscale CNN and LSTM. Guo et al. [10] introduce a hypergraph learning approach to model iEEG signals by detecting high frequency oscillations. Yu and Hu [11] propose EDANN to learn domain-invariant representations of iEEG data from multiple subjects by domain adversarial training. Jiang et al. [8] develop a novel method using short-time resting-state connectivity to identify the seizure onset zone (SOZ) from interictal EEG signals. Wang et al. [12] conduct cross-subject iEEG seizure detection based on adaptive feature fusion of brain network features and single-channel features. However, these works are all supervised that relies heavily on labeled data, which is often difficult and expensive to obtain at scale. Furthermore, most works for intracranial data modeling process each channel independently, ignoring the spatial correlation. Although Chen et al. [23] propose BrainNet which contains a graph diffusion component which measures the brain wave diffusion among channels, their work only focus on the information in time domain and is limited to an individual subject.
Pre-training on brain signals. Pre-training on time series [37, 38, 27, 39] has shown good performance in many scenarios (e.g., weather, traffic flow, exchange rates), including some works designed for brain signals. Banville et al. [17] learn representations from EEG signals by a self-supervised temporal context prediction task, revealing clear latent structures related to physiological and clinical phenomena. Kostas et al. [18] address the problem of limited labeled data on EEG by using a contrastive self-supervised learning task to pre-train a model named BENDR, which is then fine-tuned for downstream tasks. Potter et al. [19] propose an unsupervised approach to model EEG signals using a transformer-based model with a signal reconstruction task. Cai et al. [43] propose to study the self-supervised learning framework for brain signals that can be applied to pre-train either scalp or intracranial EEG data. Wang et al. [20] propose a pre-training work named BrainBERT for intracranial recordings and conduct experiments on iEEG data. They adopt time-frequency representations but ignore the spatial correlation between channels, which is critical in modeling intracranial recordings (additional commentary on differences between our work and BrainBERT is in App. D).
We propose a task-agnostic foundation model, Brant, which learns powerful representations of intracranial recordings. Brant is the largest pre-training model on brain signals, whose design (1) attends a long temporal dependency; (2) captures the spatial correlation between channels; and (3) extracts information from both time and frequency domains. Experimentally, Brant achieves consistent SOTA performance on various downstream tasks w.r.t. medical scenarios. Further analysis shows the effectiveness and benefit of a large-scale pre-trained model in the field of medicine. Brant is an off-the-shelf model with its code and weights, which significantly alleviates the issue of sample and label efficiency and can directly participate in other medical research and treatment.
Limitations and future works. By pre-training on a large amount of intracranial data, Brant contains over 500M parameters, far more than other existing works on brain signals. However, compared to other fields such as CV and NLP in which the models can reach billions of parameters and achieve good performance on a variety of tasks by zero-shot learning, there is still potential for further improvement of our work. In the future, by scaling up our dataset, the scale of our model can be further expanded to capture higher-level semantic information from neural data, revealing more complicated brain activities and dynamics, to provide assistance for more healthcare applications.
This work is supported by NSFC (No.62176233), the National Key Research and Development Project of China (No.2018AAA0101900) and the Fundamental Research Funds for the Central Universities.
[31] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
[32] Raluca Maria Aileni, Sever Pasca, and Adriana Florescu. Eeg-brain activity monitoring and predictive analysis of signals using artificial neural networks. Sensors, 20(12):3346, 2020.
[33] DK Thara, BG Premasudha, TV Murthy, and Syed Ahmad Chan Bukhari. Eeg forecasting with univariate and multivariate time series using windowing and baseline method. International Journal of E-Health and Medical Communications (IJEHMC), 13:1–13, 2022.
[34] Farrokh Mansouri, Katharine Dunlop, Peter Giacobbe, Jonathan Downar, and José Zariffa. A fast eeg forecasting algorithm for phase-locked transcranial electrical stimulation of the human brain. Frontiers in Neuroscience, 11, 2017.
[35] Anke Karabanov, Axel Thielscher, and Hartwig Roman Siebner. Transcranial brain stimulation: closing the loop between brain and stimulation. Current Opinion in Neurology, 29(4):397-404, 2016.
[36] Atsunori Kanemura, Yuhsen Cheng, Takumi Kaneko, Kento Nozawa, and Shuichi Fukunaga. Imputing missing values in eeg with multivariate autoregressive models. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 2639-2642, 2018.
[37] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshit Kumar, and Steven C. H. Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. CoRR, abs/2202.01575, 2022.
[38] Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency, 2022.
[39] Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. CoRR, abs/2106.14112, 2021.
[40] Maliheh Kadivar, Elahe Movahedi Moghadam, Reza Shervin Badv, Raziye Sangsari, and Maryam Saeedy. A comparison of conventional electroencephalography with amplitude-integrated EEG in detection of neonatal seizures. Medical Devices: Evidence and Research, pages 489-496, 2019.
[41] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis, 2023.
[42] Petr Nejedly, Vaclav Kremen, Vladimir Sladky, Jan Cimbalnik, Petr Klimes, Filip Plesinger, Filip Mivalt, Vojtech Travnicek, Ivo Viscor, Martin Pail, et al. Multicenter intracranial EEG dataset for classification of graphoelements and artifactual signals. Scientific data, 7:179, 2020.
[43] Donghong Cai, Junru Chen, Yang Yang, Teng Liu, and Yafeng Li. Mbrain: A multi-channel self-supervised learning framework for brain signals, 2023.
Ethics Statement
The data collection and experiments conducted in our work have been approved by the Institutional Review Board (IRB) and passed ethical review. All participants have signed informed consent forms.
A Intuition of Using PSD as Weights
PSD displays the spectral power distribution in different frequencies of brain signals. The power distribution in different frequency bands is associated with different brain functional states. For example, during wakefulness,  $\alpha$  and  $\beta$  waves are more active; during sleep,  $\delta$  and  $\theta$  waves are more prominent; and  $\gamma$  waves will become significant during epileptic seizures. Therefore, in medical research, the power distribution in different frequency bands is often used as a feature reflecting the functional state of the brain.
In our work, we treat the power distribution as the weights of learnable representations  $f_{i}, i = 1,\dots,8$  in 8 standard frequency bands. The intuition is that the frequency bands with higher power are more representative of the brain's physiological state, and should be given more weight in their representations. In this way, the model can focus more on the frequency bands where the main power of the brain signals resides.
B Details of the Dominant Frequency and Phase
Dominant frequency. Dominant frequency is defined as the frequency related with the greatest power or amplitude in the brain signal. Dominant frequency can be calculated from Fourier Transform spectrum. Mathematically, given a brain signal series  $s(t), t \in (t_0, t_1)$ , the amplitude  $A(\omega)$  on frequency  $\omega$  is:
$$
A (\omega) = | F (\omega) | = \left| \int_ {t _ {0}} ^ {t _ {1}} s (t) e ^ {- i \omega t} d t \right|, \tag {5}
$$
and the dominant frequency  $\omega_{D}$  is:
$$
\omega_ {D} = \arg \max  _ {\omega} A (\omega) \tag {6}
$$
Phase. The instantaneous phase of a complex-valued function signal  $s(t)$  is the real-valued function:
$$
\psi (t) = \operatorname {a n g l e} (s (t)), \tag {7}
$$
where  $angle(\cdot)$  is the angle between the positive real axis and the direction of complex number  $s(t)$ .
C Details of Baselines
Firstly, we compare our model to the existing self-supervised or unsupervised pre-training works on brain signals. The Details of these baseline models are given here:
- RP, TS, CPC: Three self-supervised learning approaches with three self-supervised learning pretext tasks of relative positioning (RP), temporal shuffling (TS) and contrastive predictive coding (CPC) to learn representations of EEG signals based on StagerNet.
- BENDR: A self-supervised training model that learn compressed representations to model completely novel raw EEG sequences recorded with differing hardware, and different subjects performing different tasks.
- MVTS: An unsupervised transformer-based anomaly detection approach to train an autoencoder with a masking strategy on non-seizure signals.
- BrainBERT: A reusable transformer for intracranial field potential recordings enables classifying complex concepts and decoding neural data.
Secondly, we compare our model with the advanced self-supervised or unsupervised pre-training works designed for general time series, including:
- CoST: A time series representation learning framework for long sequence time series forecasting, which applies contrastive learning methods to learn disentangled seasonal-trend representations.
- TF-C: A decomposable pre-training model, where the self-supervised signal is provided by the distance between time and frequency components, each individually trained by contrastive estimation.
- PatchTST: An efficient design of Transformer-based models for multivariate time series forecasting and self-supervised representation learning, based on the segmentation of time series into subseries-level patches.
- TS-TCC: An unsupervised time series representation learning framework to learn robust temporal and discriminative representations by designing a tough cross-view prediction task and a contextual contrasting module.
Furthermore, as seizure detection is the most important application scenario of intracranial recordings, to evaluate Brant's ability in seizure detection, we compare our model with some seizure detection methods on EEG or iEEG. These baselines include handcrafted feature based methods and a deep learning based method:
- Spectral Power: The spectral power in a particular frequency band means the power of a signal in that frequency band. The rhythmic activity on brain signals is typically described in terms of the spectral power or its ratio on some standard frequency bands.
- Rhythmicity Spectrogram: This feature displays a density spectral array of frequency and power characteristics of brain signals, providing a graphical depiction of the amplitude of primary rhythmic EEG components present in four frequency bands.
- Amplitude-integrated EEG: A trace of electrical activity displayed on a semilogarithmic graph of peak-to-peak amplitude over time, which can be used to monitor and diagnose seizure activity.
- SEEG-Net: A model that can address the problems of sample imbalance, cross-subject domain shift, and poor interpretability and realizes high-sensitivity SEEG pathological activity detection. Since the source code of SEEG-Net is not released, we re-implement it to conduct experiments.
D Detailed Differences between Brant and BrainBERT
BrainBERT is also a pre-training work conducted on intracranial recordings, but our work is different with them in the following aspects.
Data and model scale. Pre-training often requires a large amount of data by which the model fully learns the distributions, and the pre-trained model usually contains a huge quantity of parameters to fit the data. According to the paper of BrainBERT and the model weights released by its authors [20], the total amount of data and parameters in BrainBERT is 43.7 hours and 43.18M, respectively. Brant is pre-trained on 2528 hours of data and the scale of Brant reaches to 505.69M parameters.
Method. As discussed in the main paper, Brant considers long-term dependency, channel correlation, time and frequency domains when modeling intracranial recordings. BrainBERT provides contextualized neural embeddings and adopts time-frequency representations, but ignores the channel correlation which reflects the propagation of brain waves.
Downstream tasks. BrainBERT is evaluated on four classification tasks related with the audio listened from the subjects, which mainly focus on a particular stimulus, i.e., passive movie viewing. Brant is evaluated on various downstream tasks, including signal forecasting, frequency-phase forecasting, imputation and seizure detection, all of which are medically valuable (described in Sec. 3.3).
E Details of the Datasets
Pre-training dataset. The pre-training dataset contains 1.01TB data with a sampling rate of  $1000\mathrm{Hz}$ . The details of the pre-training dataset are given in Tab. 7.
Downstream dataset. The downstream dataset contains 29.39 GB data with a sampling rate of  $1000\mathrm{Hz}$ . The details of the downstream dataset with seizure labels are given in Tab. 8. For each downstream task, we sample a small subset from the downstream dataset for fine-tuning and evaluation. The sampling and split strategies are described in the setup of downstream tasks in Sec. 3.3.