Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI
Authors: Weibang Jiang
Liming Zhao
Bao-liang Lu
Venue: ICLR 2024 spotlight (2024)
Keywords: EEG
brain-computer interface
representation learning
Primary area: applications to neuroscience & cognitive science

Abstract: The current electroencephalogram (EEG) based deep learning models are typically designed for specific datasets and applications in brain-computer interaction (BCI), limiting the scale of the models and thus diminishing their perceptual capabilities and generalizability. Recently, Large Language Models (LLMs) have achieved unprecedented success in text processing, prompting us to explore the capabilities of Large EEG Models (LEMs). We hope that LEMs can break through the limitations of different task types of EEG datasets, and obtain universal perceptual capabilities of EEG signals through unsupervised pre-training. Then the models can be fine-tuned for different downstream tasks. However, compared to text data, the volume of EEG datasets is generally small and the format varies widely. For example, there can be mismatched numbers of electrodes, unequal length data samples, varied task designs, and low signal-to-noise ratio. To overcome these challenges, we propose a unified foundation model for EEG called Large Brain Model (LaBraM). LaBraM enables cross-dataset learning by segmenting the EEG signals into EEG channel patches. Vector-quantized neural spectrum prediction is used to train a semantically rich neural tokenizer that encodes continuous raw EEG channel patches into compact neural codes. We then pre-train neural Transformers by predicting the original neural codes for the masked EEG channel patches. The LaBraMs were pre-trained on about 2,500 hours of various types of EEG signals from around 20 datasets and validated on multiple different types of downstream tasks. Experiments on abnormal detection, event type classification, emotion recognition, and gait prediction show that our LaBraM outperforms all compared SOTA methods in their respective fields. Our code is available at https://github.com/935963004/LaBraM.

Published as a conference paper at ICLR 2024
LARGE BRAIN MODEL FOR LEARNING GENERIC REP- RESENTATIONS WITH TREMENDOUS EEG DATA IN BCI
Wei-Bang Jiang', Li-Ming Zhao Bao-Liang Lu
1 Shanghai Jiao Tong University Shanghai Emotionhelper Technology Co., Ltd. 9359630048s jtu. edu.
edu. en, liming.
com, bll allu@situ. edu. en
ABSTRACT
The current electroencephalogran (EEG) based deep learning models are typi- cally designed for specific datasets and applications brain-computer interac- tion (BCI), limiting the scale of the models and thus diminishing their pereep- tual capabilities and generalizability. Recently, Large Language Models (LLMs) have achieved unprecedented success in text processing, prompting to explore the capabilities of Large EEG Models (LEMs). We hope that LEMs can break through the limitations of different task types of EEG datasets, and obtain uni- versal perceptual capabilities of EEG signals through unsupervised pre-training. Then the models can be fine-tuned for different downstream tasks. However, com- pared to text data, the volume of EEG datasets is generally small and the for- mat varies widely. For example, there can be mismatched numbers of electrodes. unequal length data samples, varied task designs, and low signal-to-noise ratio. To overcome these challenges, we propose a unified foundation model for EEG called Large Brain Model (LaBraM). LaBraM enables cross-dataset learning by segmenting the EEG signals into EEG channel patches. Vector-quantized neu- ral spectrum prediction is used to train semantically rich neural tokenizer that encodes continuous raw EEG channel patches into compact neural codes. We then pre-train neural Transformers by predicting the original neural codes for the masked EEG channel patches. The LaBraMs were pre-trained on about 2,500 hours of various types of EEG signals from around 20 datasets and validated multiple different types of downstream tasks, Experiments on abnormal detection. event type classification, emotion recognition, and gait prediction show that our LaBraM outperforms all compared SOTA methods in their respective fields. Our code is available at
com/ 935963004 LLaBraM
INTRODUCTION
Eleetrreencephalogrrahyy (EEG) is a method record electrogram of the spontaneous electrical activity of the brain. It is typically non-invasive, with the EEG electrodes placed along the scalp using the international 10-20 system. EEG signals can be formulated as a matrix of real numbers X E RCXT, where C' is the number of EEG electrodes (channels) that may vary depending on the acquisition equipment used, and represents the total number of samples, which is related to the collection time and sampling rate. As highly objective physiological signals, EEG has demon- strated remarkable potential in seizure epilepsy classification (Boonyakitanont et al. (020), acute stress detection (Sharma et al., 2022), sleep stage classification (Aboalayon al. (016)). motor im- agery recognition (Amin et al. 2019), abnormal identification (Roy et al. (2019). emotion analysis (Suhaimi et al. 2020), and auditory attention detection (Biesmans al. 2016).
Numerous deep learning models have been proposed to address the aforementioned tasks in their respective fields. Some works apply convolutional neural networks (CNN) across and within raw EEG channels to encode spatial and temporal features (Lawhern et al. 2018), while others prepro- cess the data using short-time Fourier transform (STFT) and employ Graph Neural Network (GNN)
"Li-Ming Zhao and Bao-Liang Lu co-corresponding authors.

---

Published as a conference paper at ICLR 2024
on the resulting spectrograms to obtain semantic features of brain area links (Song et al. 2018
Researchers also segment the signal and use a CNN segment encoder with a downstream sequence model such as recurrent neural networks (RNN) to capture temporal dynamics (Xu et al. 2020
These models primarily focus on EEG samples that adhere to specific task formats, mainly because the equipment used to collect EEG differs between datasets, which introduces mismatched channels and variable lengths. Meanwhile, EEG data collection is quite expensive, which makes it challeng- ing to build large EEG datasets specifically designed for particular task. To prevent overfitting, the parameters of these models need to be regulated, which in turn hampers the model's ability to learn EEG expressions and limits its generalizability. Consequently, we discovered that current EEG models are typically proprietary and lack the capacity to perform cross-task learning.
Recently, we have been impressed by the capabilities of L.LSs (Ouyang et al., 2022, Wei et al, 2022). Specifically. Transformer- models have demonstrated promising results in natural lan- guage processing tasks, which highlights the potential of self-supervised pre-training as a means for harnessing large-scale data. These masked language modeling tasks involve randomly mask- ing some proportion of tokens within a text and then recovering the masked tokens based on the Transformer encoding results of the corrupted text. Motivated by these methods, we propose to ap- ply reconstruction ideas to pre-train neural Transformers. However, it is a daunting task to directly apply LLM-style pre-training to EEG data. The challenges are summarized follows:
1) Lack of sufficient EEG data. The acquisition of ERG data is significantly challenging compared to natural language and image data. Moreover, the annotation of EEG data usually requires a lot of effort on the part of experts in the corresponding field, thus leading to the fact that only small labeled datasets exist for specific tasks in BCI, where EEG signals are often collected from a small number of participants, typically less than tens of hours in duration. As result, there is currently no single EEG dataset that is large enough to support the training of LEMs. It remains problems Q1: how to utilize large-scale unlabeled EEG data? and Q2: how much data is needed to train LEMs?.
2) Diverse configurations of EEG collection. Despite the availability of the international 10-20 system to ensure standardization in EEG testing, users may choose to collect data using EEG caps with different electrode numbers or patch electrodes based on their practical application needs. Thus, how to handle the diverse formats of EEG data in order to match the input units of neural Trans- formers remains a significant research endeavor.
3) Lack of effective EEG representation learning paradigm. Low signal-to-noise ratio (SNR) and different types of noise are the greatest challenges. Additionally, balancing temporal and spatial characteristics crucial for effective EEG representation learning. Despite the availability of various deep learning-basee based EEG representation learning paradigms, such as CNN, RNN, and GNN, for raw EEG data, many researchers still prefer to design artificial EEG features due to these challenges.
In this paper, our objective is to devise a versatile large EEG model that can efficiently handle diverse EEG datasets with varying channels and lengths. By utilizing unsupervised training on substantial amount of EEG data, we envision the model to possess universal EEG data comprehension capa- bilities, enabling it quickly adapt to various EEG downstream tasks. We collected over 2,500 hours of diverse EEG data across various tasks and formats from about 20 datasets. These datasets were primarily obtained from publiely available EEG datasets, as well as our own collected EEG data. Raw EEG signals were first segmented into EEG channel patches to deal with the issues of variant electrodes and time length. Vector-quantized neural spectrum prediction used to train a semantically rich neural tokenizer to generate neural vocabulary. Specifically, the tokenizer was trained by predicting the Fourier spectrum of the original signal. During pre-training, part of EEG patches are masked while the objective of the neural Transformer is to predict masked tokens from visible patches. We pre-trained three models with varying parameter sizes, ranging from 5.8M to 369M, which are the largest models in BCI ever, and fine-tuned them on four distinct types of down- stream tasks encompassing both classification and regression. The contributions of this work are summarized follows:
Large-scale EEG pre-training We collected and pre-trained a large-scale neural Transformer model on more than 2,500 hours of diverse EEG data. As far as we know, this is the first time such extensive and varied datasets have been utilized for EEG pre-training.
Being compatible with various EEG configurations. LaBraMs are unified models that are able to handle EEG signals with various channels and time lengths with the assistance of the flexible
2

---

Published as a conference paper at ICLR 2024
Temperal Finhalding
Tempporal
Output Embedding
Imput EBG Signals
Crmup Norm
Adg
Fee Evessed
Add Norm
Cony
Spatial
Eimeedding
The overall architecture of LaBraM, i.e., neural Transformer. All input EEG signals will first be segmented into EEG patches through a fixed-length time window, and then temporal encoder will be applied to each patch to extract temporal features. Afterward, temporal and spatial embeddings are added to the patch features to carry temporal and spatial information. At last, the sequence of embeddings is passed into the Transformer encoder by patch-wiss attention to obtain the final output.
Figure I:
spatial and temporal embeddings. Hence, one pre-trained LaBraM can adapt to any downstream dataset with different configurations.
Effective EEG representation learning. The utilization of the neural Transformer allows the model to effectively capture both temporal and spatial features EEG signals with varying chan- nels and lengths, making it suitable for wide range of downstream tasks in EEG analysis. We further define a neural codebook that offers a compact, versatile, and meaningful representation of EEG signals. We resolve Q1 by leveraging this codehook pre-train LaBraM by masked EEG modeling. The empirical performance demonstrates the effectiveness of our proposed method and paves the way for further development in aligning this codebook with natural language.
Comprehensive experiments on downstream datasets. We evaluate our LaBraMs four rep- resentative downstream tasks in BCI, where they surpass all SOTA methods by a large margin. Additionally, conduct experiments answer Q2 by scaling the pre-training data size and conclude the amount of pre-training data required for models different sizes in Section
METHOD
In this section, we detail the whole framework of LaBraM. first formulate the multi-channel
EEG signals X RCXT, where C' is the number of EEG electrodes (channels) and T is the total timestamps. The electrode set is formulated as Cx = {c,, Czy...., ... where Cx S (c1, (2, (0,,2,,9cc111 and C is the universal set of channels in the international 10-20 system.
2.1 MODEL ARCHITECTURE
We introduce the neural Transformer, general architecture for decoding EEG signals that can deal with any input EEG signals with arbitrary number of channels and time length, as illustrated in Fig The key operation for achieving this is segmenting the EEG signals into patches, inspired by patch embeddings in images (Dosovitskiy
al., 202T). Assume that the timestamp for each sample is and the stride is s. X can be segmented into [-=]+
+ 1 samples, and each sample Cxt, We use w-length window without overlap to segment each EEG channel into patches, obtaining # = {Fay.e E R" 1,2, ..., C, = 1, ...; [_ The total number of the patches a
|x|| -C CL
Temporal Encoder: As EEG is of high resolution in the temporal domain, it is vital to extract tem- poral features before patch-wise interaction by self-attention. We employ a temporal encoder which consists of several temporal convolution blocks to encode each EEG patch into a patch embedding. The temporal convolution block is composed of 1-D convolution layer, a group normalization layer (Wu & He, 2018), and a GELU activation function (Hendryeks & Gimpel 2016). We denote the output patch embeddings from the temporal encoder as
t
e = =caa, & E RdIj = 1,2, 2,... C. = 1,2,...
(1)
w
3

---

Published as a conference paper at ICLR 2024
Neural Tokenizer Training
Noural Codebook
Vl-2
Lookup
Reploce
Fourier Spectrum
Tekenizer
Neural
748
Doeder
Naumi
603
Temporal Spotial Embooddng
Temporal Fucodar
Trumsformre Block:
Tranffimmst Block
Trinfommtt Block
7488
460
191
514
Symmetric Mask
Pre-truining
Figure 2: Overview of neural tokenizer training and LaBraM pre-training. Up: We train a neural tokenizer discretize EEG signals into discrete neural tokens by reconstructing the Fourier spec- trum. Down: During pre-training, part of EEG patches are masked while the objective is to predict masked tokens from visible patches.
where d the dimension of the embeddings.
Temporal & Spatial Embedding. In order to enable the model to be aware of the tempo- ral and spatial information of patch embeddings, we initialize temporal embedding list TE {tei, tetmaa) } and spatial embedding list SE ( seiet) both of which are d- dimension and are set learnable during training. Note that tmaa is the hyperparameter determining the maximum number of time patches and | tmaz. Meanwhile, for each channel c, find its corresponding spatial embedding se; in the spatial embedding list SE. Thus, given one ar- bitrary output embedding Ccy, , in Equation from the temporal encoder, add the corresponding temporal and spatial embeddings to it:
{eas, + teg + se,,li==11 1,2,,,, C, 1,2
(2)
where temporal and spatial embeddings act as absolute position encoding.
Transformer Encoder. Finally, the sequence of embeddings will be directly fed into the Trans- former encoder (Vaswani al. 2017). To make the training of Transformer more stable and effi cient, we incorporate some modifications (Dehghani et al. (202)). First, we add layer normalization to the queries and keys before the dot-product attention mechanism, which avoids over-large values in attention logits:
Attention K, K)V) soflmaas LN(Q)LN(K)² V
ddhead
where dhead is the dimension of one head the multi-head attention and LN denotes the layerNorm (Ba et al.) 2016). Next, we omit the bias term in QKV computations, which accelerates the training without performance degradation. For downstream tasks, we use average pooling on the output embeddings followed by task-specific prediction heads.

---

Published as a conference paper at ICLR 2024
2.2 NEURAL TOKENIZER TRAINING
Prior to pre-training LaBraM through masking and prediction, we need to tokenize the EEG into discrete tokens. We propose the vector-quantized neural spectrum prediction, which is trained by predicting the Fourier spectrum, as shown in Figure 2 The key components are the neural tok- enizer which encodes EEG samples into patch representations and the neural decoder which de- codes the Fourier spectrum from neural embeddings. The idea is basically inspired by VQ-VAE (Van Den Oord et 2017) which encodes images into discrete latent representations.
K} E RK*D, where K is the number of the discrete neural embeddings and D the dimensionality of each embedding. Given an EEG signal sample the neural tokenizer whose backbone is just described in Section 2.1 first encode to patch representations p = {pili = 1, ,..,, N), where = CL After that, we utilize quantizer quantize all the patch representations into the neural codebook embeddings. The codebook looks up the nearest neighbor of each patch ps in the neural codebook V. This procedure can be formulated as
= 1, Neural Tokenizer. We define a neural codebook V = 1,
2 = arg min [[(2(p;) - f2(2))ll2.
(4)
j
where & represents & normalization and 2; is the quantized vector after the quantizer. This is equivalent to finding the closest neural embedding by cosine similarity and such Eg normalization improves the codebook utilization (Peng et al. 2022).
Fourier Spectrum Prediction. Unlike images that of high signal-to-noise ratio, EEG signals are of low signal-to-noise ratio and have characteristics of apparent stochasticity, nonstationarity. and nonlinearity nature, which make it hard to reconstruct the original signals well (Moss et al. 2004). In our previous experiments, the loss fails to converge while directly reconstructing raw EEG signals. Instead, the frequency and phase distribution from the Fourier spectrum of EEG signals reveals the underlying neurophysiological activities of the brain (Wu et al., [002). Therefore, we propose to reconstruct the amplitude and phase from discrete neural tokens for training the neural tokenizer and neural decoder. For an EEG patch F.,k = [1).), of channel and time in a sample z, we apply the Discrete Fourier Transform (DFT) as follows
2Tj N
Σ
x|n) exp(
mn),
(5)
where E [1, N] and j is the imaginary unit. We rewrite Equation 5 Jusing Euler's formula as
N
2x mn) - jæ((x] sin mn).
2T
x Iin]
cos(
(6)
N
N
Note that TH indicates the spectrum of the sequence at frequency wm amplitude and phase can calculated as
2mm
Consequently, the
N
A
V Re(Bm)) + Im Im())))
Im (mmm
(T
6th = arctan(
(8)
Re(a
where Re and Im stand for the real and imaginary parts of a complex number. worthwhile to mention that adopt z-score normalization to normalize A and 8t within sample for stable convergence.
After being tokenized by the quantizer, the normalized discrete neural embeddings (62(vz,) N,....N} are passed into the neural decoder that comprises several Transformer blocks. The output representations are aggregated by average pooling followed by two specific prediction heads to regress the spectrum amplitude a and phase 08, respectively. The mean squared error (MSE) loss is utilized to guide the prediction. Ultimately, the total loss for training the vector-quantized neural spectrum prediction is defined as
N
C
Σ RED 1=1
of
A 1o + (gg(()()))))) -l - (v..)(v2.))11 (9)
5

---

Published as a conference paper at ICLR 2024
where D is all EEG data and sg represents the hoopggradient operation that is defined an identity at the forward pass and has zero gradients. To make the codebook update more stable, we employ the exponential moving average strategy (Van Den Oord et al. 2017).
2.3 PRE-TRAINING LABRAM
Masked EEG Modeling. To enforce LaBraM learning generic representations with tremendous EEG data, we propose masked EEG modeling. The whole procedure is presented in Figure 2 As formulated in Section 2.1 given an EEG sample z, the temporal encoder first transforms it to patch embeddings e [ = N}. We randomly generate mask . {m,lé 1, 1.... N)
where E {0. }} with proportion of m is 1. After that, we replace the masked patches of with the learnable mask token C& TR The corrupted EEG patches can be denoted as e {e; m. Oli I...., N {eM m, = 1/i L, N which will be added by temporal and spatial embeddings, and then fed into Transformer encoder. We denote the output hidden vectors as h = {hali I,... N), which used predict the corresponding neural tokens through linear classifier:
p(o' (oo sof soflmaar(innea(())
(10)
Our objective training loss
LM=
Σ DED m,=1
logp(uileM)
(11)
Symmetric Masking, We further propose symmetric masking strategy to improve training effi ciency. We calculate the inverse of the generated mask M, obtaining M f~ m,li I,..., NN. Similarly, we use the new mask A to perform the masked EEG modeling, obtaining the masked EEG prediction loss Ltum. The motivation is from two aspects: 1) Since we introduce the neural tokenizer, there will be an extra computation overhead, i.e., one forward pass for each EEG sam- ple. Thus, the symmetric masking reuses the same discrete representations, thus improving training efficiency. 2) The symmetric masking provides more masking perspectives in one batch, increas- ing the data divergency. This simple strategy boosts downstream performance as demonstrated in [ppendix[]
Finally, the overall training objective for pre-training LaBraM is
= Lm + Laym
(12)
EXPERIMENTS
3.1 EVALUATION DATASETS
We systematically evaluate our LaBraM on the following downstream datasets:
TUAB (abnormal detection) (Obeid & Picone 2016): corpus of EEGs which are 23-channel and sampled at 256 Hz. All data have been annotated as normal or abnormal. There are total 409,455 10-second samples that we use for binary classification to predict normal/abnormal.
TUEV (event type classification) (Obeid & Picone 2016): This corpus a subset of TUEG that contains annotations of EEG segments as one of six classes: (1) spike and sharp wave (SPSW), (2) generalized periodic epileptiform discharges (GPED), (3) periodic lateralized epileptiform dis- charges (PLED), (4) eye movement (EYEM). (5) artifact (ARTF) and (6) background (BCKG). The EEG signals contain 23 channels 256 Hz and are segmented into 112,491 5-second sam- ples.
More experimental results on other BCI tasks can found Appendix
3.2 EXPERIMENT SETUP
Model Variants. We devise three different configurations of LaBraM: LaBraM-Base, LaBraM- Large, and LaBraM-Huge. The number of parameters is 5.8M for LaBraM-BB Basee 46M for LaBraM- Large, and 369M for LaBraM-Huge, respectively, which is increased by enlarging the depth of

---

Published as a conference paper at ICLR 2024
the Transformer encoder and hidden sizes. More details of the architecture settings are listed in Appendix C Unless otherwise specified, the results are from LaBraM- LaBraM-BBse in this paper.
The time window w of patch is set to 200 (1 second). To ensure stable computing resource usage, the number of patches (sequence length) is limited to 256. That means, for example, the time length of EEG with 64 (32) channels is set 4 (8) seconds. As for the window stride (data stride), it is set to 4 seconds in order to cover all training data as well as boost the training speed.
Pre-training & Fine-tuning. For pre-training LaBraM and the vector-quantized neural spectrum prediction, we collect total time of over 2,500 hours from public datasets and our self-collected data as described in Appendix Note that the four downstream datasets excluded from the pre- training datasets. For the data splitting of TUAB and TUEV, we strictly follow the same strategy as BIOT (Yang al., 2023a) to compare all methods fairly, Specifically, separation is provided by the datasets, we divide the training patients into training and validation groups by 80% and 20%, respectively. We employ binary cross-entropy (BCE) loss for TUAB (binary classification) and cross-entropy loss for TUEV (multi-class classification), respectively. Our experiments are conducted on eight A800 GPUs by Python 3. 1.4 and PyTorch 2.0.1 CUDA 11.8. The best models are trained based on the training set, selected from the validation set, and finally evaluated on the test set. We report the average and standard deviation values on five different random seeds to obtain comparable results. (sce Appendix forr more detailed hyperparameters))
the training and test
Preprocessing. We only employ very little of the necessary preprocessing. We first filter the EEG signals between 0.1 Hz and 75 Hz to remove low- wwwfrrquency noise. Then, notch filter of 50 Hz is applied to avoid power-line interference. Finally, all EEG signals are resampled to 200 Hz. As the range of EEG value is typically between -0.1 mV to 0.1 mV, we normalize it by setting the unit to 0.1 mV to guarantee the value mainly between -1 to 1.
Baselines & Metrics. The baselines are from Yang et al. (2023a), where we choose the best results to compare with. We use the following metries for comparison: 1) Balanced Accuracy: the average of recall on each class, which is utilized for both binary and multi-class classification. 2) AUC-PR: area under the precision-recall curve for binary classification. 3) AUROC: area under the receiver operating characteristic curve, which is used for binary classification as well. 4) Cohen's Kappa: a measure of agreement between categorical variables X and Y, which is calculated from the observed and expected frequencies on the diagonal of a square contingency table.
used for multi-class classification. 5) Weighted Fl: A harmonic mean of the precision and recall, where the relative contribution of precision and recall to the Fl score are equal. We use it to evaluate multi-class classification. We set AUROC as the monitor score for binary classification and Cohen's 's Kappa as the monitor score for multi-class classification.
3.3 PRE-TNAINING VISUALIZATION
Pre-training Loss
Masked EEG Modeling Accuracy
Base
15-
Large
0.4
Huge
Accuracy
Loss
10-
0.2
Base
Large
Huge
5
0.0
0
10
20
30
40
50
10
20
30
40
50
Epochs
Epochs
Figure 3: The pre- pre-training loss curve and masked EEG modeling accuracy curve.
Figure 3 compares the convergence curves of the total pre-training loss and masked EEG modeling accuracy between the base, large, and huge models. We observe that a larger model with more parameters converge
smaller loss and higher accuracy. Notably, the loss of the huge model seems to have an obvious downward trend while the accuracy tends to increase if we train it longer. This observation suggests scaling up the model size has the potential to obtain better performance.
7

---

Published as a conference paper at ICLR 2024
3.4 COMPARISON WITH STATE-OFE OF-THE
Table I and Table 2 present the results of state-of-the-art baselines as well as LaBraM from TUAB and TUEV. The results demonstrate that LaBraM-Base model outperformed all baselines on ious evaluation metrics for both tasks. Particularly in the more challenging multi-class classification task of TUEV, our model achieved a sigmificant improvement performance, In our own model, observed that as the number of model parameters increased, the LaBraM-Huge model performed the best, followed by the LaBraM-Large model and then the LaBraM- Maree model. We attribute this good performance to the increase in pre-training data volume and model parameters. We be- lieve that with sufficient data volume, large-scale EEG models can learn more generalizable EEG patterns, leading to improved performance wide range of downstream tasks in EEG analysis.
Table 1: The results of different methods on TUAB.
Methods
Model Size
Balanced Accuracy
AUC-PR
AUROC
SPaRCNet JJin
2023
0.79M
0.78%::0.0018
0.8414=0.0018
0.8676#0.0012
ContraWR Yang al. 2023b
1.6M
0.77462:0 0.0041
0.8421=000104
0.8456:0.0074
CNN-Transiommer [Pch al. 2022
3.2M
0.7777+00.002
0.8433=0.0039
0.8461+0.0013
FFCL et al. 2022)
2,4M
0 0038
0.8448=0.0065
0.856910.0051
ST-Tramslormee (Song el al.
2021
3.5M
0.7966:1: 0023
0.8521=0.0026
0.8707100.009
BIOT Yang et al. 2023a
3.2M
0.7959+0.0057
0.8092=000023
0.8815=0.0003
LaBraM-Baae
5.8M
0.8140+0.000
0.8965=0.0016
0.9022+000009
LaBraM-Lard Large
46M
0.822610.000 0.0015
0.0005
0.912710.0005
LaBraM-Huge
369M
0.8258+0.0011
0.9204-0.001
0.9162+000016
Table 2: The results of different methods on TUEV.
Methods
Model Size
Balanced Accuracy
Cohen's Kappa
Weighted F1
SPaRCNet (Jing el al. 2023
0.79M
0.416110.0262 0,4384+0.0034
0.4233=0.0181
0.702410.0104
ContraWR (Yang et al. b023b
1.6M
0.3912=0.0237
0.6893+0.0136 36
CNN-TYanstormer [Fch et al.
2022
3.2M
0.4087++0.016
0.3815=0.0134
0.6854+0.0299
FFCL (Li et al. 2022
2.4M
0.3979- 0104
0.3732=0.0188
0.6783+10.0120
ST-TTransoormrr (Song et al. 2021
3.5M
0.3984++0.02 0.0228
0.3765=0.0306
0.6823+0.019
BIOT (Yang et af. 2023a
3.2M
0.5281 +000225
0.5273=0.0249
0.7492+000082
LaBraM-Baa -ase
5.8M
0.6409+0.0000
0.6637-0.0093
0.8312+0.0052
LaBraM-Larg
46M
0.6581+0.0156
0.6622-000136
0.8315++0.0040
LaBraM-Huge
369M
0.6616:10.0170
0.6745=0.0195
0.8029+00.0886
3.5 PRE-TRAINING WITH/WITHOUT DOWNSTREAM DATASETS
During the pre-training process, we hope that the model can learn general EEG representations that are not specific to any particular task. Although no label data is used during the pre-training process,
TUAB
1.0
1.0
truin wie DD
pre train Weo DD pre-train DD
pre-truis
0.8
0.8
0.6
0.6
0.2
0.2-
0.0
Balanced BalancedAccumm Acouracy
AUC
AUROC
0.0
Balanced Acouracy
Coben's Kaypa
Weighted P]
Figure 4: A comparison of the model's performance on the TUAB and TUEV datasets when incor- porating themselves into the pre-training process or not.
8

---

Published as a conference paper at ICLR 2024
to eliminate the influence of the pretraining data downstream tasks, we compared the results with or without incorporating the downstream task dataset into the pre-training process or not. is noted that the recordings of TUAB and TUEV are disjoint from recordings of pre-training datasets. As Figure illlustratss the performance of the model on the downstream task was not significantly affected by whether or not to incorporate the downstream task datasets into the model's pre-training process. This demonstrates that our model has the capability to learn universal EEG representations, and provides guidance for the collection of more EEG data in the future. In other words, we do not need to expend significant amount of effort on labeling EEG data during the pre-training process.
3.6 SCALING DATA SIZE
Although we have collected approximately 2,500 hours of EEG data, it still relatively small com- pared to the sample size in natural language processing and image processing. We answer Q2 about the demand for data size to train LaBraMs with different sizes by scaling the pre- pre-training data size. As illustrated in Figure(3] the performance of the Base model with 500 hours of training exceeds that of the 2500-hour model on TUAB, while approaching over 90% of the 2500-hour performance on TUEV. For the Large model, performance generally improves with increased data volume, though the growth rate slows after 1000 hours. In contrast, the Huge model exhibits a noticeable upward trend in performance as data size continues to expand. Therefore, we believe that with further expan- sion of the dataset, our model can achieve better performance. The question of how much EEG data required for pre-training a large EEG model undoubtedly an important issue worth exploring in this field. Nevertheless, 2,500 hours is not the answer to this question at least. Our observation basically follows the scaling law (Kaplan et al. 2020), from which we deduce that the Huge model would continue to perform better with the data size on the order of at least ten thousand hours.
TUAB (Bolazeed
TUAB (AUCC-PR)
TUAB (AUROC)
Dau
Buc
Basc
1.11
339
3.53
8:15
6.7
3.5
0.51
0.34
(90
1168
2006
2546
159)
200)
259
1393
3109
3591
TUEV
TUEV (Wiighoed
3.55
Bose
Bise
Bose
04
0.5
03
(kmm
Tiae
Time
Figure 5: A comparison of the performance of the Base model, Large model, and Huge model on the TUAB and TUEV datasets as the pre-training data increases.
4 CONCLUSION
This paper proposes Large Brain Model (LaBraM) that learns universal embeddings through unsu- pervised pre-training over 2,500 hours of diverse EEG data. The LaBraM is capable of handling diverse EEG datasets due to the segmentation of raw EEG signals into channel patches and the use of vector-quantized neural spectrum prediction generate rich semantic tokenizer during pre- training. Additionally, the neural Transformer architecture enables effective representation learning of both temporal and spatial features of EEG signals, making it suitable for a wide range of down- stream tasks in EEG analysis. The LaBraM was validated on multiple downstream tasks, including abnormal detection, event type classification, emotion recognition, and gait prediction. Our exper- iments show that the LaBraM outperforms all SOTA methods in their respective fields. In the end, we hope our work our can have implications for future developments in EEG-bar based deep learning models with improved perceptual capabilities and generalizability.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported in part by grants from STI 2030-Major Projets+2002ZD00085500 Shang- hai Municipal Science and Technology Major Project (Grant No. 2021SH2D ZX), Medical- Engineering Interdisciplinary Research Foundation of Shanghai Jiao Tong University "Jiao Tong Star" Program (YG20232D25), and GuangCi Professorship Program of RuiJin Hospital Shanghai Jiao Tong University School of Medicine.
REFERENCES
Khald Ali Aboalayon, Miad Faezipour, Wafaa Almuhammadi, and Sacid Moslchpour. Sleep stage classification using EEG signal analysis: comprehensive survey and new investigation. Entropy, 18(9):272, 2016.
Syed Umar Amin, Mansour Alsulaiman, Ghulam Muhammad, Mohamed Amine Mekhtiche, and M Shamim Hossain. Deep Learning for EEG motor imagery classification based on multi-layer CNNs feature fusion. Future Generation Computer Systems, 101:542-554, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. Advances Neural Information Processing Systems, 13:12449-12460, 2020.
Hubert Banville, Omar Chehab, Aapo Hyvärinen. Demis-Alexander Engemann, and Alexandre Gramfort. Uncovering the structure of clinical EEG signals with self-supervised learning. Journal of Neural Engineering, 1844)004000,, 2021.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image trans- formers. In International Conference Learning Representations, 2022. URL https:
//openreview. net hww.ruewwww..eee/ooooaeeeeee.ee Bassppp-hh225o4
Wouter Biesmans, Neetha Das, Tom Francart, and Alexander Bertrand. Auditory-inspired speech envelope extraction methods for improved EEG-based auditory attention detection cocktail party scenario. IEEE Transactions Neural Systems and Rehabilitation Engineering, 25(5):
402-412, 2016.
Benjamin Blankertz, Guido Dornhege, Matthias Krauledat, Klaus- Müller, and Gabriel Curio. The non-invasive berlin brain-computer interface: fast acquisition of effective performance in untrained subjects. Neurolmage, 37()):339-55) 2007.
Poomipat Boonyakitanont, Apiwat Lek-Uthai, Krisnachai Chomtho, and Jitkomut Songsiri. A view of feature extraction and performance evaluation in epileptic seizure detection using EEG. Biomedical Signal Processing and Control, 57:101702. 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 13:1877-1901, 2020.
G Buckwalter, S Chhin, S Rahman, Obeid, and Picone. Recent advances in the TUH EEG corpus: improving the interrater agreement for artifacts and epileptiform events. 2021 IEEE Signal Processing in Medicine and Biology Symposium (SPMB), pp. 1-3. IEFE, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pp. 1691- 1703. PMLR. 2020.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang. Carlos Riquelme Ruiz, Matthias Minderer, Joan Puigcerver, Utku Evei, Manoj Kumar, Sjoerd Van
10

---

Published as a conference paper at ICLR 2024
Steenkiste, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Collier, Alexey A. Gritsenko, Vighnesh Birodkar, Cristina Nader Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetic, Dustin Tran, Thomas Kipf, Mario Lucic, Xiaohua Zhai, Daniel Keysers, Jeremiah J. Harmsen, and Neil Houlsby. Scaling vision transformers 22 billion parameters. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Pro- ceedings of the 40th International Conference on Machine Learning, volume 202 of Proceed- ings of Machine Learning Research, pp. 7480-7512. PMLR. 23-29 Jul 2023. URL https: html
//proceedings. mlr. wttpss/vw22/dehgggeeeeee
Paolo Detti, Giampaolo Vatti, and Garazi Zabalo Manrique de Lara. Eeg synchronization analysis for seizure prediction: A study on data of noninvasive recordings. Prucesses, 8(7):846, 2020.
Jacob Devlin, Ming-Wet Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recogni- tion at Scale. In International Conference on Learning Representations, 2021. URL https: //oeeneview wwt./eetttttniee YichE wwiccccNFTTyyyy.ccc
Xiao Gu. Jinpei Han, Guang-Zhong Yang, and Benny Lo. Generalizable Movement Intention Recognition with Multiple Heterogeneous EEG Datasets. In 2023 IEEE International Confer- ence on Robotics and Automation (ICRA), pp. 9858-9866 2023. doi: 10.1 10.1109/CCRA48891.2200 10160462.
Jinpei Han, Xiaoxi Wei, and A Aldo Faisal. EEG Decoding for Datasets with Heterogenous Electrode Configurations using Transfer Learning Graph Neural Networks. arXiv preprint vrXiv:2306.13109 2023.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Yongtian He, Trieu Phat Luu, Kevin Nathan, Sho Nakagome, and Jose L Contreras- Vidal. mobile brain-body imaging dataset recorded during treadmill walking with brain-computer interface. Scientific Data, 5(1):1-10, 2018.
Dan Hendryeks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: (60::660..00 15, 2016.
Wei-Bang Jiang, Li-Ming Zhao, Ping Guo, and Bao-Liang Lu. Discriminating Surprise and Anger from EEG and Eye Movements with a Graph Network. In 2021 IEEE International Conference
Bioinformatics and Biomedicine (BIBM), pp. 1353-1357, 2021. doi: 10.1109/BIBM52615.
2021 9669637.
Wei-Bang Jiang, Xuan-Hao Liu, Wei-Long Zheng, and Bao-Liang Lu. Multimodal Adaptive Emo- tion Transformer with Flexible Modality Inputs on A Novel Dataset with Continuous Labels. In Proceedings of the 31st ACM International Conference on Multimedia, MM '23, pp. 5975-5984, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701085. doi: 00.1145/5581783.3637777
URL btRp:ttps: //oi: org/10. /ttp::/:///oo.oo//.....oo 010.....11111515111333333333
Jin Jing, Wendong Ge, Shenda Hong, Marta Bento Fernandes, Zhen Lin, Chaoqi Yang. Sungtae An, Aaron Struck, Aline Herlopian, Ioannis Karakis, et al. Development of expert-level classifica- tion of seizures and rhythmic and periodic patterns during eeg interpretation. Neurology, 100(17); e1750-c1762. 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.

---

Published as a conference paper at ICLR 2024
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Louis Korezowski, Martine Cederhout, Anton Andreev, Grégoire Cattan, Pedro Luiz Coelho Ro- drigues, Violette Gautheret, and Marco Congedo. Brain Invaders calibration P300-based BCI with modulation of flash duration Dataset (bi2015a). Research report, GIPSA-lab, July 2019.
URL
Demetres Kostas, Stephane Aroca-Quellae teeeletu,,,, and Frank Rudzicz. BENDR: using transformers and contrastive self-supervised learning task learn from massive amounts of EEG data. Frontiers in Human Neuroscience, 15:653659, 2021.
Vernon J Lawhern, Amelia Solon, Nicholas R Waytowich, Stephen M Gordon, Chou Hung, and Brent J Lance. EEGNet: a compact convolutional neural network for EEG-based compter interfaces. Journal of Neural Engineering, 15(5):05013, 2018.
Hongli Li, Man Ding, Ronghua Zhang, and Chunbo Xiu. Motor imagery EEG classification algo- rithm based on CNN-LSTM feature fusion network. Biomedical Signal Processing and Control, 72:103342, 2022.
Rui Li, Le-Dian Liu, and Bao-Liang Lu. Discrimination of Decision Confidence Levels from EEG Signals. In 2021 10th International IEEE/EMBS Conference on Neural Engineering (NER), pp. 946-949, 2021. doi: 10.1109/EE499233.2220 2.21 2021.9941088
Rui Liu, Yuanyuan Chen, Anran Li, Yi Ding, Han Yu, and Cuntai Guan. Aggregating intrinsic infor- mation to enhance BCI performance through federated learning. Neural Networks, pp. 106100, 2024.
Wei Liu, Jie-Lin Qiu, Wei-Long Zheng, and Bao-Liang Lu. Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition. IEEE Transactions on Cognitive and Developmental Systems, 2021.
Wei Liu, Wei-Long Zheng, Ziyi Li, Si-Yus Yu-a Wu, Lu Gan, and Bao-Liang Lu. Identifying similarities and differences emotion recognition with EEG and eye movements among Chinese, German, and French People. Journal of Neural Engineering, 19(2):02601,, 2022.
Matthew D Luciw, Ewa Jarocka, and Benoni B Edin. Multi-channel EEG recordings during 3,936 grasp and lift trials with varying weight and friction. Scientific Data, 1(1):1-11, 2014.
Shuai Luo, Yu-Ting Lan, Dan Peng, Ziyi Li, Wei-Long Zheng, and Bao-Liang Lu. Multimodal emotion recognition in response oil paintings. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine Biology Society (EMBC), pp. 4167-4170, 2022. doi:
10.1 (0.1109/MBBC4822 000.2.222222222.. 1630.
Perrin Margaux, Maby Emmanuel, Daligault Sébastien, Bertrand Olivier, and Mattout Jérémie. Ob- jective and subjective evaluation online error correction during p300-based spelling. Advances in Human- Comput- Interaction, 2012:4 2012.
Frank Moss, Lawrence M Ward, and Walter G Sannita. Stochastic resonance and sensory informa- tion processing: a tutorial and review of application. Clinical Neurophysiology, 11(22)2267-281, 2004.
Iyad Obeid and Joseph Picone. The temple university hospital EEG data corpus. Frontiers in Neuroscience, 10:196, 2016.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama. Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Wei Yan Peh, Yuanyuan Yao, and Justin Dauwels. Transformer convolutional neural networks for automated artifact detection in scalp EEG. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 3599-3602, IEFE, 2022.
12

---

Published as a conference paper at ICLR 2024
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint S.Xiv:2208.066666 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Subhrajit Roy, Isabell Kiral-Komee, and Stefan Harrer. ChronoNet: for abnormal EEG identification. In Artificial Intelligence in Medicine: 17th Conference on Artificial Intelligence in Medicine, AIME 2019, Poznan, Poland, June 26-29, 2019, Proceedings 17, pp. 47-56. Springer, 2019.
deep recurrent neural network
Arman Savran, Koray Ciftci, Guillame Chanel, Javier Cruz Mota, Luong Hong Viet, Bülent Sankur, Lale Akarun, Alice Caplier, and Michele Rombaut. Emotion detection in the loop from brain signals and facial images. In eINTERFACE SATTEERFAACEOSSMMIIILAA NoE Summer Workshop on Multimodal Interfaces, 2006.
Gerwin Schalk, Dennis J McFarland, Thilo Hinterberger, Niels Birbaumer, and Jonathan R Wol- paw. BCI2000: a ceneral-purposse brain-computer interface (BCT) system. IEEE Transactions on Biomedical Engineering, 51(6):1034-144 2004.
Vinit Shah, Eva Von Weltin, Silvia Lopez, James Riley McHugh, Lillian Veloso, Meysam Gol- mohammadi, lyad Obeid, and Joseph Picone. The temple university hospital seizure detection corpus. Frontiers in Neuroinformattic, 12:83, 2018
Lakhan Dev Sharma, Vijay Kumar Bohat, Maria Habib, Al-Zoubi Ala'M. Hossam Faris, and Ibrahim Aljarah. Evolutionary inspired approach for mental stress detection using EEG signal. Expert Systems with Applications, 197:1 16634, 2022.
Tengfei Song, Wenming Zheng, Peng Song. and Zhen Cui. EEG emotion recognition using dy- namical graph convolutional neural networks. IEEE Transactions on Affective Computing, 11(3): 532 541, 2018
Yonghao Song, Xueyu Jia. Lie Yang, and Longhan Xie. Transformer- ppatial-tempore feature learning for EEG decoding. arXiv preprint 1xXh::200..1110,, 2021.
Nazmi Sofian Suhaimi, James Mountstephens, Jason Teo, et al. EEG-based emotion recognition: A state-of-the-ar review of current trends and opportunities. Computational Intelligence and Neuroscience, 2020, 2020.
Le-Yan Tao and Bao-Liang Lu. Emotion Recognition under Sleep Deprivation Using a Multimodal Residual LSTM Network. In 2020 International Joint Conference on Neural Networks (IJCNN, pp. 1-8, 2020. doi: 10. 1099IINNN46055 200.44866652200.9990 9206977...
Mastaneh Torkamani-Arze Azar, Sumeyra Demir Kanik, Serap Aydin, and Mujdat Cetin. Prediction of reaction time and vigilance variability from spatio-spectral features of resting-state EEG in long sustained attention task. IEEE Journal of Biomedical and Health Informatics, 255522500-255,,, 2020.
Logan Trujillo. Raw EEG Data. 2020. doi: 10.18738/TV/S22HHB. URL httpss://ci..org 10.18738/T8/552N1
Logan T Trujillo, Candice Stanfield, and Ruben D Vela. The effect electroencephalogram (EEG) reference choice on information--hhorrticc measures of the complexity and integration of EEG signals. Frontiers in Neuroscience, 11:425, 2017.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017.
13

---

Published as a conference paper at ICLR 2024
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gareett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https::/procees:: htt.s//fee2243666111100000ooaaaa Paper pdf
wwtp:rriipppcpppppppeeeeepepeeee fi1es/paper/2211/
L Veloso, J McHugh, E von Weltin, S Lopez, I Obeid, and J Picone. Big data resources for EEGs: Enabling deep learning research. In 2017 IEEE Signal Processing Medicine and Biology Sym- posium (SPMB), pp. 1-3. IEEE, 2017.
Eva Weltin, Tameem Ahsan, Vinit Shah, Dawer Jamshed, Mcysam Golmohammadi, Iyad Obeid, and Joseph Picone. Electreeneephalograpicc slowing: A primary source of error in automatic seizure detection. In 2017 IEEE Signal Processing in Medicine and Biology Symposium (SPMB), pp. 1-5. IEEE, 2017.
Christopher Wang, Vighnesh Subramaniam, Adam Uri Yaari, Gabriel Kreiman, Boris Katz, Ignacio Cases, and Andrei Barbu. BrainBERT: Self-supervised representation learning for intracranial recordings. In The Eleventh International Conference on Learning Representations, 2023. URL.
net
reUn6
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprint S0Niv:2206.07682,, 2022.
Di Wu, Siyuan Li, Jie Yang, and Mohamad Sawan. neuro2vec: Masked fourier spectrum prediction for neurophysiological representation learning. arXiv preprint arXiv:2204.12440 2022.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Zhenda Xie, Zheng Zhang. Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9653 9663, 2022.
Gaowei Xu, Tianhe Ren, Yu Chen, and Wenliang Che. A one-dimensional cnn-lstm model for epileptic seizure recognition using eeg signal analysis. Frontiers in Neuroscience, 14:578126 126, 2020.
Chaoqi Yang, Brandon Westover, and Jimeng Sun. BIOT: Biosignal transformer for cross-data learning the wild. In sevetth Conference on Neural Information Processing Systems, 2023a. / /ttp//peeeww.ee......// net. www.:eeet/ooouu.eoo htr.//orummmim wow.orumiidddlyyyyyyyyo-
Chaoqi Yang, Cao Xiao, M Brandon Westover, Jismene Sun, et al. Self-Supervised Electroenceehalo- gram Representation Learning for Automatic Sleep Staging: Model Development and Evaluation Study. JMIR AI, 2(1)e446769 2023b.
Ke Yi, Yansen Wang, Kan Ren, and Dongsheng Li. Learning Topology-Agnostic EEG Represen- tations with Geometry-Awan Modeling. In Thirty-seventhy Conference on Neural Information Processing Systems, 2023. URL https: http://ppppeevvveee ew. net./ ftnt:/form?? d=h OUySN
W. Zheng, W. Liu, Y. Lu, B. Lu, and A. Cichocki. Emotionmeter: A multimodal framework for recognizing human emotions. IEEE Transactions on Cybernetics, pp. 1-13, 2018. ISSN 2168- 2267. doi: 10. ww.119/TCYBB.018..29900
Wei-Long Zheng and Bao-Liang Lu. Investigating critical frequency bands and channels for EEG- based emotion recognition with deep neural networks. IEEE Transactions on Autonomous Mental Development, 7(3):162-175. 2015. doi: 10.T.A///.D.. 24344444
14

---

Published as a conference paper at ICLR 2024
A RELATED WORK
Self-supervieed Pre-training. In recent years, self-supervised pre-training has made significant progress in natural language processing and computer vision. BERT (Devlin et al. 2018) innova- tively proposed the idea of masking part of the input sentences and then reconstructing them. The GPT series (Radford et al. 2018 2019| Brown et al. (020) proposed to pre-train large language models by a large corpus of data in an autoregressive way. Both studies improved the fine-tuning performance significantly in various downstream tasks. In computer vision, iGPT (Chen al. 2020) firstly brought the idea from GPT to pre-train a vision model. BEÏT (Bao et al. (022) pioneerly trained vision tokenizer and leveraged BERT-like pre-training for training a vision Transformer.
MAE He et al. 2022) and SimMIM (Xie et al. (022)) practiced masked image modeling by simply reconstructing the raw pixels and achieved appreciable improvement.
Learning with Heterogeneous Datasets. MMM introduced a pre-training framework built on the unified topology and obtained topology-agnostic representations (Yi et al. 2023). HHan et al. (2023) combined graph neural networks and transfer learning for non mon-innasive motor imagery EEG decod- ing with heterogeneous electrode configurations. Gu et al.] (2023) developed two networks to learn from the shared and the complete channels across datasets, achieving coherent performance boosts. et (2004)) proposed hierarchical personalized Federated Learning EEG decoding framework, enabling datasets with disparate data formats to collaborate in the model training process.
Self-supervised Learning in BCI. Although self-supervised pre-training has achieved great suc- cess, its potential in BCI is far from being explored. BENDR (Kostas et al. 202T) adapted Wav2vec 2.0 (Baevski et al., 2020), which uses contrastive learning to learn compressed representations of raw EEG signals. Banville et al. investigated temporal context prediction well as contrastive predictive coding on two clinically relevant problems (Banville et al.] 202T). ContraWR (Yang
al. 2023b), Contrast with the World Representation, used global statisties to distinguish signals associated with different sleep stages. BrainBERT (Wang et al. (203) masks random parts of the (SEEG) spectrogram and produce original embeddings with 43.6 hours of data. However, all existing studies either concentrate on specific BCI tasks or only employ small-size datasets and models, leaving room for exploring large-scale EEG data to train large EEG models through self-supervision.
porro-ddcctoenccpalllogappp
B LABRAM PRE-TRAINING ANALYSIS
The pre-training of LaBraM can be interpreted the training of variational autoencoder (Kingmal & Welling, 2014 Bao et al. 2022). We denote the original EEG sample as the corrupted EEG by masking as x o and its Fourier spectrum (amplitude and phase) as न. The focus on the evidence lower bound (ELBO) of the log-likelihood p(ā)x-M) which involves recovering the Fourier spectrum of the original image from the masked perspective:
log logp(ū,|2; 2
(En,g (logg (xog)))-DD((lq(((l))ppppe.o.
(0% aa J.EDD
(0HLa M JED
(13) where qa (0(2(2) represents the neural tokenizer that encodes the EEG sample into discrete neural to- kens, (+ 2) denotes the neural decoder predicting the Fourier spectrum from given neural tokens, and (z) x M) is the LaBraM pre-training for masked EEG modeling, where the LaBraM encoder reconstructs neural tokens from the corrupted EEG input.
The whole framework is optimized through a two-stage procedure as (Van Den Oord et al. 2017). For the first stage, we train the neural tokenizer discrete variational autoencoder by minimizing the reconstruction loss- -E -.g,,ag (log (Z)(2)) with a uniform prior. For the second stage, we
set dh as well as Ps: fixed and learn the prior po by minimizing the loss DEL- For simplicity, defined one-point distribution with the most likely neural tokens é, arg max; 4s (z (.). Consequently, we can rewrite Equation 13as
Σ
(Berogalsha,) (log (x)) + log (og, (xM))
(14)
(R J.JED
where the first term is the objective for vector-quantized neural spectrum prediction and the second term is the objective for LaBraM pre-training.
15

---

Published as a conference paper at ICLR 2024
C
HYPERPARAMETER SETTINGS
Table 3: Hyperparameters for vector-quantized neural spectrum prediction training.
Hyperparameters
Values
Iput channels
(1,8,8)
Output channels
{8,8,8
Temporal Encoder
Kernel size
{15,3,3)
Stride
(8.1,1)
Padding
{7.1,1)
Transformer encoder layers
12
Transformer decoder layers
3
Hidden size
200
MLP size
800
Attention head number
Codebook size
8192x64
Batch size
1024
Peak learning rate
5e-5
Minimal learning rate Learning rate scheduler Optimizer
le-5
Cosine
Adam W
Adam 3
(0.9,0.99)
Weight decay
le-4
Total epochs
100
Warmup epochs
Data stride
200
Table 4: Hyperparameters for masked EEG pre-training.
Hyperparameters
LaBraM-Base
LaBraM-Large
LaBraM-Huge
Iput channels
{188,8}
[1,16,16}
[1,32,32]
Output channels
{8.8.8)
(16.16,16)
[32,32,32)
Temporal Encoder
Kernel size
(15,3.3}
Stride
(8.1.1)
Padding
[7,1,1]
Transformer encoder layers
12
24
48
Hidden size
200
400
800
MLP size
800
1600
3200
Attention head number
10
16
Batch size
512
Peak learning rate
5e-4
Minimal learning rate
le-5
Learning rate scheduler
Cosine
Optimizer
AdamW W
Adam 3
(0.9.0998)
Weight decay
0.05
Total epochs
50
Warmup epochs
5
Data stride
800
Gradient clipping
3
Layer scale init
0.1
le-5
le-6
EMA weight
0.996
Mask ratio
0.5
16

---

Published as a conference paper at ICLR 2024
Table 5: Hyperparameters for downstream fine-tuning.
Hyperparameters
Values
Batch size
512
Peak learning rate
5c-4
Minimal learning rate Learning rate scheduler
1e-6
Cosine
Optimizer
AdamW
Adam 3 Weight decay
(0.9,0.999)
0.05
Total epochs
50 (B) 30 (L/H) (B) 3 (L/H) 0.1 (B/L) 0.2 (H) 0.65 (B) 0.8 (L/H)
Warmup epochs
Drop path
Layer-wise learning rate decay
Label smoothing (multi-class classification)
0.1
D PRE-TRAINING DATASET DESCRIPTION
We describe the datasets we use for training LaBraM here.
Training datasets (for both vector-quantized neural spectrum prediction training and LaBraM pre- training. the total time is 2534.78 78 hours):
BCI Competition IV-1 Blankertz al. 2007): A motor imagery dataset containing 59 EEG channels at 1000Hz sampling rate for classes of left hand, right hand, foot (+ idle state) for subjects. The recording was made using BrainAmp MR plus amplifiers and an Ag/AgCI electrode cap. (total time: 8.21 hours)
Emobrain (Savran al. 2006): A multimodal emotion dataset where EEG (64 channels, 1024 Hz) and INIRS, are recorded by the Biosemi Active acquisition system, including 16 subjects. The emotions were elicited through a selected subset of IAPS dataset. (total time: 4.94 hours)
Grasp and Lift EEG Challenge (Luciw et al. 2014): A dataset containing EEG recordings (32 channels, 500 Hz) of 12 subjects performing grasp-and-lift (GAL) trials. The EEG cap was used in conjunction with BrainAmp EEG signal amplifier. (total time: 11.72 hours)
Inria BCI Challenge (Margaux et al. 2012): A P300-based spelling dataset including 26 sub- jects with EEG records (56 channels, 600 Hz) by Ag/AgCI EEG sensors (VSM-CTF compatible system). (total time: 29.98 hours)
EEG Motor Movement/mmagery Dataset (Schalk et al. 2004): motor imagery dataset consist- ing of 109 volunteers performing 2 baseline tasks (eye-open and eye-closed), motor movement, and motor imagery (both fists or both feet) with EEG records (64 channels, 160 Hz) using the BCI2000 system. (total time: 47.3 hours)
Raw EEG Data (Trjiilo 2020): A dataset where EEG (64 channels, 256 Hz) was recorded during the reported innformation-Interratti categorization task and the reported multidimensional Rule-Based categorization task. (total time: 34.35 hours)
Resting State EEG Data (Trujillo et al. 2017): A dataset comprising 22 subjects for a resting task of mins with 4 mins of eyes closed and 4 mins of eyes open with 64 EEG channels 256 Hz using active Ag/AgCI electrodes either mounted in a BioSemi electrode cap or via freestanding electrodes. (total time: 3.04 hours)
SEED Series Zheng & Lu 2015 Zheng et al., 2018 Liu et al. 2022): The emotional datasets including SEED (15 subjects), SEED-IV (15 subjects), SEED-GE GER (8 subjects), and SEED-FRA (8 subjects). All EEG signals (62 channels, 1000 Hz) were recorded with the ESI NeuroScan System in response to videos. (total time: 166.75 hours)
Siena Scalp EEG Database (Detti et al.) 2020p: A database consisting of EEG recordings (31 channels, 512 Hz) of 14 patients employing EB Neuro and Natus Quantum LTM amplifiers, and reusable silver/gold cup electrodes. (total time: 30.47 hours)
17

---

Published as a conference paper at ICLR 2024
SPIS Resting State Dataset (Torkamani-Azui et al. 2020): A dataset including 10 subjects, 2.5 minutes recording in each state (eyes-closed and eyes-open) prior to a 105-minute session of Sustained Attention to Response Task with fixed-sequence and varying ISIs. Monopolar EEG activity (64 channels, 2048 Hz) was collected via 64 Ag/AgCI active electrodes. (total time: 0.83 hour)
Target Versus Non-Target Korczowski et al. 2019): A dataset including 50 subjects play- ing Brain Invaders, visual P300 Brain-Computer Interface using oddball paradigm with ada- pative Riemannian Geometry (no-calibration). EEG signals (32 channels, 512 Hz) were quired by means of a research-grade amplifier (g. (g.USBamp, g.tec, Schiedlberg, Austria) and the g. GGAMMAcap. (total time: 16 hours)
TUAR (Buckwalter et al. 202T): This subset of TUEG contains annotations of 5 different artifacts with EEG recorded (23 channels, 256 Hz). (total time: 92.22 hours)
TUEP (Veloso et al. 20177: This is subset of TUEG that contains 100 subjects with epilepsy and 100 subjects without epilepsy with EEG recorded (19-23 channels, 256 Hz), as determined by certified neurologist. (total time: 591.22 hours)
TUSZ (Shah et al. 2018): This corpus has EEG signals that have been manually annotated data for seizure events (start time, stop, channel, and seizure type) with EEG recorded (19-23 channels, 256 Hz). (total time: 1 113..53 hours)
TUSL (von Weltin et al. 2017): This is another subset of TUEG that contains annotations of slowing events (23 channels, 256 Hz). This corpus has been used study common error modali- ties in automated seizure detection. (total time: 20.59 hours)
Self-collected EEG Data (Jiang et al. 2023 2021
Luo et al. 2022 Li et al. 2021 Tao & Lu
2020): We further collect EEG data from more than 140 subjects by ourselves (62 channels, 1000
Hz) with the ESI NeuroScan System. (total time: 342.23 hours)
E VISUALIZATION OF VECTOR-QUANTIZED NEURAL SPECTRUM PREDICTION
We further visualize how the amplitude and phase the Fourier domain are re- constructed. As depicted in Figure 7 although some details are missing, the overall trend of the amplitude is recon- structed well. In contrast, the recon- struction of the phase is not good as the amplitude. Nevertheless, it can be seen from Figure 16) that there is still a stable decrease in the reconstruction loss during training. which indicates the discrete codebook does learn high-level information from the Fourier domain.
Amplitude
Phase
Loss
0.2-
Loss
0.7
0.1
0
20
40
60
80
100
Epochs
Epochs
Figure 6: The reconstruction loss curve of amplitude and phase.
MA Mmm/// (/a/////////
WANN 10 MM
M M Original EBG signals
MwM. Original Amplitude
Original
Reconstructed Phase
Figure 7: Visualization of reconstructed Fourier spectrum. Note that we only visualize half of the results since DFT is conjugate symmetric.
18

---

Published as a conference paper at ICLR 2024
F MORE EXPERIMENTS ON OTHER BCI TASKS
We conduct two additional BCI tasks on the following datasets:
SEED-V (emotion recognition) (Liu et al. 2021): An emotion EEG dataset containing five emo- tion categories (happy, sad, neutral, disgust, and fear). The experiment collected EEG data (62 channels, 1000 Hz) from 20 subjects, including 10 males and 10 females. Each subject partici- pated in the experiments three times and each session included fifteen video clips corresponding to the five emotions, where each video clip lasted for several minutes. The EEG signals are segmented into 148,080 1-second samples.
MoBI (gait prediction) (He al. 2018): A mobile brain-body imaging dataset acquired during treadmill walking in a BCI task, which is a lower limb motor imagery dataset. Six goniometers were employed to record bilateral joint angles the legs (hip, knee, and ankle). The objective is to regress the angles for 12 targets (left leg and right leg). The data were collected from 8 healthy subjects, each of whom had three identical trials. The EEG signals (60 channels, 100 Hz) were recorded by the ActiCap system. Setting the stride to 50 ms, the dataset involves 575,830 2-second samples.
For SEED-V, as there are fifteen trials for one session, we separate the fifteen trials into three parts with an equal number of trials, i..., 5:5:5. We merge each part from all sessions of subjects and derive the training, validation, and test set. As SEED-V is overall balanced, we consider accuracy instead of balanced accuracy as a metric to compare performance. Note that some implementation details are bit different from default settings on this dataset due to different characteristics (peak learning rate: 5e-4 (L) Se-3 (H); total epochs: 50 (L/H), warmup epochs: 4 (L) 5 (H)).
For MoBI, each trial consisted of a 15-minute treadmill walking session (training session), followed by a 5- 5-minute treadmill walking session (test session) with a closed-loop BCI. To validate the model, split the training session into two parts: the first 10 minutes of EEG and its corresponding joint data were used as training data, while the last 5 minutes of data were used as validation data. Meanwhile, combined all the training data, validation data, and testing data of the eight subjects to form corresponding larger training datasets, validation datasets, and testing datasets. Since most angles are typically lower than 90%, the target angles are divided by 90 for normalization. We report the average value of 12 targets for each metric.
As the task of MoBI is regression, choose the following metrics to evaluate the performance of different methods: 1) Pearson's correlation: Pearson's 's correlation coefficient which is used to quantify the models' regression effect. It measures the linear correlation between two variables X and Y. 2) R2 score: R (coefficient of determination) regression score function, which measures how well statistical model predicts an outcome. 3) RMSE: Root Mean Square Error is the standard deviation of the residuals (prediction errors). R2 score is utilized as the monitor to select the best model. MSE loss is the objective to optimize the models.
The experimental results are presented in Figure 6 On SEED-V, LaBraMs outperform all baseline methods on all metrics. The phenomenon that the performance increases when the model gets larger is also observed. For MoBI, our Base model archives competitive results compared to the best bascline method. Whereas, the Large and Huge models obtain better performance among all methods.
Table 6: The results of different methods on SEED-V and MoBI. SEED-V
MoBI
Accuracy
Cohen's 's Kappa
Weighted
Pearson's Correlation
R2 Score
RMSEL
SPaRCNet
0.1032+0.0083 4..08740000447
90004-900004
0.4561+000011
D.1467+0000644
0.1344-0.0006
ContraWR
0.3603=20.009
0.1988+0.0114
0.3590=0.0091
0.3357:0.0164
0.0743--00009
0.1401 00008
ONN-Transformes
0.3665=0.008 0.0088
0.20341000000
0.3638=0.0066
0.122410.0109
0.0628-10.000
2.0099
0.1411=0.0007
FFCL
0.368611000099
0,209411.007%
0.3679-0.0062
0.3158.10.0235
0.0712-10.014
0.1396_20.0014
ST.nssssfree
0.2772
0.0783-000077
0.2625-0.0006
0.5442 0.0011
0.2911 0.0004
0.1222 0.
BIOT
0,3802+#00004
0.2247++0.0100
0.3809=0.01 14
0.2757++0.0173
0.0597+40.0069
0.1401=0.0000
LaBraM-Baa Base
0.4095+0.0066
0.2613-40.075
0.4120-0.0057
0.5383+0.00 02
0.2876+-0.003
0. 0.1222-4000003
LaBraM-Larg
0.4096=00.007
0.2639-+0.00 0090
0.4127=0.0079
0.500
0.3093+0.000 0033
0.0000
LaBraM-Huge
0.4102:20.0037
0.26446:1.00046
0.4136=0.0047
0.5632:10.0023
0.3145=0.0032
0.1196=0.0003
19

---

Published as a conference paper at ICLR 2024
G EFFECTIVENESS OF VECTOR-QUANTIZER NEURAL SPECTRUM PREDICTION
To verify the effectiveness of vector-quantized neural spectrum prediction, we elaborate on three types of experimental settings as illustrated in Table The comparison between LaBraM and Set- ting I demonstrates that the codebook is effective for masked EEG modeling. LaBraM obtains the best performance on TUEV and the lowest standard deviations on TUAB. There an interesting observation that masked EEG modeling with the assistance of training an auxiliary neural tokenizer (LaBrrM and Setting 1) performs greatly better on TUEV while the naive masked EEG modeling (Setting 2 and Setring 3) performs slightly better on TUAB. One explanation for this phenomenon is that learning semantic representations from the neural tokenizer and codebook significantly benefits high-level downstream tasks like TUEV which classifies different types of events. Whereas, TUAB a low-level downstream task where the clinically normal/abnormal EEG segments can be easily distinguished visually. Hence, simply reconstructing origin signals their Fourier spectrum is able to perform well on these low-level tasks but fails to obtain satisfying performance on high-level tasks.
Table 7: Ablations to validate the effectiveness of vector-quantized neural spectrum prediction.
TUAB
TUEV
Balanced Accurucy
AUC-PR
AUROC
Balanced Accuracy
Cohen's Kappa
Weighted Fl
LaBraM
0.8140++0.001
1.8955+0.000
0.9022+40.0009
0.6409 +0.006
0.6637--0.009
0.8312+000052
Setting 0.80581:0.00 0044
0.894911.0007
0,896410.0012
0.6162+:0.017
0.637611.0168
08170=10000$0
Setting 2
0.8261-10.0033.
0.9150-10.0016
0.90671:0.0024
0.5630-1000313
0.5910=10.016
0.7979=0000822
Setting 3
0.8166:10.0073
0.9062:10.0029
0,9053-10.0026
0.5730110.0133
0.5643:10.0089
0.7819-1.0040
Setting 1: We directly predict output embeddings of the neural tokenizer by maximizing cosine similarity instead of predicting the discrete neural tokens from the codebook.
Setting 2: We discard the neural tokenizer and directly reconstruct raw EEG patches by minimizing MSE loss.
Setting discard the neural tokenizer reconstruct the Fourier spectrum (amplitude and phase) of
raw EEG patches by minimizing MSE loss.
H ABLATION ON MASK RATIO
In this experiment, we conduct different settings of the mask ratio to explore its impact. It is noted that we introduce the symmetric masking strategy, so we only need to validate half of the mask ratios. As the mask ratio is set to r, the symmetric masking will mask 1 r proportion of EEG patches. The ablation results are provided in Table X where experiments are conducted on TUAB and TUEV. It can be induced that the best mask ratio is 0.4 (0.6) for TUAB and 0.5 (0.5) for TUEV. Moreover, 0.5 (0.5) is the second-best mask ratio for TUAB while the remaining mask ratios are incredibly close. The performance for mask ratios except 0.5 (0.5) is also similar to each other. Notably, the mask ratio of 0.5 (0.5) achieves smaller standard deviations both TUAB and TUEV. Therefore, we conclude that 0.5 (0.5) is a relatively good mask ratio for the masked EEG modeling of LaBraM pre-training.
Table 8: Performance of different mask ratios.
Mask Ratio
TUAB
TUEV
Balanced Accuracy
AUC-PR
AUROC
Balanced Accuracy
Cohen's Kappu
Weighted FI
0.5 (0.5)
0.8140::0.0019
0.8065:10.0016
0.9022-10.0009
0.6409=100065
0.6637=0.0093
0.8312
0.4 (0.6)
0.8145-L0.003
0.9083-1000030
0.9049-10.003
0.6174:100127
0.6123=0.0094
0.8067
(0.7)
0..0977
0.8950-
0.8974-40.00
0.6112440.026
0.6089-0.0155
0.8068
0.2 (0.8)
0.8039:10.0054
0.8990=1000050
0.9018-10.000 0023
0.6054=100268
0.6050=0.0152
0.8024=0.0089
0.1 (0.9)
0.0022+0.00041
4.8968+000000
0.8992-+0.0M17
0.6033+40.024
0.6181+0.0178
0.813440.0094
20

---

Published as a conference paper at ICLR 2024
I ABLATION ON SYMMETRIC MASKING
We conduct ablation study verify the contribution of the symmetric masking strategy. Table reports the results on TUAB and TUEV. It is obvious that the performance of most metrics de- creases by remarkable margin masking, the performance of the base model increases a little bit on TUAB. Nevertheless, the perfor- mance decreases in most other scenarios. This because the data is sufficient for the base model, so the symmetric masking strategy which acts like data augmentation contributes little to the model training. For larger models like LaBraM-Hugge the symmetric masking improves the downstream performance as it requires more data. This observation indicates that symmetric masking can not only boost the downstream performance but also improve stability and robustness.
both datasets, especially TUEV. Specifically, without symmetric
Table 9: Ablation study of symmetric masking (SM).
TUAB
TUEV
Balanced Accuracy
AUC-PR
AUROC
Bulanced Accuracy
Cohen's s Kappa
Weighted FI
LaBraM-Base
0,8140=1.00019
0.8965-1000116
0.9022:10.0099
0.6409
0.663710.0093 0.6279:10.0260
0.8312:0.0052
w/o SM
0.0001
0.9077.100069
0.9065:10.0034
0.6284=0.0175
0.815210.0105
LaBraM-Large
D.8226+-00000
0.91 0.0005
0.9127-H0.M005
0.6581 0.0115
0.6622-+0.0188 36
0.3315+000000
w/o SM
0.119::0.0042
0.9140=:0.007
0.9106+:0.0012
0.6548=000246
0.6601=:00012
0.83191000034
LaBraM-Huge
0.8258=000001
0.0001=0000011
0.9162++0.0066
0.6616=00.0170
0.6745:+0.0195
0.8329+00.00S
w/o SM
0.8247.10.0011
0,9188 0.0005
0.9149:10.0004
0.6261 1000178
0.6391:10.0179
2.8152:10.008S
J LABRAM WITHOUT PRE-TRAINING
In this experiment, we directly train LaBraM on the downstream datasets from scratch without pre-training to validate the effectiveness of the masked EEG modeling pre-training. The steep per- formance drop demonstrates the usefulness of pre-training, as illustrated Figure
TUAB
TUBV
1.0
pre-train wil pre-train
w/ pre-train
wio pre-train
0.6
0.6
0.4
0.4
I
0.2
0.2
0.0
0.0
Balanced Accuracy
AUC-PR
Accuracy
Kappa
Weighted
Figure 8: Comparison with model without pre-training.
K PARTIAL FINE-TUNING
In Table T0 we report the results about fine-tuning part LaBraM. We elaborate on several settings: fine-tuning all 12 Transformer blocks, fine-tuning the last 8 Transformer blocks, fine-tuning the last Transformer blocks, and linear probing. It is noteworthy that for linear probing, we set the weight decay to O. One can see that TUAB, the results of full fine-tuning, fine-tuning 12 Transformer blocks, and fine-tuning 8 Transformer blocks are quite similar. When only fine-tuning 4 Transformer blocks and linear probing, there is slight degradation in performance. On TUEV, yet, fine-tuning 8 Transformer blocks achieves the best performance on all three metrics. Notably, the results of linear probing much worse than other settings, which still have room for improvement.
21

---

Published as a conference paper at ICLR 2024
Table 10: Results of fine-tuning part of LaBraM.
Fine-tuning Part
TUAB
TUEV
Balanced Accuracy
AUC-PR
AUROC
Balanced Accuracy
Coben's 's Kappa
Weighted FI
0.81 1.00000900019
0.8965=0.0015
0,9022:=000009
0.6409++0.006
0.6637+0.0093
0.8312==0.0552
Transformer (12)
0.8141+0.0022
0.8963--0.014
0.90/2-F-0.000
0.6541-+0.025
0.6782-+0.0189
0.838++000099
Transformer
0.8134400.002
0.8960=0.0019
0.9020=000009
0.6611+0.0152
0.6820=000009
0.8406=0000336
Transformr (4)
0.807410.00332
0.8930=100005
0.8967=000018
0.118::0.0118
0.6560=200233
0.825==0.01 14
Linear Probe
0.7954100.0059
0.886410.0030
0.883510.0002
0.346110.0225
0.3968.1000399
0.697411.0161
L ABLATION ON SPATIAL EMBEDDINGS
The spatial embeddings have helped us address the challenge of heterogeneity in electrode config- urations. However, it is important to verify the effectiveness of this approach. During pre-training. we observed that the loss could not converge without spatial embeddings. This was expected, as the model needs spatial embeddings to identify the masked patch to reconstruct. During fine-tuning on downstream datasets, we discard the spatial embeddings and notice significant drop in perfor- mance, as shown in Table [1 This clearly demonstrates the importance spatial embeddings in capturing spatial information.
Table 11: Ablation study of spatial embeddings (SE).
TUEV
Balanced Accuracy
AUC-PR
AUROC
Balanced Accuracy
Cohen's Kappa
Weighted
LaBraM
0.8140=0.000 0.0019
0.8965++0000 0.8922:10.0023
0.9022-+0.0009
0.6409=000066
0.6637 0.0093
0.8012+0.0052
w/o SE
0.800400.0037
0.8888110.0018
0.5949
0.6069=10.0248
0.8040=0.0111
M DISCUSSION
Limitations. First of all, although we have collected the largest EEG dataset ever of over 2,500 hours and trained the largest model with 369M parameters ever for BCI, it still has a large margin from today's 's large vision models and large language models. Our work is only the first step to explore the feasibility of training large EEG model for learning generic representations. It is delighted find that training a large EEG model with tremendous EEG data does work and obtain appreciable per- formance gain compared to existing methods developed for specific BCI tasks. Secondly, LaBraM needs to be fully fine-tuned to adapt to downstream tasks, which might be computation-costy and memory-cosee costly. Finally, LaBraM is trained with unimodal EEG data. It is worthwhile investigate training large EEG models with other modalities.
Outlook. In view of the above limitations, our paradigm paves the way for further research, en- compassing the following aspects: 1) Collecting more EEG data from a variety of BCT tasks, and training a larger EEG model to see whether emergent abilities exist in the EEG model similar to large language models; 2) Leveraging the parameter efficient learning methods, such adapters, prompt tuning, and LoRA, to reduce the fine-tuning overhead and save space for disks; 3) Incorpo- rating other modalities like image, language, speech, and other physiological signals into large EEG models training to build new paradigms, or aligning EEG representations with other modalities in semantic space, which can be a meaningful and challenging direction for future work.
22


Reviews:
[{"review_id": "I4ANo81rTY", "reviewer": "Reviewer_tjUb", "summary": "The paper proposes a method and model for self-supervised training on large-scale EEG data coming from various datasets with different electrode configurations. First they train a neural tokenizer which learns to compress the EEG signal into vector-quantized encodings and reconstruct the amplitude and phase of the EEG signal from those. Then, given the trained neural tokenizer, a model is trained to reconstruct the vector quantized encodings of an EEG signal decoded from a masked version of the same encodings.  After these pretraining phases, the model is finetuned and evaluated on a downstream task. The authors report improved decoding accuracy on pathology diagnosis and clinical EEG event type classification compared to published work.", "questions": "1)\nI assume Table 1/2 only compares to other self-supervised models? That should be written a bit more explicitly otherwise there are other papers worth citing:\n\nhttps://www.springerprofessional.de/en/chrononet-a-deep-recurrent-neural-network-for-abnormal-eeg-ident/16824220 \nhttps://www.sciencedirect.com/science/article/pii/S1053811920305073 \nMight be in any case good to also show a purely supervised baseline.\n\n2)\nAre TUAB and TUEV disjoint recordingwise from TUAR TUEP TUSZ and TUSL?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Interesting method for self-supervised learning from EEG data\n* Large collection of publicly available datasets\n* Improved results over other self-supervised methods\n* Analysis of scaling behavior", "weaknesses": "1)\nSome papers have not been mentioned that work on heterogeneous datasets:\n* [Learning Topology-Agnostic EEG Representations with Geometry-Aware Modeling](https://openreview.net/forum?id=hiOUySN0ub)\n* [EEG Decoding for Datasets with Heterogenous Electrode Configurations using Transfer Learning Graph Neural Networks](https://arxiv.org/abs/2306.13109v1)\n* [Generalizable Movement Intention Recognition with Multiple Heterogeneous EEG Datasets](https://ieeexplore.ieee.org/document/10160462)\n\n2)\nThe choice to use mean squared error on phase values is strange to me. Due to their cyclical nature, very nearby phases, e.g., -pi+eps,pi-eps would get a large squared error that would also depend on whether one uses phases from 0 to 2pi or -pi to pi etc. So would make more sense tome to either always only use the minimum distance, so if one would put the phases on a unit circle the minimum distance on the circle, or maybe even regress fourier coefficients instead of amplitude/phase. I even wonder what would happen if one just does not put any loss on the phase prediction at all, only on the amplitudes that would be interesting to check as well.\n\n\n3)\nWhy bold lowest std in table 1 and 2, better remove that, is rather confusing\n\n4)\nFont could be a bit bigger in Figure 1 and also parts of Girue 2 (e.g., channel names on bottom)\nFig 2 the amplitude phase plot on top right is confusing to me.\nSymmetric in Figure 2 not symmetric", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uZHBCSjZeZ", "reviewer": "Reviewer_WxGC", "summary": "The authors introduce a Transformer-based model and a pretraining methodology to learn representations from EEG data in a self-supervised manner on a large collection of datasets. The approach combines two parts. First, a vector-quantized tokenizer (similar to a VQ-VAE) is trained to quantize 1-s single-channel EEG patches in order to minimize a regression objective in the spectral domain (i.e. on the amplitude and phase of the input patch). Second, a Transformer encoder is pretrained on a self-supervised task in which the model must predict the tokens that correspond to masked input patches. Three variants of the proposed model (with 5.8 up to 360M parameters) are pretrained on a diverse dataset of more than 2,500 h of EEG recordings and then finetuned on one of four supervised classification or regression downstream tasks. The proposed models outperform existing baselines on these tasks. Ablations on the amount of pretraining data, tokenization approach, masked prediction loss and masking support the proposed pretraining task and model configuration.", "questions": "1. Some of the hyperparameter choices for the windowing and tokenization steps are not clear to me. First, what were the window strides ($s$ in first paragraph of Section 2.1) used for the different datasets? Tables 3 and 4 report a “Data stride” value but I’m not sure whether that’s the same thing. Second, what is the impact of the selected patch size and patch stride on performance, i.e. are these choices important? Related to the point about how symmetric masking might be providing regularization that is useful for larger models, would a smaller window and/or stride help create more pretraining examples? \n\n2. It is not clear to me whether the weights of the temporal encoder from the tokenization step are reused in the pretraining step, or if only the architecture is the same. From the dimensions of the large and huge models I assume the weights could not be reused as the sizes are not the same.\n\n3. My understanding is that new spatial embeddings must be learned from each montage that is seen (i.e. the $i$ in Equation 2). How many different spatial embeddings were learned during pretraining? Also, what spatial embeddings were used in the different ablations of Table 10 if the model didn’t have a chance to learn a spatial embedding for TUAB and TUEV (or were examples from this EEG montage already seen in the pretraining data)?\n\n4. Section 2.3: What is the self-distillation loss term? It doesn’t seem to be in the final pretraining objective of Equation 12.\n\n5. Is there an overlap between TUAB/TUEV and the different training sets taken from the Temple University EEG Corpus (TUAR, TUEP, etc.)? If so, could this explain why including TUAB and TUEV in the pretraining set didn’t change the results much (Section 3.5)? I believe this shouldn’t impact the comparison with BIOT as the reported results from the BIOT paper appear to be from the model also pretrained on TUAB and TUEV. \n\n6. Related to the previous question, looking at Figure 4 it looks like adding TUEV to the pretraining dataset actually negatively impacts downstream performance. Is this effect significant and if so, what could be driving this decrease in performance?\n\n7. Section 3.2 on preprocessing: the notch filtering at 50 Hz will not adequately remove power line interference in datasets collected in North America, such as the TUEG datasets. I expect results to improve if the authors correctly notch filter those datasets at 60 Hz instead.\n\n8. How are the pretraining examples sampled? Since this is not described in the manuscript I would assume sequences were sampled uniformly across the entire training corpus, however I was wondering if the authors have considered more balanced sampling schemes, e.g. taking datasets, recordings and/or experimental paradigm-related information into account when sampling.\n\n9. Where did the baseline results for Table 6 come from? They don’t seem to be in the BIOT paper.\n\n10. A few typos:\n- Figure 2: “Fuorier spectrum”\n- Appendix J: “PRE-TRAINGING”\n- Equation 14: Missing closing parenthesis", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality: The proposed approach follows logically from existing work on pretraining Transformers on a corpus of combined EEG datasets, however the combination of a vector quantization tokenizer, a regression objective in the spectral domain and the pretraining on a very large set of EEG recordings appears novel.\n\nQuality: The paper is of good quality, with strong core results showing the performance of the proposed approach, and multiple supporting analyses and ablation studies that support the methodological choices that were made. A few claims might not be completely supported by the results (see Weaknesses).\n\nClarity: The paper is overall clearly written, with mostly clear descriptions of the proposed approach and of the results. Some methodological points require clarification (see Weaknesses, first point).\n\nSignificance: Overall, this study is an important step towards bridging the gap between the approaches used in large language modeling and EEG processing. The results on “EEG scaling laws” (Section 3.6) are a first attempt at answering an important question in the field of deep learning and EEG.", "weaknesses": "- Some methodological points require clarification, e.g. the impact and choice of windowing/tokenization hyperparameters (Q1), the learning and reuse of spatial embeddings (Q3), the potential overlap between pretraining and downstream datasets (Q5) and the sampling of examples during pretraining (Q8).\n\n- I don’t think the results are clear enough to deduce what is claimed in the analysis of Figure 5, i.e. that the performance saturates for the base model at 500 h and for the large model at 2000 h. For instance, the Large models seem to continue learning over 2000 h on TUEV. Similarly, the Base model might not be saturated yet; the performance curve seems pretty noisy. Maybe repeating this analysis with a log-scale would be clearer (1h, 10h, 100h, 1000h, 2500h).\n\n- The use of the term “BCI” (e.g. in the title) is confusing as this typically refers to a subset of the tasks/datasets considered in this work. For instance, the term BCI is usually used to describe cases where there is an interface between brain activity and a computer that bypasses normal communication pathways. Under this definition, tasks such as pathology detection (TUAB) or event detection (TUEV) are not BCI tasks. I would recommend adapting the language of the manuscript to make this clearer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jiqpk1Wszs", "reviewer": "Reviewer_ztTZ", "summary": "This manuscript introduces the Large Brain Model (LaBraM), an innovative approach designed to enhance the capabilities and scalability of EEG-based deep learning models in brain-computer interaction (BCI). The authors explore the potential of Large EEG Models (LEMs) for universal perceptual capabilities through unsupervised pre-training, followed by fine-tuning for various downstream tasks. However, EEG data presents unique challenges like mismatched electrodes, diverse data lengths, and low signal-to-noise ratios. To address these, LaBraM implements cross-dataset learning by segmenting EEG signals into channel patches and employs vector-quantized neural spectrum prediction for rich neural tokenization. This is further enhanced by pre-training neural Transformers to predict original neural codes for masked EEG channel patches. LaBraM is presented in three sizes, Base (5.8M), large (46M), and Huge (369M), with the \"Huge\" variant being the biggest EEG model to date. Pre-trained on approximately 2,500 hours of EEG signals from around 20 datasets, LaBraM showcases superior performance across several downstream tasks, outshining existing state-of-the-art methods. The authors also delve into the data requirements for training various sizes of the model.", "questions": "Will the pre-trained checkpoints be released for open-source development as well?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The manuscript represents a significant contribution to the field, pioneering the development of the largest Large EEG Model (LEM) for EEG decoding. By effectively addressing two pivotal challenges in this field — the utilization of large-scale EEG data and the required data volume — this work lays a solid foundation for future researchers in this field\n2. The clarity and coherence of the presentation are commendable, facilitating an in-depth understanding of the proposed methodologies.\n3. A notable strength of this work is the comprehensive experimental evaluation. The authors have conducted a plethora of experiments, providing supplementary results, detailed ablation studies, and a thorough discussion on hyperparameter settings in the appendix.\n4. The Figures, complemented by lucid annotations, further enhance the comprehensibility and accessibility of the work to the audience.", "weaknesses": "1. While the authors acknowledge the challenge of varying configurations in EEG data collection, particularly concerning electrode variations across different datasets, the manuscript does not provide a comprehensive solution to this challenge. Specifically, the Temporal & Spatial embedding section offers an ambiguous explanation regarding spatial embedding (SE). The method appears to encode channels based merely on their sequential order, which may not accurately represent the channel's functional significance or location in the brain. Given the heterogeneity in electrode configurations and positions across various datasets, it's imperative for the authors to elucidate how their approach effectively manages this inconsistency, and use qualitative results to demonstrate the model has learned different electrode configurations.\n\n2. The paper draws parallels to another LEM, BIOT, developed by Yang et al., and even follows some experimental settings from the same. Given the apparent similarities, it remains unclear as to what drives the performance improvements of the proposed model — is it solely attributed to the increased model size, enhanced training data volume, or specific architectural designs? A more in-depth comparative discussion and analysis between the two models would be beneficial to ascertain the genuine contributions and innovations of the current work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)