NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals

## Abstract

Abstract Recent advancements for large-scale pre-training with neural signals such as electroencephalogram (EEG) have shown promising results, significantly boosting the development of brain-computer interfaces (BCIs) and healthcare. However, these pre-trained models often require full fine-tuning on each downstream task to achieve substantial improvements, limiting their versatility and usability, and leading to considerable resource wastage. To tackle these challenges, we propose \method , the first multi-task foundation model that leverages the capabilities of Large Language Models (LLMs) by regarding EEG signals as a foreign language, endowing the model with multi-task learning and inference capabilities. Our approach begins with learning a text-aligned neural tokenizer through vector-quantized temporal-frequency prediction, which encodes EEG signals into discrete neural tokens. These EEG tokens, generated by the frozen vector-quantized (VQ) encoder, are then fed into an LLM that learns causal EEG information via multi-channel autoregression. Consequently, \method can understand both EEG and language modalities. Finally, multi-task instruction tuning adapts \method to various downstream tasks. We are the first to demonstrate that, by specific incorporation with LLMs, \method unifies diverse EEG tasks within a single model through instruction tuning. The largest variant \method -XL has record-breaking 1.7B parameters for EEG signal processing, and is pre-trained on a large-scale corpus comprising approximately 25,000-hour EEG data. When evaluated on six diverse downstream datasets, \method showcases the huge potential of this multi-task learning paradigm.

## 1 Introduction

Figure: Figure 1: The overall performance comparison on six tasks.
Refer to caption: x1.png

Electroencephalogram (EEG) signals have become a cornerstone in the development of brain-computer interfaces and healthcare domains, offering a non-invasive solution to capture the electrical activity of the brain. EEG measures the voltage fluctuations resulting from ionic current flows within the neurons of the brain, providing real-time insights into brain function and neural dynamics. This capability makes EEG an invaluable tool for creating interfaces that enable direct communication between the brain and external devices. EEG is advantageous due to its high temporal resolution, cost-effectiveness, and portability, and has been significantly enhanced by advanced computational methods. Therefore, a wide range of applications have been utilizing EEG signals, including but not limited to human emotion recognition (Jenke et al., ), body motor imaginary (Tabar & Halici, ), automatic sleep stage classification (Supratak et al., ), seizure epilepsy detection (Alotaiby et al., ), and fatigue detection (Gao et al., ).

While EEG signals are popular among researchers, they have several disadvantages, including the low signal-to-noise ratio, inherent nonstationarity, as well as diverse configurations in EEG data collection. Besides, there is a lack of sufficient and consistent EEG data. These challenges complicate the extraction of universal EEG representations. To overcome these problems, several studies have proposed methods compatible with diverse EEG configurations to learn effective and generic representations. For example, Yang et al. () introduce a Biosignal Transformer (BIOT), which unifies various EEG data by tokenizing channels into fix-length segments with channel and relative position embeddings for preserving spatio-temporal features. Jiang et al. () advance this approach by proposing a neural tokenizer to pre-train LaBraM by masked neural code prediction with 2,500 hours of EEG data, thus achieving state-of-the-art (SOTA) performance on various downstream tasks. Although these methods effectively address the aforementioned challenges, they still require individual fine-tuning on each downstream dataset to obtain impressive improvement. Despite increasing model size and employing large-scale unsupervised pre-training to learn generic representations, such adaptation confines the fine-tuned model to perform only a single task. Moreover, this task-specific fine-tuning demands substantial computational and storage resources.

Over the past few years, the advent of Large Language Models has brought remarkable progress and demonstrated extraordinary emergent abilities (Brown et al., ; Touvron et al., ). The development of LLMs has given rise to Multimodal Large Language Models (MLLMs) (Achiam et al., ; Liu et al., ), which unleash the potential of powerful LLMs to perform multimodal tasks. MLLMs typically integrate a modality-specific encoder, pre-aligned with text embeddings, into off-the-shelf LLMs. Inspired by MLLMs, we unveil a new direction of integrating multiple EEG tasks into a unified model by incorporating EEG signals into existing LLMs. However, there are some challenges in harnessing LLMs to understand EEG patterns, comprising:

1) EEG-text embedding alignment. Aligning EEG and text embeddings presents a great challenge. Unlike vision-language models which benefit from numerous high-quality image-text pairs, there are no established EEG-text pairs available due to the difficulty of extracting semantic information from a given EEG segment.

2) Effective Representation learning with LLMs. Mainstream methods employ masked EEG modeling to effectively extract representations for EEG signals. When integrating LLMs, how to learn generic information within the LLM paradigm remains an unsolved issue.

3) Unified multi-task learning with various EEG tasks. Integrating multiple EEG tasks into a unified model is complex due to the diversity and specificity of different tasks. Developing a model that can seamlessly handle various tasks without compromising performance on any individual task is a major challenge.

In light of the aforementioned challenges, we propose \method, a universal multi-task foundation model for EEG signal processing. \methodbuilds upon the compatibility with diverse EEG formats established by LaBraM, and it is pre-trained on a large-scale dataset comprising approximately 25,000 hours of EEG data. The training of \methodinvolves three stages. First, a text-aligned neural tokenizer is trained using vector-quantized temporal-frequency prediction to encode continuous EEG signals into discrete codes from a neural codebook, with adversarial training employed to align the EEG and text spaces. Next, the VQ encoder of the neural tokenizer is frozen to extract compact embeddings, which serve as input for a LLM. To enable the LLM to learn causal EEG representations, we propose multi-channel autoregressive pre-training, which mimics autoregressive language modeling but is tailored for multi-channel EEG signals. Finally, we elaborate instructions for various downstream datasets and employ multi-task instruction tuning to empower \methodfor multi-task learning. Experiments on six different tasks, encompassing abnormal detection, event type classification, emotion recognition, sleep stage classification, cognitive workload prediction, and slowing type classification, demonstrate \method’s superiority in multi-task learning and inference. To the best of our knowledge, we are the first to introduce instruction tuning to enable multi-task learning and inference in the field of EEG signal processing. The highlights are summarized as follows:

1) Text-aligned neural tokenizer embeddings. We introduce a text-aligned neural tokenizer that effectively bridges the gap between EEG and text data. This tokenizer uses vector-quantized temporal-frequency prediction to convert EEG signals into discrete codes, facilitating the alignment of EEG and text embeddings through adversarial training. This alignment is crucial for leveraging the strengths of LLMs in understanding and processing EEG data.

2) Large-scale multi-channel autoregressive pre-training. \methodemploys multi-channel autoregression, enabling the model to learn causal representations across different EEG channels. Pre-training on 25,000 hours of EEG data ensures that \methodcaptures a wide range of neural patterns, enhancing its ability to generalize across diverse EEG tasks.

3) Joint multi-task tuning and inference. We pioneer the use of joint multi-task tuning and inference for EEG. By elaborating specific instructions for various downstream tasks and employing multi-task instruction tuning, \methodis capable of performing multiple tasks within a single model. This not only improves efficiency by reducing the need for individual fine-tuning for each task but also ensures high performance across a spectrum of applications.

## 2 Method

In this section, we elaborate our design of \method. We first train a neural tokenizer by vector-quantized temporal-frequency prediction. Whereafter, the VQ encoder of the tokenizer will serve to encode EEG signals into embeddings aligned with text space, and the EEG embeddings will be seamlessly used as input to Large Language Models.

Given multi-channel EEG signals $X\in\mathbb{R}^{C\times T}$, where $C$ denotes the number of channels and $T$ denotes total timestamps. An EEG sample is formulated as $\bm{x}\in\mathbb{R}^{C\times L}$, where $L$ is the window size, resulting in a total number of $\lfloor\frac{T}{L}\rfloor$ samples. We pachify the EEG samples into non-overlap patches $\bm{x}=\{x_{ij}\in\mathbb{R}^{P}|i=1,...,C,j=1,...,N\}$. Let $P$ is patch size and $N=\lfloor\frac{L}{P}\rfloor$.

Figure: Figure 2: The architecture design of text-aligned neural tokenizer training. The neural tokenizer is trained by reconstructing both temporal and frequency domain of input EEG signals to discretize them into discrete neural tokens. To align EEG and text embedding space, we utilize a domain classifier through adversarial training.
Refer to caption: x2.png

### 2.1 Text-aligned Neural Tokenizer Training

To incorporate EEG into off-the-shelf Large Language Models, we first need to encode EEG signals into embeddings whose space is well-aligned with text embedding space. VQ-VAE (Van Den Oord et al., ) is a good choice that maps continuous signals to discrete tokens while preserving the key information. Our text-aligned neural tokenizer basically follows the well-established neural tokenizer of LaBraM (Jiang et al., ) with some improvements. Vector-quantized temporal-frequency prediction is utilized to train the text-aligned neural tokenizer, as illustrated in Figure 2.

Neural Tokenizer. The neural tokenizer is composed of several vital components: VQ encoder, codebook, temporal/frequency decoder, and domain classifier. The codebook $\mathcal{V}\in\mathbb{R}^{K\times D}$ contains $K$ discrete $D$-dimension embeddings. Let $h_{i}$ denote the patch representations derived from the VQ encoder. We find the nearest codes of each $h_{i}$ from codebook embeddings $\{v_{i}|i=1,...,K\}$:

$$ $z_{i}=\mathop{\arg\min}\limits_{j}\|\ell_{2}(h_{i})-\ell_{2}(v_{i})\|_{2},$ (1) $$

where $j\in\{1,...,K\}$ and $\ell_{2}$ normalization is employed so that the above distance is equivalent to cosine similarity. Consequently, an EEG sample is tokenized to $\bm{z}=\left[z_{1},...,z_{N}\right]$.

Temporal-frequency Prediction. We propose to predict both original signals and the frequency magnitude to capture the temporal and frequency domains of EEG signals. This differs from LaBraM which regresses the Fourier amplitude and phase since we observe that reconstructing the phase contributes minor to neural tokenizer training. We apply the Discrete Fourier Transform (DFT) on an EEG patch $x_{i,j}=[x[1],x[2],...,x[P]]$ of channel $i$ and time $j$, and transform the equation using Euler’s formula as follows

$$ $\tilde{x}_{i,j}^{m}=\sum_{n=1}^{M}x[n]\cos(\frac{2\pi}{M}mn)-\bm{j}x[n]\sin( \frac{2\pi}{M}mn).$ (2) $$

where $m\in[1,N]$ and $\bm{j}$ is the imaginary unit. Accordingly, we calculate the frequency magnitude as $f^{m}=\sqrt{Re(\tilde{x}_{i,j}^{m})^{2}+Im(\tilde{x}_{i,j}^{m})^{2}}$, where $Re$ and $Im$ represent the real and imaginary parts of a complex number. For stable convergence, we adopt z-score normalization to the magnitude within a sample.

After being quantized to the codebook embeddings, we feed the normalized neural embeddings $\left[\ell_{2}(z_{1}),...,\ell_{2}(z_{N})\right]$ into two separate decoders. Let $o_{i}^{t}$ and $o_{i}^{f}$ stand for the output of a temporal decoder and a frequency decoder, respectively. The optimizing target for the codebook learning is

$$ $\mathcal{L}_{1}=\sum_{\bm{x}\in\mathcal{D}}\sum_{i}\underbrace{\|o_{i}^{t}-x_{ i}\|_{2}^{2}+\|o_{i}^{f}-f_{i}\|_{2}^{2}}_{\text{reconstruction loss}}+ \underbrace{\|\bm{sg}(\ell_{2}(h_{i}))-\ell_{2}(v_{z_{i}})\|_{2}^{2}}_{\text{ codebook loss}}+\underbrace{\|\ell_{2}(h_{i})-\bm{sg}(\ell_{2}(v_{z_{i}}))\|_{ 2}^{2}}_{\text{commitment loss}},$ (3) $$

where $\mathcal{D}$ represents the whole dataset and $\bm{sg}$ denotes the stop-gradient operator that is identical during forward computation and has zero partial derivatives.

EEG-text Embedding Space Alignment. Current vision-language models usually utilize pre-trained CLIP-like (Radford et al., ) image encoders which are trained by large-scale image-text pairs and thus are embedding-wise well-aligned with text. However, when considering EEG, there are much more challenges to align EEG with text: 1) EEG signals contain complicated cognitive and non-cognitive information, which is hard to be described by human language accurately and thoroughly. For example, an EEG segment can not only contain one person’s emotion and mental states, but also represent the body movement and medical normality. 2) The labeled EEG data available to construct EEG-text pair are very limited. Therefore, we propose to align EEG with text space-wise instead of embedding-wise.

We introduce a domain classifier $\mathcal{C}$ to predict whether the embeddings are from EEG or text. During the codebook learning, we also feed some text embeddings from LLMs to train the domain classifier. A gradient reverse layer (Ganin et al., ) is added after the VQ encoder to confuse the domain classifier. Hence, the embeddings from the VQ encoder fall into the same space of text embeddings. Consequently, the training objective for text-aligned neural tokenizer training is defined as

$$ $\min\mathcal{L}_{1}+\lambda\sum_{i}d_{i}\log\mathcal{C}(h_{i}),$ (4) $$

where $d_{i}$ is the label of EEG or text domain and $\lambda=\frac{2}{1+e^{-10t/T}}-1$ is a scaling factor that gradually changes from 0 to 1.

VQ Encoder Architecture. We briefly introduce the architecture of the VQ encoder as it is almost the same as LaBraM. The temporal encoder and spatial encoder are two pivotal parts of the VQ encoder. The temporal encoder contains several blocks of 1-D convolution which aims to extract temporal features in each EEG patch. After that, learnable temporal and spatial embeddings are added according to the standard 10-20 international system to inject both time and channel information. Finally, the spatial encoder composed of vanilla Transformer blocks (Vaswani et al., ) learns interaction among patches.

Figure: Figure 3: Schematic of \methodtraining. Left: We first pre-train \methodvia multi-channel autoregression with EEG tokens output by the frozen VQ encoder. Right: The multi-task instruction tuning enables \methodto perform various BCI tasks within a single model.
Refer to caption: x3.png

### 2.2 Multi-channel Autoregressive Pre-training

Before passing EEG data into Large Language Models, we freeze the VQ encoder and first use it to encode input EEG data to EEG tokens that are aligned with the text space. After that, we load a pre-trained Large Language Model and enlarge the text vocabulary with the learned EEG codebook. The EEG tokens are added with reused temporal embeddings from the LLM and new spatial embeddings. As shown in Figure 3, \methodis then trained through multi-channel autoregression, that is, predicting the next EEG tokens based on visible EEG tokens, to endow the model with the capability of learning special patterns of EEG causal relationship. In our experiments, the multi-channel autoregressive pre-training contributes to the performance of multi-task instruction tuning.

Figure: Figure 4: The stair-stepping mask. Each row indicates attention masks for an EEG token.
Refer to caption: x4.png

Formulation. Consider a sequence of EEG tokens $\bm{h}=\{h_{ij}|i=1,...,C,j=1,...,N\}$ where $i$ denotes the channel and $j$ denote the time, and their corresponding indices of the merged text and EEG vocabulary $\bm{I}=\{I_{ij}|i=1,...,C,j=1,...,N\}$ derived from the neural tokenizer. Unlike language that can be predicted token by token intuitively, EEG signals are of various configurations, thus it is impracticable to directly predict EEG tokens one by one. We propose a multi-channel autoregressive strategy to adopt the idea of autoregression on EEG. The basic idea is that each token of a specific channel predicts the next token of the same channel, which can be formulized as

$$ $p(I_{11},I_{12},...,I_{CN})=\prod\limits_{n=1}^{N}p(I_{1n},I_{2n},...,I_{Cn}|h _{11},h_{12},...,h_{C(n-1)}).$ (5) $$

Therefore, the objective for multi-channel autoregressive pre-training is to optimize model parameters by maximizing $p(h_{1n},h_{2n},...,h_{Cn}|h_{11},h_{12},...,h_{C(n-1)})$ throughout all EEG data.

For implementation, we define stair-stepping masks where each EEG token is able to observe tokens of all channels from its current and previous time step. Figure 4 illustrates the design of our stair-stepping mask. Dark cells indicate that the elements should take part in attention.

Theory Analysis. We interpret the multi-channel autoregressive pre-training from the view of a variational autoencoder (Kingma & Welling, ). Let $x$ denote the original EEG signals, $y$ denote the temporal-frequency target of $x$, and $\hat{x}$ be the EEG tokens to be predicted. Assume that EEG signals $x$ can be generated by a random process with a latent variable $z$. We use $q_{\phi}(\bm{z}|x_{i})$ to denote the VQ encoder encoding EEG signals into discrete neural codes, $p_{\psi}(y_{i}|z_{i})$ to stand for the temporal and frequency decoder reconstructing temporal-frequency domain from encoded neural codes, and $p_{\theta}(\bm{z}|x_{i})$ to represent multi-channel autoregressive pre-training. Consider the log-likelihood $p(y|x)$ and its evidence lower bound (ELBO), involving predicting the temporal-frequency domain of the EEG signals from the next time point:

$$ $\sum_{i=1}^{N}\log p(y_{i}|x_{i})\geq-\sum_{i=1}^{N}(\mathbb{E}_{z_{i}\sim q_{ \phi}(\bm{z}|x_{i})}[-\log p_{\psi}(y_{i}|z_{i})]+KL(q_{\phi}(\bm{z}|x_{i}),p_ {\theta}(\bm{z}|x_{i})),$ (6) $$

where the first term is the reconstruction loss and the second term is Kullback-Leibler divergence between $q$ and EEG-text conditional prior. Our training paradigm encompasses two-stage learning processes: 1) The neural tokenizer is optimized by minimizing the reconstruction loss. 2) A LLM learns the prior $p_{\theta}$ by minimizing $KL$ loss with $q_{\phi}$ and $p_{\psi}$ fixed. The sequence $z_{i}$ can be sampled from $q_{\phi}(\bm{z}|x_{i})$ or one-point distribution $z_{i}=\arg\max_{z}q_{\phi}(\bm{z}|x_{i})$ where we choose the latter for simplicity. Therefore, Equation 6 can be rewritten as

$$ $-\sum_{i=1}^{N}(\mathbb{E}_{z_{i}\sim q_{\phi}(\bm{z}|x_{i})}[-\log p_{\psi}(y _{i}|z_{i})]-\log p_{\theta}(z_{i}|x_{i})),$ (7) $$

where the latter term is the negative log-likelihood loss for multi-channel autoregressive pre-training.

### 2.3 Multi-task Instruction Tuning

In this stage, we aim to leverage the power of LLMs to integrate different downstream datasets as a whole. Instruction tuning is introduced to handle various downstream tasks, as shown in Figure 3. It is worthwhile to mention that in both multi-channel autoregressive pre-training and multi-task instruction tuning stages, we feed the model a few text data at each iteration to preserve the language modeling capability of LLMs. We build instructions for each downstream dataset and the instruction design can be found in Appendix B. A special token $[$SEP$]$ is used to concatenate EEG and text instructions, indicating the modality switch. Notably, the loss is only calculated on the answer part of the text to make the prediction more stable. Suppose $x^{p}$ represents the EEG tokens along with the question part of the instruction (prompt), and $t^{a}$ represents the answer part of the instruction. Let the sequence length of $t^{a}$ be $L$, and this procedure can be written as

$$ $p(t^{a}|x^{p})=\prod\limits_{i=1}^{L}p(t^{a}_{i}|x^{p},t^{a}_{,<i}),$ (8) $$

where $t^{a}_{,<i}$ is the answer tokens before the current prediction token $t^{a}_{i}$.

## 3 Experiments

**Table 1: Information of datasets used for downstream evaluation.**
| Dataset | #Channel | Sampling Rate | Duration | #Sample | Task |
| --- | --- | --- | --- | --- | --- |
| TUAB | 23 | 256 Hz | 10 seconds | 409,455 | Binary classification |
| TUEV | 23 | 256 Hz | 5 seconds | 112,491 | 6-class classification |
| SEED | 62 | 1000 Hz | 4 seconds | 38,475 | 3-class classification |
| HMC | 4 | 256 Hz | 30 seconds | 137,243 | 5-class classification |
| Workload | 19 | 500 Hz | 4 seconds | 2,088 | Binary classification |
| TUSL | 23 | 256 Hz | 10 seconds | 245 | 3-class classification |

### 3.1 Downsream Datasets

We consider six different EEG datasets with highly varied data sizes to comprehensively evaluate \method, where the detailed information is listed in Table 1: 1) TUAB (Harati et al., ) (abnormal detection): This dataset contains EEG records that are classified as clinically normal or abnormal. 2) TUEV (Harati et al., ) (event type classification): This corpus contains six events involving periodic lateralized epileptiform discharge, generalized periodic epileptiform discharge, spike and/or sharp wave discharges, artifact, and eye movement. 3) SEED (Zheng & Lu, ) (emotion recognition): There are 3 emotions (positive, negative, and neutral) elicited by videos from 15 subjects. There are 15 trials in each session and each subject underwent 3 sessions. 4) HMC (Alvarez-Estevez & Rijsman, ) (sleep stage classification): HMC was developed for automatic sleep scoring, involving 5 sleep stages (wake, NREM-1, NREM-2, NREM-3, REM) from 154 subjects. 5) Workload (Zyma et al., ) (cognitive workload classification): This dataset contains 36 subjects performing serial subtraction. We regard mental workload trials as high workload and the last 60 seconds of the rest EEG as low workload. 6) TUSL (von Weltin et al., ) (slowing event classification): TUSL aims to differentiate between seizure, slowing, and complex background events.

For the data division, we split each dataset into training, validation, and test sets: 1) TUAB and TUEV: Since the training and test division is provided by the original datasets, we further divide the training patients into training and validation groups by 80% and 20% randomly. 2) SEED: We split total 15 trials into training, validation, and test trials by 9:3:3 according to the chronological order, and merge all sessions into the final training, validation, and test set. 3) HMC: subjects from number 1 to 100 form the training set while subjects from number 101 to 126 and number 126 to 154 are validation and test sets, respectively. 4) Workload: The training, validation, and test sets are derived by subjects from number 0 to 25, number 26 to 30, and number 31 to 35, respectively. 5) TUSL: The training, validation, and test sets are splitted by 60%:20%:20%.

### 3.2 Experimental Setup

Model Configurations. \methodis compatible with any causal LLM as its base language model. For simplicity and saving computing resources, we adopt GPT-2 (Radford et al., ) as our base language model. Accordingly, \methodhas three variants, \method-B, \method-L, and \method-XL, which have 254M, 500M, and 1696M parameters (including the parameters of the VQ encoder), respectively. Unless otherwise noted, \methodrefers to \method-B. The patch size $P$ is set to 200 (1 second), consistent with that of LaBraM. To maintain compatibility with GPT-2, the maximum sequence length (number of patches) is set to 1024. For input samples with sequence lengths shorter than 1024, we pad zeros to ensure the length is equal to 1024 at neural tokenizer training and multi-channel autoregressive pre-training stages. The attention values for these zero paddings will be masked.

Data Preprocessing. To eliminate environmental and physiological artifacts from EEG signals, we employ several necessary preprocessing methods. First, we apply a bandpass filter with cutoff frequencies of 0.1 Hz and 75 Hz to remove low-frequency noise. To avoid power-line interference, we use a notch filter at 50 Hz or 60 Hz, depending on the geographic region of data collection. Additionally, all signals are resampled to 200 Hz to reduce computational complexity. Given that EEG signal values typically range between -100 $\mu$V to 100 $\mu$V, all values are divided by 100 for normalization.

Training & Environment Settings. To facilitate the training of \method, a huge volumn of data is required. About 25,000 hours of EEG data from multiple public EEG datasets are collected after cleaning and filtering, which are listed in Appendix C. All experiments are conducted on eight NVIDIA A100-80G GPUs with Python 3.11.8 and PyTorch 2.2.2 + CUDA 12.1. For instruction tuning, the results are obtained using the final model after training. Notably, we choose the largest logits as the prediction at evaluation and test instead of beam search which is widely used in current LLMs to obtain stable results. For other scenarios, the baselines are in a single-task manner and trained on individual datasets. Their best models are selected based on the best performance on the validation set, and then evaluated on the test set. The average and standard deviation values are reported using three random seeds to ensure comparable results. Baselines and other detailed hyperparameter settings are provided in Appendix D.

### 3.3 Experimental Results

We present all results in Table 2, 3, and 4. Underlined values represent the best results for single-task methods, while bold values indicate the best results for \method. Notably, it’s important to note that direct comparisons between \methodand the baseline single-task methods are not entirely fair, as the baselines are trained and tested on individual datasets. Although \methodis still a few steps away from the state-of-the-art LaBraM, it achieves performance comparable to most other single-task baselines. Specifically, \methodperforms similarly to BIOT and surpasses all other baselines except LaBraM on TUEV. Additionally, \methodranks second on TUSL. For \method-L and \method-XL, the performance is further enhanced on most downstream datasets with larger model capacity. However, the imbalance in data size among different downstream datasets poses a challenge for \method, as it reaches optimal performance at different training times for different datasets. Additionally, we find that models with more parameters are more prone to overfitting, which might account for the performance degradation observed on HMC since sleep patterns are of low complexity and smaller models might be sufficient to capture the relevant features in EEG signals. On TUSL, the model performance appears to be not very stable due to the extremely limited data samples.

**Table 2: Results on TUAB and TUEV.**
| Methods | Multi-task | TUAB | TUEV |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Balanced Acc. | AUC-PR | AUROC | Balanced Acc. | Cohen’s Kappa | Weighted F1 |  |  |
| SPaRCNet | ✗ | 0.7896$\pm$0.0018 | 0.8414$\pm$0.0018 | 0.8676$\pm$0.0012 | 0.4161$\pm$0.0262 | 0.4233$\pm$0.0181 | 0.7024$\pm$0.0104 |
| ContraWR | ✗ | 0.7746$\pm$0.0041 | 0.8421$\pm$0.0104 | 0.8456$\pm$0.0074 | 0.4384$\pm$0.0349 | 0.3912$\pm$0.0237 | 0.6893$\pm$0.0136 |
| CNN-Transformer | ✗ | 0.7777$\pm$0.0022 | 0.8433$\pm$0.0039 | 0.8461$\pm$0.0013 | 0.4087$\pm$0.0161 | 0.3815$\pm$0.0134 | 0.6854$\pm$0.0293 |
| FFCL | ✗ | 0.7848$\pm$0.0038 | 0.8448$\pm$0.0065 | 0.8569$\pm$0.0051 | 0.3979$\pm$0.0104 | 0.3732$\pm$0.0188 | 0.6783$\pm$0.0120 |
| ST-Transformer | ✗ | 0.7966$\pm$0.0023 | 0.8521$\pm$0.0026 | 0.8707$\pm$0.0019 | 0.3984$\pm$0.0228 | 0.3765$\pm$0.0306 | 0.6823$\pm$0.0190 |
| BIOT | ✗ | 0.7959$\pm$0.0057 | 0.8792$\pm$0.0023 | 0.8815$\pm$0.0043 | 0.5281$\pm$0.0225 | 0.5273$\pm$0.0249 | 0.7492$\pm$0.0082 |
| LaBraM | ✗ | 0.8140$\pm$0.0019 | 0.8965$\pm$0.0016 | 0.9022$\pm$0.0009 | 0.6409$\pm$0.0065 | 0.6637$\pm$0.0093 | 0.8312$\pm$0.0052 |
| \method-B | ✓ | 0.7826$\pm$0.0065 | 0.6975$\pm$0.0081 | 0.7816$\pm$0.0079 | 0.4560$\pm$0.0048 | 0.4285$\pm$0.0048 | 0.7153$\pm$0.0028 |
| \method-L | ✓ | 0.7876$\pm$0.0034 | 0.7099$\pm$0.0034 | 0.7876$\pm$0.0034 | 0.4132$\pm$0.1235 | 0.4414$\pm$0.0996 | 0.7387$\pm$0.0400 |
| \method-XL | ✓ | 0.7969$\pm$0.0091 | 0.7219$\pm$0.0082 | 0.7884$\pm$0.0194 | 0.4679$\pm$0.0356 | 0.4570$\pm$0.0498 | 0.7359$\pm$0.0219 |

**Table 3: Results on SEED and HMC.**
| Methods | Multi-task | SEED | HMC |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Balanced Acc. | Cohen’s Kappa | Weighted F1 | Balanced Acc. | Cohen’s Kappa | Weighted F1 |  |  |
| SPaRCNet | ✗ | 0.5596$\pm$0.0244 | 0.3464$\pm$0.0372 | 0.5585$\pm$0.0297 | 0.4756$\pm$0.1109 | 0.3147$\pm$0.1315 | 0.4108$\pm$0.1310 |
| ContraWR | ✗ | 0.6106$\pm$0.0078 | 0.4220$\pm$0.0129 | 0.6137$\pm$0.0085 | 0.4242$\pm$0.0541 | 0.2340$\pm$0.0554 | 0.2987$\pm$0.0288 |
| CNN-Transformer | ✗ | 0.6161$\pm$0.0384 | 0.4262$\pm$0.0601 | 0.6150$\pm$0.0463 | 0.6573$\pm$0.0141 | 0.5961$\pm$0.0105 | 0.6896$\pm$0.0065 |
| FFCL | ✗ | 0.5808$\pm$0.0322 | 0.3732$\pm$0.0462 | 0.5743$\pm$0.0402 | 0.4427$\pm$0.0702 | 0.2542$\pm$0.0654 | 0.2902$\pm$0.0485 |
| ST-Transformer | ✗ | 0.5479$\pm$0.0091 | 0.3261$\pm$0.0169 | 0.5505$\pm$0.0091 | 0.2559$\pm$0.0141 | 0.0503$\pm$0.0183 | 0.1428$\pm$0.0122 |
| BIOT | ✗ | 0.7097$\pm$0.0024 | 0.5682$\pm$0.0051 | 0.7134$\pm$0.0027 | 0.6862$\pm$0.0041 | 0.6295$\pm$0.0113 | 0.7091$\pm$0.0147 |
| LaBraM | ✗ | 0.7318$\pm$0.0019 | 0.5994$\pm$0.0031 | 0.7354$\pm$0.0021 | 0.7286$\pm$0.0101 | 0.6812$\pm$0.0073 | 0.7554$\pm$0.0024 |
| \method-B | ✓ | 0.5554$\pm$0.0075 | 0.3393$\pm$0.0117 | 0.5599$\pm$0.0068 | 0.6737$\pm$0.0050 | 0.6188$\pm$0.0057 | 0.7126$\pm$0.0034 |
| \method-L | ✓ | 0.6006$\pm$0.0047 | 0.4067$\pm$0.0063 | 0.6048$\pm$0.0050 | 0.6658$\pm$0.0550 | 0.5929$\pm$0.0715 | 0.6896$\pm$0.0504 |
| \method-XL | ✓ | 0.6034$\pm$0.0010 | 0.4082$\pm$0.0036 | 0.6063$\pm$0.0030 | 0.5761$\pm$0.1084 | 0.4795$\pm$0.1466 | 0.5883$\pm$0.1286 |

**Table 4: Results on Workload and TUSL.**
| Methods | Multi-task | Workload | TUSL |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Balanced Acc. | AUC-PR | AUROC | Balanced Acc. | Cohen’s Kappa | Weighted F1 |  |  |
| SPaRCNet | ✗ | 0.5977$\pm$0.0071 | 0.6638$\pm$0.0314 | 0.6717$\pm$0.0172 | 0.4185$\pm$0.0452 | 0.1399$\pm$0.0799 | 0.3500$\pm$0.0968 |
| ContraWR | ✗ | 0.6966$\pm$0.0332 | 0.7668$\pm$0.0408 | 0.7685$\pm$0.0317 | 0.5857$\pm$0.0662 | 0.3567$\pm$0.0968 | 0.5458$\pm$0.0798 |
| CNN-Transformer | ✗ | 0.5793$\pm$0.0230 | 0.5306$\pm$0.0459 | 0.5663$\pm$0.0349 | 0.3575$\pm$0.0151 | 0.0306$\pm$0.0179 | 0.2235$\pm$0.0251 |
| FFCL | ✗ | 0.7069$\pm$0.0197 | 0.7823$\pm$0.0099 | 0.7857$\pm$0.0234 | 0.3819$\pm$0.0688 | 0.0628$\pm$0.0888 | 0.2120$\pm$0.0786 |
| ST-Transformer | ✗ | 0.6103$\pm$0.0056 | 0.5716$\pm$0.0071 | 0.6375$\pm$0.0078 | 0.4000$\pm$0.0329 | 0.0860$\pm$0.0449 | 0.3793$\pm$0.0459 |
| BIOT | ✗ | 0.6655$\pm$0.0665 | 0.7189$\pm$0.0722 | 0.7342$\pm$0.0536 | 0.5758$\pm$0.0303 | 0.2012$\pm$0.0212 | 0.2394$\pm$0.0040 |
| LaBraM | ✗ | 0.6609$\pm$0.0204 | 0.7174$\pm$0.0234 | 0.7272$\pm$0.0165 | 0.7625$\pm$0.0131 | 0.6407$\pm$0.0304 | 0.7614$\pm$0.0210 |
| \method-B | ✓ | 0.6172$\pm$0.0113 | 0.5824$\pm$0.0080 | 0.6253$\pm$0.0160 | 0.6734$\pm$0.0436 | 0.5107$\pm$0.0617 | 0.6743$\pm$0.0394 |
| \method-L | ✓ | 0.6311$\pm$0.0250 | 0.5869$\pm$0.0155 | 0.6247$\pm$0.0339 | 0.5314$\pm$0.0530 | 0.2961$\pm$0.0810 | 0.5243$\pm$0.0680 |
| \method-XL | ✓ | 0.6345$\pm$0.0442 | 0.5889$\pm$0.0423 | 0.6130$\pm$0.0764 | 0.6845$\pm$0.0304 | 0.5194$\pm$0.0461 | 0.6839$\pm$0.0297 |

### 3.4 Ablation on Robustness

Our instruction design for some datasets (TUEV, HMC, and TUSL) follows multiple-choice questions. To validate the robustness of \method, we enumerate the orders of options and randomly select one from all possible combinations during data fetching of the multi-task instruction tuning stage. Figure 5 illustrates the results on whether shuffling the options. We can conclude that on TUEV and HMC, \methodwith shuffle obtains comparable performance compared to those without shuffle. Nevertheless, it seems that the shuffle operation significantly degrades the performance on TUSL. We attribute this phenomenon to the lack of data for TUSL because TUSL has much fewer number of data samples compared to the other two datasets. It is expected that \methodwill achieve similar results if given more data. In general, \methodhas good robustness against arbitrary order of options, which indicates that \methoddoes understand the linguistic meaning of the questions when predicting.

Figure: Figure 5: Ablation study on whether shuffling the options of instructions.
Refer to caption: x5.png

### 3.5 Ablation on Instruction Data Size

We utilize TUAB, TUEV, and HMC datasets to scale the instruction data size and validate the performance of \methodand other baseline methods, as these three datasets have a relatively large number of samples. The results, illustrated in Figure 6, show that \methoddemonstrates consistent performance under all conditions. For TUAB, \method, LaBraM, and CNN-Transformer exhibit stable performance. For TUEV and HMC, only \methodand LaBraM are relatively unaffected by changes in data size. These findings indicate that \methodis robust and maintains high performance even with varying instruction data sizes, highlighting its effectiveness in multi-task learning scenarios.

Figure: Figure 6: Comparison of different methods under different proportions of instruction data.
Refer to caption: x6.png

### 3.6 Visualization Curves of Multi-channel Autoregression

We visualize the pre-training loss, accuracy, and validation perplexity of \methodin Figure 7. We observe that the loss stably converges while the validation perplexity decreases with training, which means \methodcan generalize well to unseen EEG data. Intuitively, a larger model with more parameters obtains smaller loss and perplexity. Additionally, \method-L achieves similar validation perplexity with \method-XL, indicating that current pre-training data size still cannot satisfy the training with billion-level parameters.

Figure: Figure 7: The training and validation visualization of multi-channel autoregressive pre-training.
Refer to caption: x7.png

### 3.7 Ablation on Multi-channel Autoregressive Pre-training

The proposed multi-channel autoregressive pre-training aims at mimicing current causal LLMs by predicting the next EEG tokens for each channel. It is expected to benefit downstream tasks through learning causal representations. We perform an ablation study to assess the impact of the proposed multi-channel autoregressive pre-training on \method. The results, shown in Figure 8, reveal a significant performance improvement when \methodis pre-trained with this approach, underscoring the effectiveness of multi-channel autoregressive pre-training.

Figure: Figure 8: Ablation study on multi-channel autoregressive pre-training.
Refer to caption: x8.png

## 4 Conclusion

In this paper, we introduce \method, the first universal multi-task foundation model for EEG signal processing. By integrating EEG signals into a Large Language Model framework, \methodleverages advanced text-aligned neural tokenizer embeddings, large-scale multi-channel autoregressive pre-training, and joint multi-task tuning to address the inherent challenges of EEG-based BCI and healthcare tasks. Our extensive experiments across six diverse EEG datasets demonstrate the model’s superior performance in multi-task learning and inference. Overall, \methodrepresents a significant step forward in the field of brain-computer interfaces and healthcare domains, showcasing the great potential of LLMs to revolutionize EEG signal processing and multi-task learning. We believe that \methodwill pave the way for more sophisticated and versatile EEG applications, ultimately enhancing the interaction between humans and machines.

## Appendix A Related Work

Large-scale Pre-training for Neural Signals. With the success of self-supervised learning in computer vision and natural language processing, several studies have emerged to learn effective EEG representations. Kostas et al. () first propose BENDER, which adapts contrastive learning to derive compressed representations from massive EEG datasets. MMM (Yi et al., ) introduces a pre-training framework with multi-dimensional position encoding, multi-level channel hierarchy, and a multi-stage pre-training strategy to learn topology-agnostic representations. BIOT (Yang et al., ) tokenizes diverse biosignals into unified segments, enabling cross-data learning despite mismatched channels, variable lengths, and missing values. Brant (Zhang et al., ) pre-trains on a large corpus of private intracranial EEG data, capturing long-term dependencies, spatial correlations, and both time and frequency domains. Following BIOT, LaBraM (Jiang et al., ) further leverages large-scale 2,500 hours public EEG data, and innovatively introduces a neural tokenizer that encodes continuous EEG signals into discrete codes for masked EEG modeling, thus obtaining a considerable improvement. Unfortunately, all these methods require fine-tuning for specific downstream tasks and cannot perform multi-task learning and inference.

Multimodal Large Language Models. Recent years have seen remarkable achievements of LLMs. In light of the complementarity between language and other modalities, Multimodal Large Language Models have been a rising hotspot. The release of GPT-4 (Achiam et al., ) shows the extraordinary multimodal understanding and generation abilities, thus leading to a research frenzy over MLLMs. LLaVA (Liu et al., ) connects a vision encoder and an LLM, introducing the idea of visual instruction tuning for general-purpose visual and language understanding. Similarly, Zhu et al. () propose MiniGPT-4, which aligns a frozen visual encoder with a frozen advanced LLM, presenting numerous advanced multi-modal abilities. Different from the above MLLMs, CogVLM (Wang et al., ) bridges the gap between the frozen pre-trained LLM and visual encoder by a trainable visual expert in the attention and FFN layers. Chen et al. () present InternVL-1.5, closing the capability gap between open-source and proprietary commercial MLLMs by utilizing a strong vision encoder, dynamic high-resolution, and high-quality bilingual dataset.

## Appendix B Instruction Design

**Table 5: Information of instruction design for downstream datasets.**
| Dataset | Instruction Description |
| --- | --- |
| TUAB | $[$SEP$]$ Question: Is this EEG segment abnormal? Answer: {Yes, No} $[$END$]$ |
| TUEV | $[$SEP$]$ Question: Which event type does this EEG segment belong to? Options: (A) spike and slow wave. (B) generalized periodic epileptiform discharge. (C) periodic lateralized epileptiform discharge. (D) eye movement. (E) artifact. (F) background. Answer: {(A), (B), (C), (D), (E), (F)} $[$END$]$ |
| SEED | $[$SEP$]$ Question: Which emotion type does this EEG segment belong to? Answer: {Positive, Neutral, Negative} $[$END$]$ |
| HMC | $[$SEP$]$ Question: Which sleep type does this EEG segment belong to? Options: (A) Wake. (B) NREM-1. (C) NREM-2. (D) NREM-3. (E) REM. Answer: {(A), (B), (C), (D), (E)} $[$END$]$ |
| Workload | $[$SEP$]$ Question: Is this EEG segment of high workload? Answer: {Yes, No} $[$END$]$ |
| TUSL | $[$SEP$]$ Question: Which type does this EEG segment belong to? Options: (A) background. (B) seizure. (C) slowing. Answer: {(A), (B), (C)} $[$END$]$ |

## Appendix C Pre-training Dataset Description

We utilize multiple EEG datasets with various configurations. The detail information of all the datasets are listed in 6. The total time after data cleaning is close to 25,000 hours.

**Table 6: Information of datasets used for pre-training.**
| Dataset | #Channel | Rate (Hz) | Time (h) | Description |
| --- | --- | --- | --- | --- |
| TUEG (Obeid & Picone, ) | 17-23 | 250-1024 | $\sim$24,000 | A corpus of 26,846 clinical EEG recordings collected at Temple University Hospital. |
| SEED Series (Zheng et al., ; Liu et al., ; ) | 62 | 1000 | 170.54 | These datasets including SEED-IV (15 subjects), SEED-V (20 subjects), SEED-GER (8 subjects), and SEED-FRA (8 subjects) in response to emotional videos. |
| BCI Competition IV-1 (Blankertz et al., ) | 59 | 1000 | 8.21 | A motor imagery dataset containing 2 classes of left hand, right hand, foot (+ idle state) for 7 subjects. |
| Emobrain (Savran et al., ) | 64 | 1024 | 4.94 | A multimodal emotion dataset including 16 subjects. The emotions were elicited through a selected subset of the IAPS dataset. |
| Grasp and Lift (Luciw et al., ) | 32 | 500 | 11.72 | A dataset containing 12 subjects performing grasp-and-lift (GAL) trials. |
| Inria BCI (Margaux et al., ) | 56 | 600 | 29.98 | A P300-based spelling dataset including 26 subjects. |
| Motor Movement/Imagery (Schalk et al., ) | 64 | 160 | 47.3 | A motor imagery dataset consisting of 109 volunteers performing 2 baseline tasks, motor movement, and motor imagery. |
| Raw EEG Data (Trujillo, ) | 64 | 256 | 34.35 | EEG was recorded during reported Information-Integration categorization and reported multidimensional Rule-Based categorization tasks. |
| Resting State (Trujillo et al., ) | 64 | 256 | 3.04 | A dataset comprising 22 subjects for a resting task of 8 mins with 4 mins of eyes closed and 4 mins of eyes open. |
| Siena Scalp EEG Database (Detti et al., ) | 31 | 512 | 30.47 | A database consisting of 14 patients. |
| SPIS Resting State (Torkamani-Azar et al., ) | 64 | 2048 | 0.83 | A dataset including 10 subjects, 2.5 minutes recording in eyes-closed and eyes-open prior to a 105-minute session of Sustained Attention to Response Task with fixed-sequence and varying ISIs. |
| Target Versus Non-Target (Korczowski et al., ) | 32 | 512 | 16 | A dataset including 50 subjects playing Brain Invaders, a visual P300 Brain-Computer Interface using oddball paradigm with adaptive Riemannian Geometry (no-calibration). |
| Self-collected EEG corpus | 62 | 1000 | 342.23 | A mixed self-collected EEG datasets of more than 140 subjects under various conditions. |

## Appendix D Detailed Experimental Settings

### D.1 Hyperparameter Settings

**Table 7: Hyperparameters for neural tokenizer.**
| Hyperparameters | Values |  |
| --- | --- | --- |
| Temporal Encoder | Iput channels | {1,16,16} |
| Output channels | {16,16,16} |  |
| Kernel size | {15,3,3} |  |
| Stride | {8,1,1} |  |
| Padding | {7,1,1} |  |
| Transformer encoder layers | 12 |  |
| Transformer decoder layers | 3 |  |
| Hidden size | 768 |  |
| MLP size | 3072 |  |
| Attention head number | 12 |  |
| Codebook size | 8192$\times$128 |  |
| Batch size | 512 |  |
| Peak learning rate | 5e-5 |  |
| Minimal learning rate | 1e-5 |  |
| Learning rate scheduler | Cosine |  |
| Optimizer | AdamW |  |
| Adam $\beta$ | (0.9,0.999) |  |
| Weight decay | 1e-4 |  |
| Total epochs | 50 |  |
| Warmup epochs | 5 |  |
| Data overlap | None |  |
| Gradient clipping | None |  |

**Table 8: Hyperparameters for autoregressive pre-training.**
| Hyperparameters | \method-B | \method-L | \method-XL |
| --- | --- | --- | --- |
| Model size | 254M | 500M | 1696M |
| Transformer encoder layers | 12 | 24 | 48 |
| Hidden size | 768 | 1024 | 1600 |
| MLP size | 3072 | 4096 | 6400 |
| Attention head number | 12 | 16 | 25 |
| EEG batch size | 480 (B), 512 (L, XL) |  |  |
| Text batch size | 32 (B), 64 (L, XL) |  |  |
| Peak learning rate | 6e-4 |  |  |
| Minimal learning rate | 6e-5 |  |  |
| Learning rate scheduler | Cosine |  |  |
| Optimizer | AdamW |  |  |
| Adam $\beta$ | (0.9,0.95) |  |  |
| Weight decay | 0.1 |  |  |
| Total epochs | 20 |  |  |
| Warmup epochs | 2 |  |  |
| Data overlap | None |  |  |
| Gradient clipping | 1 |  |  |

**Table 9: Hyperparameters for instruction tuning.**
| Hyperparameters | Values |
| --- | --- |
| Instruction batch size | 512 |
| Text batch size | 128 |
| Peak learning rate | 5e-4 (B), 5e-5 (L), 2e-5 (XL) |
| Minimal learning rate | 5e-5 (B), 5e-6 (L), 2e-6 (XL) |
| Learning rate scheduler | Cosine |
| Optimizer | AdamW |
| Adam $\beta$ | (0.9,0.95) |
| Weight decay | 0.1 |
| Total epochs | 5 (B, L), 3 (XL) |
| Warmup ratio | 0.1 |
| Gradient clipping | 1 |

### D.2 Metrics

Considering the class imbalance of most downstream EEG datasets, we use the following metrics for comparison:

- •
Balanced Accuracy: The average of recall (sensitivity) obtained on each class. It is particularly useful for evaluating classification performance on imbalanced datasets. This metric is particularly useful when evaluating models on imbalanced datasets.
- •
AUC-PR: A performance measurement for binary classification problems. It is the area under the curve plotted with precision (y-axis) against recall (x-axis) for different threshold values.
- •
AUROC: It is the area under the curve plotted with the true positive rate (sensitivity) on the y-axis and the false positive rate (1 - specificity) on the x-axis for different threshold values. AUROC provides an aggregate measure of performance across all possible classification thresholds, indicating the ability of the model to distinguish between classes.
- •
Cohen’s Kappa: A measure of agreement between categorical variables $X$ and $Y$, calculated from the observed and expected frequencies on the diagonal of a square contingency table. It is used for multi-class classification.
- •
Weighted F1: The weighted F1 score is the harmonic mean of precision and recall, taking into account the support (the number of true instances) of each class. The weighted F1 score accounts for class imbalance by giving more importance to classes with a higher number of instances.

AUROC and Cohen’s Kappa are used as the monitor score for binary classification and multi-class classification, respectively.

### D.3 Baselines

We mainly consider BIOT (Yang et al., ) and the state-of-the-art EEG foundation model LaBraM (Jiang et al., ) as our baseline method, where BIOT is a generic biosignal learning model pre-trained on multiple datasets in a supervised way, and LaBraM is pre-trained on 2,500 hours data through masked EEG modeling and has learned generic representations for various EEG signals. Five other supervised methods including SPaRCNet (Jing et al., ), ContraWR (Yang et al., ), CNN-Transformer (Peh et al., ), FFCL (Li et al., ), and ST-Transformer (Song et al., ) are also utilized as our baselines. As there are no multi-task methods available in EEG signal processing yet, these baselines are solely fine-tuned on each downstream dataset and cannot perform multiple tasks. We use the default settings for these baselines in the BIOT paper. The batch size is 512 for TUAB, TUEV, SEED, and HMC. As the data size of Workload and TUSL is particularly small, the batch size of these two datasets is set to 32 and 16, respectively.

Figure: (a) TUAB
Refer to caption: x9.png

## Appendix E Attention Visualization

To explore the mechanism of \method, we visualize the attention scores of the answer parts in the instructions for all 12 Transformer layers, as drawn in Figure 9. Firstly, we observed several commonalities across datasets: For the text part, the attention tends to concentrate more in shallow layers whereas for the EEG part, attention gains more in deeper layers. This pattern suggests that \methodprimarily processes text questions in the shallow layers and focuses on EEG tokens in the deeper layers to generate answers. Interestingly, in the case of multiple-choice questions, \methodpays close attention to the options (A, B, C, etc.) between the 6th and 9th layers. Analyzing critical EEG channels for different tasks, we find that \methodseems to aggregate information to Cz for most datasets. For HMC, F4 and C3 are crucial, while O2 is less effective in sleep stage classification.

## Appendix F Analysis of Neural Tokenizer

### F.1 Ablation on Temporal-frequency Prediction

Temporal and frequency domains are two pivotal aspects of EEG signals. To investigate the importance of these two domains for different downstream tasks, we study three variants by setting the reconstruction target in neural tokenizer training as only the temporal domain, only the frequency domain, and both temporal and frequency domains (original \method). Figure 10 shows the comparison between the three variants. Interestingly, it can be found that the temporal domain plays a more crucial role on TUAB, Workload, and TUSL. On the contrary, reconstructing the frequency components obtains better performance on TUEV, SEED, and HMC, indicating that the frequency domain is of great importance for event classification, emotion recognition, and sleep stage classification. By combining the two domains, most tasks achieve similar or higher performance, demonstrating the effectiveness of our neural tokenizer that excavates compact EEG representations for language models.

Figure: Figure 10: Ablation study on reconstructing temporal or frequency domain in neural tokenizer.
Refer to caption: x15.png

### F.2 Visualization of EEG and Text Embeddings

To evaluate the effectiveness of EEG-text embedding space alignment, we visualize the embeddings in Figure 11 using t-SNE (Van der Maaten & Hinton, ). The EEG embeddings expand outside the text space without alignment. In this case, we find that the model fails to predict the answers we expect in multi-task instruction tuning, i.e., the model will output random words instead of options like (A), (B), or (c) in choice questions. When training with alignment, the EEG space mostly aligns with text space, resulting in normal prediction in instruction tuning, proving the necessity of EEG-text alignment.

Figure: Figure 11: Representation visualization of EEG and text by t-SNE. Left: training neural tokenizer without alignment. Right: training neural tokenizer with alignment.
Refer to caption: x16.png

## Appendix G Ablation on Different Pre-training Epochs

We conduct an ablation study on tuning the pre-trained models from different epochs to testify the best pre-training epoch. As shown in Table 10 11 12, we use the pre-trained models of 5, 10, 15, and 20 epochs. Bold represents the best results and underline represents the second best results. It can be found that pre-training for 20 epochs obtains the most bold and underlined results. The performance of 5 epochs gets its best on SEED and HMC while the model from 10 epochs achieves the best result on Workload. Overall, pre-training for more epochs can lead to good performance in different tasks.

**Table 10: Results on TUAB and TUEV.**
| Pre-trained Epochs | TUAB | TUEV |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Balanced Acc. | AUC-PR | AUROC | Balanced Acc. | Cohen’s Kappa | Weighted F1 |  |
| 5 | 0.7763$\pm$0.0037 | 0.6891$\pm$0.0040 | 0.7762$\pm$0.0038 | 0.4609$\pm$0.0696 | 0.3914$\pm$0.0670 | 0.6975$\pm$0.0249 |
| 10 | 0.7738$\pm$0.0040 | 0.6995$\pm$0.0038 | 0.7647$\pm$0.0143 | 0.4693$\pm$0.0175 | 0.4625$\pm$0.0091 | 0.7353$\pm$0.038 |
| 15 | 0.7780$\pm$0.0050 | 0.6951$\pm$0.0054 | 0.7735$\pm$0.0112 | 0.4557$\pm$0.0277 | 0.4216$\pm$0.0215 | 0.7131$\pm$0.0101 |
| 20 | 0.7826$\pm$0.0065 | 0.6975$\pm$0.0081 | 0.7816$\pm$0.0079 | 0.4560$\pm$0.0048 | 0.4285$\pm$0.0048 | 0.7153$\pm$0.0028 |

**Table 11: Results on SEED and HMC.**
| Pre-trained Epochs | SEED | HMC |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Balanced Acc. | Cohen’s Kappa | Weighted F1 | Balanced Acc. | Cohen’s Kappa | Weighted F1 |  |
| 5 | 0.5641$\pm$0.0103 | 0.3505$\pm$0.0172 | 0.5679$\pm$0.0123 | 0.6956$\pm$0.0136 | 0.6269$\pm$0.0053 | 0.7118$\pm$0.0048 |
| 10 | 0.5553$\pm$0.0089 | 0.3376$\pm$0.0143 | 0.5592$\pm$0.0091 | 0.6763$\pm$0.0054 | 0.6161$\pm$0.0069 | 0.7065$\pm$0.0057 |
| 15 | 0.5543$\pm$0.0156 | 0.3365$\pm$0.0244 | 0.5572$\pm$0.0164 | 0.6543$\pm$0.0151 | 0.5991$\pm$0.0153 | 0.6674$\pm$0.0265 |
| 20 | 0.5554$\pm$0.0075 | 0.3393$\pm$0.0117 | 0.5599$\pm$0.0068 | 0.6737$\pm$0.0050 | 0.6188$\pm$0.0057 | 0.7126$\pm$0.0034 |

**Table 12: Results on Workload and TUSL.**
| Pre-trained Epochs | Workload | TUSL |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Balanced Acc. | AUC-PR | AUROC | Balanced Acc. | Cohen’s Kappa | Weighted F1 |  |
| 5 | 0.5816$\pm$0.0235 | 0.5483$\pm$0.0160 | 0.5815$\pm$0.0236 | 0.5342$\pm$0.0235 | 0.2950$\pm$0.0345 | 0.5241$\pm$0.0260 |
| 10 | 0.6540$\pm$0.0192 | 0.6123$\pm$0.0165 | 0.6501$\pm$0.0178 | 0.5920$\pm$0.0560 | 0.3884$\pm$0.0876 | 0.5984$\pm$0.0574 |
| 15 | 0.5701$\pm$0.0282 | 0.5426$\pm$0.0177 | 0.5682$\pm$0.0255 | 0.5910$\pm$0.0629 | 0.3915$\pm$0.0999 | 0.5868$\pm$0.0536 |
| 20 | 0.6172$\pm$0.0113 | 0.5824$\pm$0.0080 | 0.6253$\pm$0.0160 | 0.6734$\pm$0.0436 | 0.5107$\pm$0.0617 | 0.6743$\pm$0.0394 |

## Appendix H Discussion

Limitations. \methodrepresents the first attempt to integrate various EEG downstream tasks into a unified model, achieving promising results across multiple downstream datasets. However, it has some limitations: 1) Although \methodcan surpass certain single-task baselines, it still lags behind state-of-the-art methods that are end-to-end trained on each downstream dataset. 2) \methodis somewhat sensitive to hyperparameter settings, and may not yield satisfactory results without careful tuning. 3) With limited high-quality EEG-text pairs available, this paper only employs coarse-grained alignment between EEG and language, i.e., space-wise alignment, which can pose challenges for LLMs in extracting useful information from EEG tokens.

Outlook. Reflecting on the outlook part of the LaBraM paper, this work explores the first and third suggested directions. Looking ahead, we foresee several potential improvements: 1) Utilizing more advanced LLMs as the base models. While this paper uses GPT-2, a relatively small LLM, and still achieves promising results in the multi-task paradigm, leveraging newer, more advanced open-source LLMs such as LLaMA 3 (Dubey et al., ) may significantly enhance \method’s multi-task learning capabilities. 2) Adopting the mixture-of-experts approach is another promising direction. Given the modality gap between EEG and language, using modality-specific experts may improve multimodal learning with LLMs. 3) Developing finer-grained EEG and text alignment methods, such as describing EEG samples with predefined sentences and aligning EEG and text descriptions at the VQ training stage by adding a contrastive learning loss, may further enhance performance.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. It would be highly beneficial for the research community if you could share the code for NeuroLM as supplementary material. 

2. The expected benefits of large-scale pretraining are not fully realized on the TUAB dataset, with only minimal improvement observed. I was anticipating that the extensive pretraining would lead to more significant performance gains. Could you provide insights into why the pretraining did not result in better performance on TUAB?
soundness: 4
strengths: 1. The paper presents an innovative method that enables handling multiple diverse EEG tasks within a single unified model without the need for task-specific fine-tuning. the authors advance the capability to perform multi-task learning in EEG signal processing, overcoming limitations of traditional models that require separate fine-tuning for each task.

2. The authors conduct thorough experiments on six diverse downstream EEG tasks, including abnormal detection, event type classification, emotion recognition, sleep stage classification, cognitive workload detection, and slowing event classification. This extensive evaluation provides a robust comparison with existing state-of-the-art models, demonstrating the versatility and potential of NeuroLM across different EEG applications.

3. The paper includes detailed ablation studies examining the effectiveness of various components of the proposed model, such as the EEG-text embedding alignment, multi-channel autoregressive pre-training, and the robustness to instruction variations. These analyses validate the contributions of each component, offer insights into the model's performance, and enhance the understanding of how each part contributes to the overall effectiveness.

4. The authors provide clear and detailed descriptions of the model architecture, training procedures, data preprocessing steps, and experimental setups. This transparency facilitates reproducibility and allows other researchers to replicate and build upon the work more easily.
summary: The paper presents NeuroLM, an innovative multi-task foundation model that bridges the gap between electroencephalogram (EEG) signals and language models. By converting EEG signals into discrete tokens aligned with text embeddings, NeuroLM enables Large Language Models (LLMs) to process EEG data effectively. The model employs a neural tokenizer trained through vector-quantized temporal-frequency prediction to encode EEG signals, and integrates these tokens into an LLM using multi-channel autoregressive pre-training. Through multi-task instruction tuning, NeuroLM adapts to various downstream EEG tasks without requiring task-specific fine-tuning. Evaluated on six diverse EEG datasets, NeuroLM demonstrates its potential to unify multiple EEG tasks within a single model.
weaknesses: Overall the paper has a good contribution and is well structured and clear. Here are some tips for improvements: 

1. Even though NeuroLM's performance is currently lower compared to LaBraM on most tasks, the authors could emphasize the benefits brought by the unified instruction tuning approach. Specifically, since NeuroLM is instruction-tuned, it may exhibit better generalization to unseen tasks or prompts, enabling zero-shot or few-shot learning without additional fine-tuning. This generalizability could make NeuroLM more adaptable to new EEG tasks compared to models like LaBraM that require task-specific fine-tuning, thus presenting a significant advantage of their approach.

2. The section on methodology especially the codebook and its origin from VQ-VAE,  and how it functions within NeuroLM, and the stop gradient operator justification in equation (3), could benefit from some explanation.

3. For the EEG-text alignment, while an appendix mentions visualization, a more detailed analysis quantifying the impact of EEG-text alignment on model performance would strengthen the claims.

4. Change the colormap for Figure 1, currently the color for NeuroLM and LaBraM are very similar and hard to distinguish.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. The paper mentions that NeuroLM performs well on several EEG tasks, but is there any evaluation of how well the model generalizes across different datasets?
2. How sensitive is the model to noisy data and how does it perform under varying degrees of EEG signal quality variation?
soundness: 3
strengths: 1. NeuroLM demonstrates for the first time the ability of a model to handle multiple EEG tasks simultaneously with instruction tuning.
2. NeuroLM is evaluated on six different downstream datasets, demonstrating its broad applicability and superior learning capabilities.
3. NeuroLM reduces the need for repeated training through instruction adjustment, making the model more efficient.
summary: This paper introduces a multi-task foundation model called NeuroLM, which is integrated into large language models (LLMS) for electroencephalogram (EEG) signal processing. NeuroLM uses a text-aligned neural tokenizer to encode continuous EEG signals into discrete neural labels by learning through vector-quantized temporal-frequency prediction. These labels are subsequently used as input to the LLM through multi-channel autoregressive pre-training, which enables the model to learn a causal representation of EEG signals. Finally, through multi-task instruction tuning, NeuroLM is able to handle multiple downstream tasks, such as anomaly detection, event type classification, and emotion recognition, in a single model.
weaknesses: 1. While NeuroLM performs well in multi-task learning, there are still gaps in its performance on some tasks compared to state-of-the-art models specifically designed and optimized for a single task (e.g., LaBraM).
2. Model performance fluctuates greatly across different tasks and datasets.
3. Although the model reduces the need for a large amount of labeled data through pre-training and multi-task learning, its performance and generalization ability still rely on high-quality training data.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: - How robust is NeuroLM across different LLM architectures?
- What role does adversarial training play compared to alternative strategies?
- Can you show some examples of fine-tuning that lead to SOTA performance?
soundness: 2
strengths: - Innovative Idea : build the first EEG multi-task foundation model. 
- Trained on large-scale EEG dataset (> 25K hours), diverse data sources enhance model's generalization ability. 
- Consider EEG signals as foreign language and expand scope of LLMs including EEG tasks
- Instruction Tuning for Multi-Tasking: joint multi-task instruction handle various tasks in a unified manner.
summary: Introduction of **NeuroLM**: a multi-task foundational model to process EEG signals. It leverages the capabilities of LLMs, employing a unified approach encoding the signals into discrete tokens. It addresses challenges like the usual need for individual task fine-tuning. 
Model is utilizing a text-aligned neural tokenizer trained using an adversarial strategy and multi-channel autoregressive pre-training to enhance the understanding of the signals. Experimental results demonstrate a competitive performance across 6 different tasks.
weaknesses: - Absence of Fine-Tuning Examples: The authors position NeuroLM as a foundation model and present solid performance across six downstream tasks. However, the study does not include an exploration of fine-tuning capabilities to demonstrate NeuroLM's potential for achieving SOTA performance which could underscore its utility in practical applications.

- Dependency on GPT-2: The authors only test GPT-2 so this study limits insights into the broader potential of this innovative approach. The model aims to bridge EEG and language so evaluating additional language models is necessary for a more comprehensive understanding of the model robustness and adaptability. 

- EEG-Text Alignment: The use of adversarial training is innovative. The absence of ablation studies limits understanding of its specific impact. It could enhance paper's clarity and support the results if it was shown how this approach contributes to the model's performance. 

- Complexity and Reproducibility: The pipeline's complexity may pose challenges for reproducibility. Enhanced clarity in certain sections would better support the community in replicating and extending this work.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: Concern 1:
The paper introduces an EEG-Text Embedding Space Alignment module in Figure 2, built on LaBraM’s neural tokenizer. However, I have concerns regarding the effectiveness and explanation of this alignment.

Fundamentally, EEG and Text have no inherent connection, especially since the EEG signals are not language-evoked. The use of a single gradient reverse layer may only distinguish EEG from text domains rather than fully align their embeddings. To clarify the module's effectiveness, could the authors:

1) Provide visualizations, such as t-SNE plots, comparing EEG and Text embedding spaces before and after alignment to illustrate the degree of alignment achieved.
2) Clarify the source of the Text embeddings and their generation method. The authors do not explain where these embeddings come from or how they are generated, leaving an important gap in understanding.
3) Include an ablation study to quantify the impact of the EEG-Text Embedding Space Alignment module on downstream task performance, showing its necessity and contribution to the overall model.

Concern 2:
The Theory Analysis section in 2.2 of Multi-channel Autoregressive Pre-training provides a high-level explanation by drawing an analogy to VAE, optimizing for reconstruction loss and KL divergence, but it lacks a detailed explanation of how causal relationships between EEG channels are learned or how this approach is theoretically grounded in the autoregressive framework.

To enhance the theoretical foundation of the multi-channel autoregressive approach, could the authors:

Include mathematical formulations demonstrating how this approach models dependencies between EEG channels and captures causal relationships.

Concern 3:
The paper lacks a detailed discussion regarding instruction design and how variations in prompt formulations affect downstream task performance. While instruction tuning addresses multiple tasks, the authors do not examine how different prompts might affect task performance. Could the authors provide an analysis of how different prompts influence task performance?

Concern 4:
The paper does not sufficiently discuss the impact of data curriculum or input order on the results. Although instruction tuning is introduced, there is no exploration of whether sequencing the data in a specific way (e.g., starting with simpler tasks or data) influences model performance across the various downstream tasks. It would be helpful if the authors could provide an analysis of how the ordering of data inputs affects model learning and task outcomes.
soundness: 3
strengths: - The approach of treating EEG signals as a form of language and integrating them with LLMs is novel and creative, offering a new direction for EEG signal processing.

- NeuroLM shows potential in unifying different EEG tasks within a single model, reducing the need for task-specific fine-tuning and streamlining BCI applications.

- The experiments cover six diverse datasets, demonstrating the model's versatility and robustness in multi-task settings.
summary: The paper introduces NeuroLM, a universal multi-task foundation model for EEG signal processing that consists of three key stages: a text-aligned neural tokenizer, multi-channel autoregressive pre-training, and multi-task instruction tuning. Traditional EEG pre-trained models often require full fine-tuning for each downstream task, resulting in resource inefficiencies. NeuroLM addresses this issue by leveraging Large Language Models (LLMs) and treating EEG signals as a foreign language, demonstrating the potential of this multi-task learning paradigm across multiple EEG-related tasks.
weaknesses: My concerns mainly focus on four areas, which are detailed in the Questions section. These include the effectiveness of EEG-Text Embedding Space Alignment, the theoretical foundation of multi-channel autoregressive pre-training, the lack of discussion on instruction design's impact on downstream tasks, and the absence of an analysis on data curriculum or input order. I hope the authors can address these concerns in the rebuttal stage.

Meta-review: The paper presents an innovative method for predicting on diverse EEG tasks within a single unified model without the need for task-specific fine-tuning.

Pros:
- The approach is considered original and novel
- Models are trained on a very large EEG corpus
- Results are provided on diverse set of tasks
- Paper provides an ablation study

Minor concerns:
- Code is not shared and given the complexity of the pipeline reproducibility of the results would certainly be a big challenge for the field. Yet authors promise to release code and weights upon acceptance. Please make sure that the code allows full replication of the results.
- Remaining experiments (impact of LLM, instructions etc.) should be considered but realistically in future works.

As the AC, I would strongly encourage this community to collaborate in order to facilitate benchmarks and comparisons of methods on standard tasks with minimal code changes.

Decision: Accept (Poster)