NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals
Authors: Weibang Jiang
Yansen Wang
Bao-liang Lu
Dongsheng Li
Venue: ICLR 2025 Poster (2025)
Keywords: EEG
large language model
multi-task learning
Primary area: applications to neuroscience & cognitive science
TL;DR: NeuroLM integrates EEG signals and language into a single model, enabling efficient multi-task learning for various EEG applications.

Abstract: Recent advancements for large-scale pre-training with neural signals such as electroencephalogram (EEG) have shown promising results, significantly boosting the development of brain-computer interfaces (BCIs) and healthcare. However, these pre-trained models often require full fine-tuning on each downstream task to achieve substantial improvements, limiting their versatility and usability, and leading to considerable resource wastage. To tackle these challenges, we propose NeuroLM, the first multi-task foundation model that leverages the capabilities of Large Language Models (LLMs) by regarding EEG signals as a foreign language, endowing the model with multi-task learning and inference capabilities. Our approach begins with learning a text-aligned neural tokenizer through vector-quantized temporal-frequency prediction, which encodes EEG signals into discrete neural tokens. These EEG tokens, generated by the frozen vector-quantized (VQ) encoder, are then fed into an LLM that learns causal EEG information via multi-channel autoregression. Consequently, NeuroLM can understand both EEG and language modalities. Finally, multi-task instruction tuning adapts NeuroLM to various downstream tasks. We are the first to demonstrate that, by specific incorporation with LLMs, NeuroLM unifies diverse EEG tasks within a single model through instruction tuning. The largest variant NeuroLM-XL has record-breaking 1.7B parameters for EEG signal processing, and is pre-trained on a large-scale corpus comprising approximately 25,000-hour EEG data. When evaluated on six diverse downstream datasets, NeuroLM showcases the huge potential of this multi-task learning paradigm.

Published as a conference paper at ICLR 2025
A UNIVERSAL MULTI-TASK FOUNDATION MODEL FOR BRIDGING THE GAP BETWEEN LAN- GUAGE AND EEG SIGNALS
NEUROLM:
Wei-Bang Jiang'; Yansen Wang", Bao-LLang Lu', Dongsheng Li²
Shanghai Jiao Tong University 2 Microsoff Research Asia
edu. http://ett..cuu..o.aaaeeeeeeeeee petps://gittnn com/ 0tamm93550 935963004/Neurol.,
dongsii/emicroofftt com
ABSTRACT
Recent advancements for large-scale pre-training with neural signals such elec- troencephalogram (EEG) have shown promising results, significantly boosting the development of brain-computer interfaces (BCIs) and healthcare. However, these pre-trained models often require full fine-tuning on each downstream task achieve substantial improvements, limiting their versatility and usability, and leading to considerable resource wastage. To tackle these challenges, we propose MeuroLM, the first multi-task foundation model that leverages the capabilities of Large Language Models (LLMs) by regarding EEG signals foreign lan- guage, endowing the model with multi-task learning and inference capabilities. Our approach begins with learning a text-aligned neural tokenizer through vector- quantized
prediction, which encodes EEG signals into dis- crete neural tokens. These EEG tokens, generated by the frozen rector-quantized (VQ) encoder, are then fed into an LLM that Iearns causal EEG information via multi-channel autoregression. Consequently, NeuroLM can understand both EEG and language modalities. Finally, multi-task instruction tuning adapts MeuroLM various downstream tasks. We are the first to demonstrate that, by specific incorporation with LLMs, NeuroLM unifies diverse EEG tasks within a single model through instruction tuning. The largest variant NeuroLM-XL has record- breaking 1.7B parameters for EEG signal processing, and is pre- prainedined on a large- scale corpus comprising approximately 25, 25,000-hour EEG data. When evaluated on six diverse downstream datasets, NeuroI.M showcases the huge potential of this multi-task learning paradigm.
INTRODUCTION
Electroenccephalogram (EEG) signals have become a corner- stone in the development of brain-computer interfaces and healthcare domains, offering non-invasive solution to capture the electrical activity of the brain. EEG measures the voltage fluctuations resulting from ionic current flows within the neu- rons of the brain, providing real-time insights into brain func- tion and neural dynamics. This capability makes EEG an in- valuable tool for creating interfaces that enable direct commu- nication between the brain and external devices. EEG is advan- tageous due to its high temporal resolution, cost-effecttveeness, and portability, and has been significantly enhanced by ad- vanced computational methods. Therefore, a wide range of applications have been utilizing EEG signals, including but not limited to human emotion recognition (Jenke et al.] 2014).
teat Chall feaeeeeeeen
Confralio
Koag
Sexctal
Charltaate
Figure 1: Comparison on six tasks.
body motor imaginary (Tabar & Halici| (0166), automatic sleep stage classification (Supratak al. 2017). seizure epilepsy detection (Alotaiby et al. 2014), and fatigue detection (Gao et al. 2019).
Work done during Wei-Bang' intersship at Microsoft Research Asia. Correspondence to Yansen Wang.

---

Published as a conference paper at ICLR 2025
While EEG signals are popular among researchers, they have several disadvantages, including the low signal-to-noise ratio, inherent nonstationarity, as well as diverse configurations in EEG data collection. Besides, there is lack of sufficient and consistent EEG data. These challenges compli- cate the extraction of universal EEG representations. To overcome these problems, several studies have proposed methods compatible with diverse EEG configurations to learn effective and generic representations. For example. Yang et al., (023a)) introduce a Biosignal Transformer (BIOT), which unifies various EEG data by tokenizing channels into fix-length segments with channel and relative position embeddings for preserving spatio-temporal features. Jiang et al. (2024) advance this ap- proach by proposing neural tokenizer to pre-train LaBraM by masked neural code prediction with 2,500 hours of EEG data, thus achieving state-of-the-art (SOTA) performance on various down- stream tasks. Although these methods effectively address the aforementioned challenges, they still require individual fine-tuning on each downstream dataset to obtain impressive improvement. De- spite increasing model size and employing large-scale unsupervised pre- pre-training to learn generic representations, such adaptation confines the fine-tuned model to perform only single task. More- over, this task-specific fine-tuning demands substantial computational and storage resources.
Over the past few years, the advent of Large Language Models has brought remarkable progress and demonstrated extraordinary emergent abilities (Brown et al. 2020; Touvron et al., (023). The development of LLMs has given rise Multimodal Large Language Models (MLLMS) (Achiam let al. 2023 L Liu et al. 2023), which unleash the potential of powerful LLMS to perform multimodal tasks. MLLMs typically integrate a modality-specific encoder, pre-aligned with text embeddings, into off-the-shelf LLMs. Inspired by MLLMs, we unveil a new direction of integrating multiple EEG tasks into a unified model by incorporating EEG signals into existing LLMs. However, there are some challenges in harnessing LLMs to understand EEG patterns, comprising:
1) EEG-text embedding alignment. Aligning EEG and text embeddings presents a great challenge. models which benefit from numerous sishh-qality image-text pairs, there are no established FEGGttett pairs available due to the difficulty of extracting semantic information from given EEG segment.
Unlike
2) Effective Representation learning with LLMs. Mainstream methods employ masked EEG modeling to effectively extract representations for EEG signals. When integrating LLMs, how to learn generic information within the LLM paradigm remains an unsolved issue.
3) Unified multi-task learning with various EEG tasks. Integrating multiple EEG tasks into unified model is complex due to the diversity and specificity of different tasks. Developing a model that can seamlessly handle various tasks without compromising performance any individual task is a major challenge.
In light of the aforementioned challenges, propose NeuroLM, universal multi-task foundation model for EEG signal processing. NeuroLM builds upon the compatibility with diverse EEG for- mats established by LaBraM, and it pre-trained on a large-scatl scale dataset comprising approximately 25,000 hours of EEG data. The training of NeuroLM involves three stages. First, a text-aligned neu- ral tokenizer is trained using rector-quantized EEG signals into discrete codes from a neural codebook, with adversarial training employed to align the EEG and text spaces. Next, the VQ encoder of the neural tokenizer is frozen to extract compact embeddings, which serve as input for a L.M. To enable the L.MM to learn causal EEG representa- tions, we propose multi-channel autoregressive pre-training. which mimics autoregressive language modeling but is tailored for multi-channe EEG signals. Finally, we elaborate instructions for various downstream datasets and employ multi-task instruction tuning to empower MeuroLM for multi-task learning. Experiments on six different tasks, encompassing abnormal detection, event type classifi- cation, emotion recogmition, sleep stage classification, cognitive workload prediction, and slowing type classification, demonstrate NeuroLM's superiority in multi-task learning and inference. To the best of our knowledge, we are the first to introduce instruction tuning to enable multi-task learning and inference in the field of EEG signal processing. The highlights are summarized as follows:
camporal-frequenge prediction to encode continuous
1) Text-aligned neural tokenizer embeddings. We introduce a text-aligned neural tokenizer that effectively bridges the gap between EEG and text data. This tokenizer uses vector-quantized cmmporal-frqquene
prediction to convert EEG signals into discrete codes, facilitating the alignment of EEG and text embeddings through adversarial training. This alignment crucial for leveraging the strengths of LLMs in understanding and processing EEG data.
2

---

Published as a conference paper at ICLR 2025
BBG signals
VQ Eneoder
Codebook
Temporaa Encoder
Sperial Encoder
Temponal
look
Spotial & Tempural Embookinge
G.
PCI Vyleomoeent
Com
Group Norm
GELA
fermer Blocks
Temppord SOODENORACAAN
Pa-2
Frequanncy Dacoder
M5 Mymayoontt
Pt-1
01 Puapeberryerty
Frepurncy
revanse gradieut
Text or
BEG-Text Embeddlig Space Aligament
Clasetier
Domain
Text vocah
EBG
Figure 2: The architecture design of text-aligned neural tokenizer training. The neural tokenizer is
trained by reconstructing both temporal and frequency domain of input EEG signals to discretize them into discrete neural tokens. To align EEG and text embedding space, we utilize a domain classifier through adversarial training.
2) Large-scale multi-channel autoregressive pre-training. NeuroI employs multi-channe au- toregression, enabling the model to learn causal representations across different EEG channels. Pre- training on 25,000 hours of EEG data ensures that NeuroLM captures wide range of neural pat- terns, enhancing its ability to generalize across diverse EEG tasks.
3) Joint multi-task tuning and inference. We pioneer the use of joint multi-task tuning and in- ference for EEG. By elaborating specific instructions for various downstream tasks and employing multi-task instruction tuning, MeuroLM is capable of performing multiple tasks within single model. This not only improves efficiency by reducing the need for individual fine-tuning for each task but also ensures high performance across a spectrum of applications.
METHOD
In this section, we elaborate design of NeuroLM. We first train neural tokenizer by vector- quantized temporal-frequenee prediction. Whereafter, the VQ encoder of the tokenizer will serve encode EEG signals embeddings aligned with text space, and the EEG embeddings will be seamlessly used as input to Large Language Models.
RCXT, where C denotes the number of channels and T denotes total timestamps. An EEG sample is formulated as e gCxL, where L is the window size, resulting in total number of f samples. We pachify the EEG samples into non-overlap patches x {Ti E R = 1,... C,j = 1.., NJ. Let P is patch size and = [
Given multi-channel EEG signals X
2.1 TEXT-ALIGNED NEURAL TOKENIZER TRAINING
To incorporate EEG into thf-the-shelf Large Language Models, we first need encode EEG signals into embeddings whose space is well-aligned with text embedding space. VQ-VAE (Van Den Oord al., (017) is a good choice that maps continuous signals discrete tokens while preserving the key information. Our text-aligned neural tokenizer basically follows the well-established neural tokenizer of LaBraM (Jiang et al. 2024) with some improvements. Vector-quantized temporal- frequency prediction is utilized to train the text-aligned neural tokenizer, as illustrated in Figure ?
Neural Tokenizer. The neural tokenizer composed of several vital components: VQ encoder, codebook, temporal/frequency decoder, and domain classifier. The codebook V E KxD contains K discrete D-dimension embeddings. Let f;; denote the patch representations derived from the VQ encoder. We find the nearest codes of each h; from codebook embeddings {culi 1, 1,... K}:
2 arg min (1,) (2(12)l22
(I)
where j E (1,..., K} and & normalization employed so that the above distance is equivalent to cosine similarity. Consequently, an EEG sample is tokenized to [21, 2N].
3

---

Published as a conference paper at ICLR 2025
Temporal-frequency Prediction. We propose predict both original signals and the frequency magnitude to capture the temporal and frequency domains of EEG signals. This differs from LaBraM which regresses the Fourier amplitude and phase since observe that reconstructing the phase contributes minor to neural tokenizer training. We apply the Discrete Fourier Transform (2),,,,),....,,111 of channel i and time j, and transform the
(DFT) EEG patch vij equation using Euler's formula as follows
M
2 mm) ju(m] sin(
2 M mn).
Σ
xx1 cos(
M
where E [] and j is the imaginary unit. Accordingly, we calculate the frequency magnitude as fm = Re(x))))) Im( where Re and Im represent the real and imaginary parts of a complex number. For stable convergence, we adopt z-score normalization to the magnitude within sample.
After being quantized to the codebook embeddings, we feed the normalized neural embeddings ((2(z))...,, [2x)) into two separate decoders. Let of and of stand for the output of a temporal decoder and frequency decoder, respectively. The optimizing target for the codebook learning is
L1
Σ RED
Σ
10 x, 2 + oof Sill2 + [(sg(f((h)) (2( (16) (2((2((1)))))))) ))
reconstruction
codebook
commitment
where D represents the whole dataset and sg denotes the hop-gradient operator that is identical during forward computation and has zero partial derivatives.
EEG-text Embedding Space Alignment. Current vision-language models usually utilize pre- trained CLIP-like (Radford al., 2021) image encoders which trained by large-scale image-text pairs and thus are embedding wise well-aligned with text. However, when considering EEG, there much more challenges align EEG with text: 1) EEG signals contain complicated cognitive and non-cognitive information, which is hard to be described by human language accurately and thor- oughly, For example, an EEG segment can not only contain one person's 's emotion and mental states, but also represent the body movement and medical normality. 2) The labeled EEG data available to construct EEG-text pair are very limited. Therefore, we propose align EEG with text space-wise wiiee instead of embedding-wise.
We introduce a domain classifier C to predict whether the embeddings are from EEG or text. During the codebook learning, we also feed some text embeddings from LLMs to train the domain classifier. gradient reverse layer (Ganin et al. 2016) is added after the VQ encoder to confuse the domain classifier. Hence, the embeddings from the VQ encoder fall into the same space of text embeddings. Consequently, the training objective for extt-aliged neural tokenizer training is defined as
min Lı > Σ di log C(hi);
2 where d is the label of EEG or text domain and 101- 10//r 1 a scaling factor that gradually
changes from to 1.
VQ Encoder Architecture. We briefly introduce the architecture of the VQ encoder as it is almost the same as LaBraM. The temporal encoder and spatial encoder are two pivotal parts of the VQ
encoder. The temporal encoder contains several blocks of 1-D convolution which aims extract
temporal features in each EEG patch. After that, learnable temporal and spatial embeddings are added according to the standard 10-20 international system inject both time and channel infor- mation. Finally, the spatial encoder composed of vanilla Transformer blocks (Vaswani et al. 2017 learns interaction among patches.
2.2 MULTI-CHANNEL AUTOREGRESSIVE PRE-TRAINING
Before passing EEG data into Large Language Models, we freeze the VQ encoder and first it to encode input EEG data to EEG tokens that are aligned with the text space. After that, we load a pre-trained Large Language Model and enlarge the text vocabulary with the learned EEG codebook.
4

---

Published as a conference paper at ICLR 2025
Text vocah
Neural Tokeriierr
Large Language Models (consal Transfocmer)
Codebook Indiees
Laage Language Models
[ST2]
posttce
(causal Trinsfomee)
EEG sluza
VQ
Build Instructions
di
<is
dii
a12
Downstream Dutnsets
abnormal deticttion
emotion reongrition
VQ Encoder
event classification
sleep classification
Stage Multi-channel autoregressive pre treitrann
Stage Joint instruction tuning with various taoks
Figure 3: Schematic of NeuroLM training. Left: We first pre-train NeuroLM via multi-channel autoregression with EEG tokens output by the frozen VQ encoder. Right: The multi-task instruction tuning enables MeuroLM to perform various BCI tasks within single model.
The EEG tokens are added with reused temporal embeddings from the LLM and new spatial embed- dings. As shown in Figure 3 MeuroLM is then trained through multi-channel autoregression, that is, predicting the next EEG tokens based on visible EEG tokens, to endow the model with the capa- bility of learning special patterns of EEG causal relationship. In our experiments, the multi-channel autoregressive pre-training contributes to the performance of multi-task instruction tuning.
Formulation. Consider a sequence of EEG tokens h (heyli = 1,....C. C = I,... T} where denotes the channel and j denote the time, and their corresponding indices of the merged text and EEG vocabulary = {Lylh 1,... C.j T) derived from the neural tokenizer. Unlike language that can predicted token by token intuitively, EEG signals of various configurations, thus it is impracticable directly predict EEG tokens one by one. We propose a multi-channel autoregressive strategy adopt the idea of autoregression on EEG. The basic idea is that each token of specific channel predicts the next token of the same channel, which can be for- mulized as
Cimee
462
Fime
2
<h2
<13
ch2
Lch
The stair-stepping mask. Each row indicates atten- tion masks for an EEG token.
Figure 4:
p(111 I12, ICT) II p(Tnns I2n, Ica h1, h12,, (i,,,,,,hh.hh,.
t=1
Therefore, the objective for multi-channel autoregressive pre-training is to optimize model parame- ters by maximizing p(hu,, hat, hot /,,,,,,,,,,,,,.,...,, /o,,--.,,h(.- throughout all EEG data.
For implementation, we define stair-stepping masks where each EEG token is able to observe tokens of all channels from its current and previous time step. Figure illustrates the design of our stair- stepping mask. Dark cells indicate that the elements should take part in attention.
Theory Analysis. We interpret the multi-channel autoregressive pre-training from the view of variational autoencoder (Kingma & Welling 2014). Let denote the original EEG signals, denote the temporal- -rruuuuncfrrecyy target of x, and be the EEG tokens be predicted. Assume that EEG signals x can be generated by random process with latent variable z. We use (z (Z.ar.)) to denote the VQ encoder encoding EÉG signals into discrete neural codes, Pe: (34]=;) stand for the temporal and frequency decoder reconstructing emporal-frrquenge domain from encoded neural codes, and po(z (z(z))))) to represent multi-channel autoregressive pre-training. Consider the log-likelihood p(yla) (x) and its evidence lower bound (ELBO), involving predicting the temporal-frequencc domain of the EEG signals from the next time point:
Σ ((p((( 2
Σ (t log P
Pe (v.l=)) (L(ga())aa) po())()))
where the first term is the reconstruction loss and the second term is Kullback-Leiller divergence between g and EEG-text conditional prior. Our training paradigm encompasses two-stage learning
5

---

Published as a conference paper at ICLR 2025
processes: 1) The neural tokenizer is optimized by minimizing the reconstruction loss. 2) A LLM learns the prior pe by minimizing LL loss with de and Pe fixed. The sequence 2, can be sampled from qa (z) one-point distribution 2; arg max; (z(z)z,) where choose the latter for simplicity. In this case, 2 is from the codebook V and 2 = [a.,. z.rr.. Therefore, Equation can be rewritten
T
Σ (B (Barn log (ws
Σ /=2
log po (25,5 (r,,))));
where the latter term the negative log-likelihood loss for multi-channel autoregressive pre-training and Fi.j denotes latent variables of all channels at time step j.
2.3 MULTI-TASK INSTRUCTION TUNING
In this stage, we aim to leverage the power of LLMs to integrate different downstream datasets as a whole. Instruction tuning is introduced handle various downstream tasks, shown in Figure is worthwhile to mention that in both multi-channel autoregressive pre-training and multi-task instruction tuning stages, we feed the model a few text data each iteration preserve the language modeling capability of LLMs. We build instructions for each downstream dataset and the instruction design can be found in Appendix B A special token [SEP] is used to concatenate EEG and text instructions, indicating the modality switch. Notably, the loss is only calculated on the answer part of the text to make the prediction more stable. Suppose x" represents the EEG tokens along with the question part of the instruction (prompt), and t represents the answer part of the instruction. Let the sequence length of be L, and this procedure can be written as
p(t xP)
II p(t re to t.).
(8)
where Ee is the answer tokens before the current prediction token 49.
3 EXPERIMENTS
3.1 DOWNSREAM DATASETS
We consider six different EEG datasets with highly varied data sizes to comprehensively evalu- ate Neurol where the detailed information is listed in Table [ 1) TUAB (Harati et al. 2015) (abnormal detection): This dataset contains EEG records that are classified as clinically normal or abnormal. 2) TUEV (Harati et al. 2015) (event type classification): This corpus contains six events involving periodic lateralized epileptiform discharge, generalized periodic epileptiform discharge, spike and/or sharp wave discharges, artifact, and eye movement. 3) SEED (Zheng & Lu 2015) (emotion recognition): There are emotions (positive, negative, and neutral) elicited by videos from 15 subjects. There are 15 trials in each session and each subject underwent sessions. 4) HMC Alvarez-Estevezz & Rijsman 202I) (sleep stage classification): HMC was developed for au- tomatic sleep scoring, involving 5 sleep stages (wake, NREM- NREM-2, NREM-3, REM) from 151 subjects. 5) Workload (Zyma et al.] 2019) (cogmitive workload classification): This dataset contains 36 subjects performing serial subtraction. We regard mental workload trials as high work- load and the last 60 seconds of the rest EEG as low workload. 6) TUSL (von Weltin et al., 2017)
Table 1: Information of datasets used for downstream evaluation.
Dataset
#Channel
Sampling Rate
Duration
#Sample
Task
TUAB
23
256 Hz
10 seconds
409,455
Binary classification
TUEV
23
256 Hz
seconds
112,491
6-class classification
SEED
62
1000 Hz
seconds
38,475
3-class classification
HMC
4
256 Hz
30 seconds
137,243
5-class classification
Workload
19
500 Hz
seconds
2,088
Binary classification
TUSL
23
256 Hz
10 seconds
245
3-class classification

---

Published as a conference paper at ICLR 2025
(slowing event classification): TUSL aims to differentiate between seizure, slowing, and complex background events.
For the data division, we split each dataset into training, validation, and test sets: 1) TUAB and Since the training and test division is provided by the original datasets, we further divide the training patients into training and validation groups by 80% and 20% randomly. 2) SEED: We split total 15 trials into training, validation, and test trials by 9:3:3 according to the chronological order, and merge all sessions into the final training, validation, and test set. 3) HMC: The first 100 subjects form the training set while the middle 25 subjects and the last 26 subjects are validation and test sets, respectively. 4) Workload: The training, validation, and test sets are derived by subjects from number O to 25, number 26 to 30, and number 31 to 35, respectively. 5) TUSL: The training, validation, and test sets are splitted by 60%:20%:00%
TUEV:
3.2 EXPERIMENTAL SETUP
Model Configurationss NeuroLM is compatible with any causal LLM as its base language model. For simplicity and saving computing resources, we adopt GPT-2 (Radford al. 2019) base language model. Accordingly, NeuroLM has three variants, NeuroL.M-B, NeuroL.MLL, and MeuroLM-XL, which have 254M 500M, and 1696M parameters (including the parameters of the VQ encoder), respectively. Unless otherwise noted, NeuroLM refers to NeuroLM-B, Text embed- dings which are randomly sampled from GPT-2's vocabulary for each batch, are utilized for EEG- text alignment. The patch size P is set to 200 (1 second), consistent with that of LaBraM. To maintain compatibility with GPT-2, the maximum sequence length (number of patches) is set to 1024. For input samples with sequence lengths shorter than 1024, we pad zeros to ensure the length equal to 1024 at neural tokenizer training and multi-channel autoregressive pre-training stages. The attention values for these zero paddings will be masked.
Data Preprocessing. To eliminate environmental and physiological artifacts from EEG signals, we employ several necessary preprocessing methods. First, we apply bandpass filter with cutoff frequencies of 0.1 Hz and 75 Hz. To avoid power-line interference, we use a notch filter at 50 Hz or 60 Hz. depending on the geographic region of data collection. Additionally, all signals are resampled to 200 Hz to reduce computational complexity. Given that EEG signal values typically range between- 100 vV to 100 VV all values are divided by 100 for normalization.
Training & Environment Settings. To facilitate the training of NeuroL.,, a huge volumn of data is required. About 25,000 hours of EEG data from multiple public EEG datasets are collected after cleaning and filtering, which are listed in Appendix C All experiments are conducted on eight NVIDIA A100-80G GPUs with Python 3.8 and PyTorch 2.2.2 + CUDA 12.1. For instruction tuning, the results are obtained using the final model after training. Notably, we choose the largest logits as the prediction evaluation and test instead of beam search which is widely used in current LLMMs to obtain stable results. For other scenarios, the baselines are in a single-task manner and trained on individual datasets. Their best models are selected based on the best performance on the validation set, and then evaluated on the test set. The average and standard deviation values are reported using three random seeds to ensure comparable results. Baselines and other detailed hyperparameter settings are provided in Appendix D
3.3 EXPERIMENTAL RESULTS
We present all results in Table 3 and 4 Underlined values represent the best results for single- task methods, while bold values indicate the best results for NeuroLM. Notably, it's important to note that direct comparisons between NeuroLM and the baseline single-task methods are not en- tirely fair, as the baselines are trained and tested on individual datasets. Although NeuroLM is still a few steps away from the vaae-oo-thhear LaBraM, achieves performance comparable to most other single-task baselines. The key strength of NeuroL. lies in its unified instruction-tunn -uuning, which has the potential to enable generalization to novel tasks or prompts without the need for extensive task- specific fine-tuning. For MeuroLM-L and MeLL the performance is further enhanced on most downstream datasets with larger model capacity. However, the imbalance in data size among different downstream datasets poses challenge for NeuroLM, it reaches optimal performance different training times for different datasets. Additionally, we find that models with more parame- ters are more prone to overfitting, which might account for the performance degradation observed on
7

---

Published as a conference paper at ICLR 2025
Table 2:
Results on TUAB and TUEV.
Methods
Multi-task
TUAB
TUEV
Balanced Acc.
AUC-PR
AUROC
Balanced Ace.
Coben's Kappa
Weighted F1
SPaRCNet
X
0.789620.0018
0.841120.0018
0.867620.0012
0:4161=0.0262
0,423340.0181
01102420.0104
ContraWR
X
0.7746+0.004
0.8421000104
0.8456+0.0074
0.384-0.0349
0.3912+0.0237
0.093
CNN-Tramsforn Tramsfommr
0.7777.0.0022
0.8433.000009
0.846110.0013
0.408710.0161
0.381511000134
0.85410.0293
FFCL
X
0.7848:0.0038
0.848=00.006
0.8569=0.0051
0.3979=0.0104
0.3732:40.00 01.0000018 88
0.6783=0.0130
ST-Transformer
X
0.0003- 0.8521-0.000
0.8707+000019
0.3964+0.0228
0.3765 03765-+0.0306
0.6823+0.0190
BIOT
X
0.795911.0055
0.8792=0.0023
0.881520.00433
0.5273:10.0249 0.5281=000225
0.749220000822
LaBraM-Base
X
0.8140+0.0019
0.8965=00000 0016
0.9022+0.0009
0.4490000005
0.6637+000093
0.8312+0.0052
M-u
0.782610.0065
0.6975=0.0081
0.781620.0079
0.456020.0048
0,4265110.0488
0.71532000028
M-LL
0.7876+0.0034
0.7099=0.0034
0,7876+0.0034
0.4132=0. 1235
0,4414+0.0996
0.7387=+00000
NeuroLM-XX
0.796910.0091
0.7219-0.0082
0.788410.0194
0.4679.000038
0.4570.10.0498
0.7359
Table 3: Results on SEED and HMC.
Methods
Multi-task
SEED
HMC
Ralanced Acc.
Cohen's Kappa
Weighted FI
Balanced Acc.
Cohen's Kappa
Weighted
SPaRCNet
x
0.559620.0244
0.346420.0372
0.5585=0.0297
0.4756=0.110
0.3147:0.1315
0:110==0:1110
ContraWR
X
0.6106+0.0078
0.4220-001299
0.424++0.0541 0.613770.0085
0,2340+0.0554
0.2987+0.0288
CNN-Transfomer
X
0.61611000384
0.4262=0.0601
0.615011..063
0.657320:0141
0.596110.0105
0.689610.0065
FFCL
X
0.5808+0.0822 0.3732=0 0462
0.5743+0.0402
0.4427=0.0022
0,2542:0.000 0.644
0.2902=000088
ST.Transformer
X
0.5479 0.0211=0.0169
0.5005
0.2559:0.0141
0.0503+10.0183
0.1428:-0.0122
BIOT
X
0.709711.0024 0.5682=0.0051
0.713420.0027
0.6862=000041
0.6295:10.0113
0.709120.0117
LaBraM-Base
X
0,7318+0000099
0.5994=0.0031
0.7354=0.0021
0.7286=0.0101
0.6812+000073
0.7554=0.0024
Neurol M-B
0.5554+0.0075
0.3393-0.0117
0.5599+0.0068
0.6737=000050
0.6188 +0.0057
0.7126+0.0034
M-LroLM-L
0.600610.0047
0.4067=0.0063
0.604810.0050
0.665820.0550
0.59291100015
0.689610.0004
MeuroLM-XL
0.6034:0.0010
0.4082:0.0036
0.6063=0.0030
0,5761=0. 1084
0,4795:+0.1466
0,5883=0. 1266
Table 4: Results on Workload and TUSL.
Methods
Multi-task
Worklond
TUSL,
Rallancdd Ace.
AUC-PR
AUROC
Balanced Ane.
Cohen's Kappa
Weighted Fl
SPaRCNet
X
0.597710.0071
0.6638-0.0014
0.671710.0172
0.0155.0.0452
0.139910.0799
0.3500-0.0968
ContraWR
X
0.69662000332
0.7668=0.4408
0.768520.0517
0.5657=0.0662
0.356710.0966
0.545820.0798
Trn-Tranffrrmer
X
0.5793+-0.0230
0.5306-0.0459
0.5663+0003399
0.3575=0.0151
0.0306+00.0179 79
0.2235+00.0251
FFCL
0.7069
0.7823-0.0099
0.7857
0.381910.0688
0.0628.10.0888
0.212010.00866
ST-Transformee
0.6103=20.0066
0.5716=0.0071
0.6375=00.00 0078
0.4000:20.3329
0.0860:100444
0.3793=0.0459
BIOT
0.6655+0.0665
0.7189-0.0722
0.7342+0.0536
0.5758-0.0303
0.2012+40.0212
0.2394+0.0040
LaBruM-Base
0.660910.0204
0.717420.0034
0.727210.0065
0.7625:000131
0.6407
0.714420.0200
Neurol.M-B
0.6172+0.0113
0.5824-0.0080
0.6253:000160
066344:000466
0.5107:10.0617
0.0794
MeuroLM-L
0.631311.0250
0.586920.0155
0.624720.0539
0.531420.0530
0.296110.0810
0.524320.0680
MeWroLM-XI.
0.6345:0.0422
0.589900:0433
0.61 30+0000644
0.6845=0.0304
0.5194#0.0461
0.6899=0.0297
HMC since sleep patterns are of low complexity and smaller models might be sufficient to capture the relevant features in EEG signals. On TUSL, the model performance appears to be not very stable due to the extremely limited data samples.
3.4 ABLATION ON ROBUSTNESS
Our instruction design for some datasets (TUEV, HMC, and TUSL) follows multiple-choice ques- tions. To validate the robustness of NeuroLM. we enumerate the orders of options and randomly select one from all possible combinations during data fetching of the multi-task instruction tuning stage. Figure [3) illustrates the results on whether shuffling the options. We can conclude that on TUEV and HMC, NeuroLM with shuffle obtains comparable performance compared to those with- out shuffle. Nevertheless, it seems that the shuffle operation significantly degrades the performance on TUSL. We attribute this phenomenon to the lack of data for TUSL because TUSL has much fewer number of data samples compared to the other two datasets. It is expected that MeuroLM will achieve similar results if given more data. In general, NeuroLM has good robustness against arbitrary order of options, which indicates that NeuroLM does understand the linguistic meaning of the questions when predicting.
8

---

Published as a conference paper at ICLR 2025
TUEV
HMC
TUSL
1.0
1.0
1.0
shuffle shuftle
shufite
W/ shutfle
shuffil
0.8
0.4
0.4
0.4
0.2
0.2
0.2
Bulunced Accuracy Kappa
Weighted
VallaceedAAscuraaa
Kappa
Weighted
Ballnced Accunacy Kappo
Weighted
Figure 5: Ablation study whether shuffling the options of instructions.
3.5 ABLATION INSTRUCTION DATA SIZE
We utilize TUAB, TUEV, and HMC datasets to scale the instruction data size and validate the per- formance of MeuroLM and other baseline methods, as these three datasets have a relatively large number of samples. The results, illustrated in Figure 6 show that NeuroLM demonstrates consistent performance under conditions. For TUAB, NeuroLM, LaBraM, and CNN-Transforner exhibit stable performance. For TUEV and HMC, only NeuroLM and LaBraM are relatively unaffected by changes in data size. These findings indicate that NeuroLM is robust and maintains high perfor- mance even with varying instruction data sizes, highlighting its effectiveness in multi-task learning scenarios.
T1AA Accunsyy)
(AUBOC)
0.8
175
0.75
0.82
0.0
1.5
11
Tememage
Peusrago (1)
Panergg
Accemny)
TUEV Keypo)
(Wrighezed F1)
0.5
0.8
0.5
0.4
0.1
0.2
20
166
56
160
16
6
1/6
Tenerage
Peumtage (1)
Parerage
Ancumsy)
EMC Kepps)
(Weighted Fi)
0.7
0.5
0.5
0.2
0.8
0.5
0.
0.2
18
106
20
26
180
18
60
Pamerage
Poursigg
Panolage
Figure 6: Comparison of different methods under different proportions of instruction data.
3.6 VISUALIZATION CURVES OF MULTI-CHANNEL AUTOREGRESSION
We visualize the pre-training loss, accuracy, and validation perplexity of NeuroLM in Figure 7 We observe that the loss stably converges while the validation perplexity decreases with training, which means NeuroLM can generalize well to unseen EEG data. Intuitively. a larger model with more parameters obtains smaller loss and perplexity. Additionally, M-LL achieves similar

---

Published as a conference paper at ICLR 2025
validation perplexity with NeuroLM-XX. indicating that current pre-training data size still cannot satisfy the training with billion-level parameters.
0.30
1000
0.25
Validaties proploaity
Prestiiinn Looe
0.10
0.05
0.00
41k
20k
40k
tecation
iterations
Mpocht
Figure 7: The training and validation visualization of multi-channel autoregressive pre-training.
3.7 ABLATION ON MULTI-CHANNEL AUTOREGRESSIVE PRE-TRAINING
The proposed multi-channel autoregressive pre-training aims mimicing current causal L.MSs by predicting the next EEG tokens for each channel. It is expected to benefit downstream tasks through learning causal representations. We perform an ablation study to assess the impact of the proposed multi-channel autoregressive pre-training on NeuroLM. The results, shown in Figure & reveal a sig- nificant performance improvement when MeuroLM is pre-trained with this approach, underscoring the effectiveness of multi multi-channel autoregressive pre-training.
1.0
TUAB
1.0
1.0
SEED
wi pro-irening
w/pre-tmiiiiing
w/ pre-tmining
pee-tritinng
pee-tmining
pro-trining
0.0
Selanced booureY
AUROC
0.6
Balaaeed
Cobes's
Weighied
0.5
Balonedd
Cobeai
Weighted
HMC
Worklead
TUSL
1.0
1.0
pre-irainigg
vr miiiiig
trining
peo tnisin
pre-traniin
0.8
0.6
0.6
Auaracy
Kapps
Wiighse
9.0
Assuracy
AUC-PR
AUROC."
Acesticy
Kappa
Weightdd
Figure 8: Ablation study on multi-channel autoregressive pre-training.
CONCLUSION
In this paper, we introduce NeuroLM. the first universal multi-task foundation model for EEG sig- nal processing. By integrating EEG signals into a Large Language Model framework, NeuroLM leverages advanced text-aligned neural tokenizer embeddings, large-scale multi-channel autoregres- sive pre-training, and joint multi-task tuning to address the inherent challenges of EEG-bbaed BCI and healthcare tasks. Our extensive experiments across six diverse EEG datasets demonstrate the model's superior performance in multi-task learning and inference. Overall, NeuroLM represents a significant step forward in the field of brain-computer interfaces and healthcare domains, show- casing the great potential of LLMs revolutionize EEG signal processing and multi-tat task learning. We believe that NeuroLM will pave the way for more sophisticated and versatile EEG applications, ultimately enhancing the interaction between humans and machines.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
B. Lu acknowledges the following grants: STI 2030-Major Project++20222200085550 National Natural Science Foundation of China (Grant No. 62376158), Shanghai Municipal Science and Technology Major Project (Grant No. 2021SHZD ZX), Meniieal-Eggineering Interdisciplinary Re- search Foundation of Shanghai Jiao Tong University "Jiao Tong Star" Program (YG2023ZD25, YG20247D225) Shanghai Pilot Program for Basic Research Shanghai Jiao Tong University (No. 21TQ1400203), and Shanghai Jiao Tong University SEIEE-Shanghai EmoRays Technology Co., Ltd Joint Laboratory of Affective Brain-Computer Interfaces.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
Turkey Alotaiby, Saleh Alshebeili, Tariq Alshawi, Ishtiaq Ahmad, and Fathi Abd El-Samie. EEG seizure detection and prediction algorithms: survey. EURASIP Journal on Advances in Signal Processing, 2014:1-21, 2014.
Diego Alvarez-Estevez and Roselyne M Rijsman. Inter-database validation of a deep learning ap- proach for automatic sleep scoring. PloS one, 16(8):e025611, 2021.
Benjamin Blankertz, Guido Dornhege, Matthias Krauledat, Klaus- Müller, and Gabriel Curio. The non-invasive berlin brain computr interface: fast acquisition of effective performance in untrained subjects. Neurolmage, 37(2):539-33 5505000,,,05,,5, 2007.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamnrrs Advances in neural information processing systems, 33:1877-1901, 2020.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong. Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.1682.1 2024.
Paolo Detti, Giampaolo Vatti, and Garazi Zabalo Manrique de Lara. Beg synchronization analysis for seizure prediction: A study on data of noninvasive recordings. Processes, 8(7):866, 2020.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21778 2024.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural net- works. Journal of Machine Learning Research, 17(59):1-35, 2016. URL http: //mmlr.
Dtc/pappers/v//15-225 html
Zhongke Gao, Xinmin Wang, Yuxuan Yang, Chaoxu Mu, Qing Cai, Weidong Dang, and Siyang convolutional neural network for driver fatigue evaluation.
Zuo. BEG-based IEEE Transactions on Neural Networks and Learning Systems, 30(9):2755-2763, 2019.
A. Harati, M. Golmohammadi, S. Lopez. Obeid, and J. Picone. Improved EEG event classification using differential energy. In 2015 IEEE Signal Processing in Medicine and Biology Symposium (SPMB), pp. 1-4, 2015. doi: 10. 10..11///.MB...15..5555.
Robert Jenke, Angelika Peer, and Martin Buss. Feature Extraction and Selection for Emotion Recognition from EEG. IEEE Transactions Affective Computing, 5(3):327-339, 2014. doi: 10.1109/TAFFC.2214223333
Wei-Bang Jiang, Li-Ming Zhao, Ping Guo, and Bao-Liang Lu. Discriminating Surprise and Anger from EEG and Eye Movements with Graph Network. In 2021 IEEE International Conference on Bioinformatiis and Biomedicine (BIBM), pp. 1353-1357, 2021. doi: 10.1109/BIB522115.
2021.966699 96666377.
11

---

Published as a conference paper at ICLR 2025
Wei-Bang Jiang, Xuan-Hao Liu, Wei-Long Zheng, and Bao-Liang Lu. Multimodal Adaptive Emo- tion Transformer with Flexible Modality Inputs on A Novel Dataset with Continuous Labels. In Proceedings of the 31st ACM International Conference on Multimedia, MM '23, pp. 5975-5984, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701085. doi: 10.1145/3581783. 788
URI //tp///oiioo org 114553101111118881111
Wei-Bang Jiang, Li-Ming Zhao, and Bao-Liang Lu. Large brain model for learning generic represen- tations with tremendous EEG data in BCI. In The Twelfth International Conference on Learning Representations, 2024. URL bettss///peerrevvee.eeee//eeee
Jin Jing, Wendong Gc, Shenda Hong, Marta Bento Fernandes, Zhen Lin, Chaoqi Yang, Sungtae An, Aaron Struck, Aline Herlopian, Ioannis Karakis, et al. Development of expert-level classifica- tion of seizures and rhythmic and periodic patterns during eeg interpretation. Neurology, 100(17): el e1750ee1 762, 2023.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Louis Korczowski, Martine Cederhout, Anton Andreev, Grégoire Cattan, Pedro Luiz Coelho Ro- drigues, Violette Gautheret, and Marco Congedo. Brain Invaders calibration-less P300-based BCI with modulation of flash duration Dataset (bi2015a). Research report, GIPSA-lab, July 2019.
URL BtLLhtts::// http://haaascccceeeeaaaa---0-0//
Demetres Kostas, Stephane Aroca-Ouellette and Frank Rudzicz. Bendr: contrastive self-supervised learning task to learn from massive amounts of eeg data. Frontiers in Human Neuroscience, 15:653659, 2021.
using transformers and a
Hongli Li, Man Ding, Ronghua Zhang, and Chunbo Xiu. Motor imagery EEG classification algo- rithm based on CNN-LSTM feature fusion network. Biomedical signal processing and control, 72:103342, 2022.
Rui Li, Le-Dian Liu, and Bao-Liang Lu. Discrimination of Decision Confidence Levels from EEG Signals. In 2021 10th International IEEE/EMBS Conference on Neural Engineering (NER), pp. 946-949, 2021.
doi: 10. 1.1109/NE499283.. 2211449923322211...22 2021.944088..
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In
Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:
7/ /oopereeveew www. mntt/formm? wwt.:m:dddw222mmmFmmmmmmm
Wei Liu, Jie-Lin Qiu, Wei-Long Zheng, and Bao-Liang Lu. Comparing recognition performance and robustness of multimodal deep learning models for multimodal emotion recognition. 1EEE Transactions on Cognitive and Developmental Systems, 2021.
Wei Liu, Wei-Long Zheng, Ziyi Li, Si- Yuan Wu, Lu Gan, and Bao-Liang Lu. Identifying similarities and differences in emotion recognition with eeg and eye movements among chinese, german, and french people. Journal of Neural Engineering, 19(2):026012,, 2022.
Matthew D Luciw, Ewa Jarocka, and Benoni B Edin. Multi-channel EEG recordings during 3,936 grasp and lift trials with varying weight and friction. Scientific Data, 1(1):1-11, 2014.
Shuai Luo, Yu-Ting Lan, Dan Peng, Ziyi Li, Wei-Long Zheng, and Bao-Liang Lu. Multimodal emotion recognition in response to oil paintings. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 4167-4170, 2022. doi: 10. 10.1109//MBC48229..000 28222.99722 1630.
Perrin Margaux, Maby Emmanuel, Daligault Sébastien, Bertrand Olivier, and Mattout Jérémie. Ob- jective and subjective evaluation of online error correction during p300-based spelling. Advances in Staaan-oomput Interaction, 2012:4-4, 2012.
Iyad Obeid and Joseph Picone. The temple university hospital eeg data corpus. Frontiers in neuro- science, 10: 195498, 2016.
12

---

Published as a conference paper at ICLR 2025
Wei Yan Peh, Yuanyuan Yao, and Justin Dauwels. Transformer Convolutional Neural Networks for Automated Artifact Detection in Scalp EEG. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 3599-3602, 2022. doi:
10. 109/00/BC44229 20022992292299999
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 18)):, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Arman Savran, Koray Ciftci, Guillame Chanel, Javier Cruz_Mota, Luong Hong Viet, Bülent Sankur, Lale Akarun, Alice Caplier, and Michele Rombaut. Emotion detection in the loop from brain signals and facial images. In e/NTERFACE "06-SIMILAR NoE Summer Workshop on Multimodal Interfaces, 2006.
Gerwin Schalk, Dennis J McFarland, Thilo Hinterberger, Niels Birbaumer, and Jonathan R Wol- paw. BC12000: a general- purpose brain-computer interface (BCI) system. IEEE Transactions on Biomedical Engineering, 51(6): 1043, 2004.
Yonghao Song, Xueyu Jia. Lie Yang. and Longhan Xie. Transformer-aase pratial-temporal feature learning for EEG decoding. arXiv preprint arXiv:2106.11170, 2021.
Akara Supratak, Hao Dong, Chao Wu, and Yike Guo. DeepSleepNet: A Model for Automatic Sleep Stage Scoring Based on Raw SingleChaamn EEG. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 25(1 11919199989111 10099980-0,,,-000910011000188208 2017. doi: 10.1 10.1199/NNSRE2211..2211111
Yousef Rezaci Tabar and Ugur Halici. A novel deep learning approach for classification of EEG motor imagery signals. Journal of Neural Engineering, 14(1):016033, 2016.
Le-Yan Tao and Bao-Liang Lu. Emotion Recognition under Sleep Deprivation Using Multimodal Residual LSTM Network. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-8, 2020. doi: 10. 109/ICCN486005 2020..2009577
Mastaneh Torkamani
Sumeyra Demir Kanik, Serap Aydin, and Mujdat Cetin. Prediction of reaction time and vigilance variability from spatio-spectral features of resting-state EEG in a long sustained attention task. IEEE Journal of Biomedical and Health Informatics, 24(9):2550-22
2020.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Logan Trujillo. Raw EEG Data. 2020. doi: 10.18833//8/SS22HBB 10. 1..1888/8//522NF
Logan T Trujillo, Candice Stanfield, and Ruben D Vela. The effect of electroencephalogram (EEG) reference choice on iitommttion-thoretii measures of the complexity and integration of EEG signals. Frontiers in Neuroscience, 11:425, 2017.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 911)), 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamnet (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL ffF25555eessSiiieeeeoooooaaaaaee
fi
13

---

Published as a conference paper at ICLR 2025
Eva Weltin, Tameem Ahsan, Vinit Shah, Dawer Jamshed, Meysam Golmohammadi, Iyad Obeid, and Joseph Picone. Electroencephalograpiccc slowing: A primary source of error in automatic seizure detection. In 2017 IEEE Signal Processing in Medicine and Biology Symposium (SPMB), pp. 1-5. IEEE, 2017.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang. Lei Zhao, Xixuan Song, et al. Cogvim: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023.
Chaoqi Yang, M Brandon Westover, and Jimeng Sun. BIOT: Biosignal transformer for cross-data learning the wild. In Thirly-seventh Conference on Neural Information Processing Systems, 2023a. httpt:tss:::peeeeeeeeeeee
net Eorum? wtwprruummd=c2Lyyyyyyeo
Chaoqi Yang, Cao Xiao, M Brandon Westover, Jimeng Sun, et al. Self-supervised electroencephalo gram representation learning for automatic sleep staging: model development and evaluation study. JMIR AI, 2(1):e46769, 2023b.
Ke Yi, Yansen Wang, Kan Ren, and Dongsheng Li. Learning topology-agnostic EEG representations with geometry-awarr modeling. In Shirtt--evenrhh Conference on Neural Information Processing Systems, 2023. URL
Daoze Zhang, Zhizhang Yuan, Yang Yang. Junru Chen, Jingjing Wang, and Yafeng Li. Brant: Foun- dation model for intracranial neural signal. In Thirty-eevntth Conference Neural Information Processing Systems, 2023. /tLtbttt:////
htt. do d=DDk 9vaJyE
W. Zheng. W. Liu, Y. Lu, B. Lu, and A. Cichocki. Emotionmeter: A multimodal framework for recognizing human emotions. IEEE Transactions on Cybernetics, pp. 1-13, 2018. ISSN 2168- 2267. doi: 10. CBB.TTCYB.2118.79997
Wei-Long Zheng and Bao-Liang Lu. Investigating critical frequency bands and channels for EEG- based emotion recognition with deep neural networks. IEEE Transactions on Autonomous Mental Development, 7(3):122-175 2015, doi: 10. 10.L109/AAMDD2211222333
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint DXX1::2304.1092, 2023.
Igor Zyma, Sergii Tukaev, Ivan Seleznov, Ken Kiyono, Anton Popov, Mariia Chemykh, and Oleksii Shpenkov. Fllettrencephalogrrms during mental arithmetic task performance. Data, 4(1):14, 2019.
14

---

Published as a conference paper at ICLR 2025
A
RELATED WORK
Large-scale Pre-training for Neural Signals. With the success of self-supervised learning in com- puter vision and natural language processing, several studies have emerged to learn effective EEG representations. Kostas et al. (2021) first propose BENDER, which adapts contrastive learning to derive compressed representations from massive EEG datasets. MMM (Yi et al. 2023) introduces pre-training framework with multi-dimensional position encoding, multi-level channel hierarchy, and multi-stage pre-training strategy to learn topology-agnose representations. BIOT YYngg et al.) 2023a) tokenizes diverse biosignals into unified segments, enabling cross-data learning despite mis- matched channels, variable lengths, and missing values. Brant (Zhang et al. 2023) pre-trains on a large corpus of private intracranial EEG data, capturing long-term dependencies, spatial correla- tions, and both time and frequency domains. Following BIOT, LaBraM (Jiang et al.) (024) further leverages large-scale 2,500 hours public EEG data, and innovatively introduces a neural tokenizer that encodes continuous EEG signals into discrete codes for masked EEG modeling, thus obtain- ing a considerable improvement. Unfortunately, all these methods require fine-tuning for specific downstream tasks and cannot perform multi-task learning and inference.
Multimodal Large Language Models. Recent years have seen remarkable achievements of LLMs. In light of the complementarity between language and other modalities, Multimodal Large Lan- guage Models have been a rising hotspot. The release of GPT-4 (Achiam et al., (023) shows the extraordinary multimodal understanding and generation abilities, thus leading to a research frenzy over MLLMs. LLaVA (Liu et al. (2023) connects a vision encoder and an LLM, introducing the idea of visual instruction tuning for gurrroe-purpose visual and language understanding. Similarly, Zhu it al. (2023) propose MiniGPT-4, which aligns a frozen visual encoder with a frozen advanced LLM, presenting numerous advanced multi-modal abilities. Different from the above MLLMs, CogVLM [Wang et al.) 2023) bridges the gap between the frozen pre-trained LLM and visual encoder by trainable visual expert in the attention and FFN layers. CCenn et al. (2024) present Intern
closing the capability gap between open-source and proprietary commercial MLLMs by utilizing strong vision encoder, dynamic high-resolution, and high-quality bilingual dataset.
B INSTRUCTION DESIGN
Table 5: Information of instruction design for downstream datasets.
Dataset
Instruction Description
TUAB
SEP]
Question: Is this EEG segment abnormal? Answer: YYes, No} [END]
[SE)]
Question: Which event type does this EEG segment belong to? Options: (A)
spike and slow wave. (B) generalized periodic epileptiform discharge. (C) periodic
TUEV
lateralized epileptiform discharge. (D) eye movement. (E) artifact. (F) background.
Answer: {(A), (B), (C), (D), (E), (F)) [END]
[SE) Question: Which emotion type does this EBG segment belong to? Answer: Possitive, Neutral, Negative} [END]
SEED
SSP] Question: Which sleep type does this EEG segment belong to? Options: (A)
HMC
Wake. (B) NREM-1. (C) NREM-2. (D) NREM-3. (E) REM. Answer: ((A), (B), (C),
(D), (F)) [FND]
Workload
SEP) Question: Is this EEG segment of high workload? Answer: (Yes, No} [END]
SEP] Question: Which type does this EEG segment belong to? Options: (A) background. (B) seizure. (C) slowing. Answer: ((A), (B), (C) [END]
TUSL
C PRE-TRAINING DATASET DESCRIPTION
We utilize multiple EEG datasets with various configurations. The detail information of all the datasets are listed 6 The total time after data cleaning is close to 25,000 hours.
15

---

Published as a conference paper at ICLR 2025
Table 6: Information of datasets used for pre-training.
Rate (Hz)
Time
Dataset
#Channel
Description
(h)
corpus of 26,846 clinical EEG
TUEG Obeid &
250- 1024
17-23
244,00
recordings collected Temple University Hospital.
Picone
2016)
These datasets including SEED-IV (15 subjects), SEED-V V (20 subjects), SEED-GER (8 subjects), and SEED-FRA (8 subjects) response to emotional videos.
SEED Series Zheng
et a.
2018
Liu
62
1000
170.54
ct al.
2022) 2021
BCI Competition IV-1 Blankertz
A motor imagery dataset containing classes of left hand, right hand, foot (+
59
1000
8.21
al.
2007)
idle state) for 7 subjects.
multimodal emotion dataset including 16 subjects. The emotions were elicited through a selected subset of the IAPS dataset.
Emobrain Savran
64
1024
4.94
ct al. 2006
Grasp and Lift
A dataset containing 12 subjects performing grasp-and-lift (GAL) trials.
32
500
11.72
(Luciw et al. 2014)
Inria BCI Margaux
A P300-based spelling dataset including 26 subjects.
56
600
29.98
al. 2012
Motor
motor imagery dataset consisting 109 volunteers performing 2 baseline tasks, motor movement, and motor imagery.
Movement/Imager
64
160
47.3
Schalk et al. 2004
EEG was recorded during reported Information- Informatiin-Integration categorization and reported multidimensional Kule-Based categorization tasks.
Raw EEG Data
64
256
34.35
Trujilo 2020
dataset comprising 22 subjects for a resting task of 8 mins with 4 mins of eyes closed and 4 mins of eyes open.
Resting State Trujillo et al. 2017)
64
256
3.04
Siena Scalp EEG Database (Detti
31
512
30.47
database consisting of patients.
et al. 2020
A dataset including 10 subjects, 2.5 minutes recording in eyes-closed and eyes-open prior 105-minute session of Sustained Attention to Response Task with fixed-sequence and varying ISIs.
SPIS Resting State Torkamani-Azz
64
2048
0.83
et al. 2020
A dataset including 50 subjects playing
Target Versus
Brain Invaders, visual P300 Brain- Prain-Computer Interface using oddball paradigm with adaptive Riemannian
Non- Non-Target
32
512
16
Korczowski et al.
2019
Geometry (no-calibration).
Self-collected EEG
corpus (Jiang al.
A mixed self-collected BEG datasets of more than 140 subjects under various conditions.
2023
2021
Luo
62
1000
342.23
al.
2022
Li al.
202
Tao & Lu
2020
16

---

Published as a conference paper at ICLR 2025
D DETAILED EXPERIMENTAL SETTINGS
D.I HYPERPARAMETER SETTINGS
Table Hyperparameters for neural tokenizer.
Hyperparameters
Values
1ut channels
(1,16,16)
Output channels
{16.16,16)
Temporal Encoder
Kernel size
{15,3,3}
Stride
{8.1,1)
Padding
{7.1,1)
Transformer encoder layers
Transformer decoder layers
3
Hidden size
768
ML.P size
3072
Attention head number
12
Codebook size
8192x 128
EEG Batch size
512
Text Batch size
128
Peak learning rate
5e-5
Minimal learning rate
le-5
Learning rate scheduler
Cosine
Optimizer
AdamW
Adam 3 Weight decay
(0.9,0.99)
1e-4
Total epochs
50
Warmup epochs
5
Data overlap
None
Gradient clipping
None
Table 8: Hyperparameters for autoregressive pre-training.
Hyperparameters
NeuroLM-L NeuroLM--B
NeuroLM-XL
Model size
254M
500M
1696M
Transformer encoder layers
12
24
48
Hidden size
768
1024
1600
MLP size
3072
4096
6400
Attention head number
12
16
25
EEG batch size
480 (B), 512 (L, XL)
Text batch size
32 (B). 64 (L, XL)
Peak learning rate
6e-4
Minimal learning rate
6e-5
Learning rate scheduler
Cosine
Optimizer
AdamW
Adam /
(0.9.0.95)
Weight decay
0.1
Total epochs
20
Warmup epochs
Data overlap
None
Gradient clipping
17

---

Published as a conference paper at ICLR 2025
Table 9: Hyperparameters for instruction tuning.
Hyperparameters
Values
Instruction batch size
512
Text batch size
128
Peak learning rate
5e-4 (B), 5e-5 (L.) 2e-5 (XL.
Minimal learning rate Learning rate scheduler
5e-5 (B), 5e-6 (L., 2e-6 (XL)
Cosine
Optimizer
AdamW
Adam 3 Weight decay
(0.9,0.95)
0.1
Total epochs
(B, L), (XL) 0.1
Warmup ratio
Gradient clipping
1
D.2 METRICS
Considering the class imbalance of most downstream EEG datasets, we use the following metrics for comparison:
Balanced Accuracy: The average of recall (sensitivity) obtained on each class. is partic- ularly useful for evaluating classification performance imbalanced datasets. This metric is particularly useful when evaluating models on imbalanced datasets.
AUC-PR: A performance measurement for binary classification problems. It is the area under the curve plotted with precision (y-axis) against recall (x-axis) for different threshold values.
is the area under the curve plotted with the true positive rate (sensitivity) on the y-axis and the false positive rate (1 specificity) the x-axis for different threshold values. AUROC provides an aggregate measure of performance across all possible classification thresholds, indicating the ability of the model distinguish between classes.
AUROC:
. Cohen's Kappa:
A measure of agreement between categorical variables X and Y, calcu- lated from the observed and expected frequencies on the diagonal of a square contingency table. It is used for multi-class classification.
Weighted F1: The weighted Fl score is the harmonic mean of precision and recall, taking into account the support (the number true instances) each class. The weighted score accounts for class imbalance by giving more importance to classes with higher number of instances.
AUROC and Cohen's Kappa are used as the monitor score for binary classification and multi-class classification, respectively.
D.3 BASELINES
We mainly consider BIOT Yang et al. 2023a) and the state-of-the-art EEG foundation model LaBraM (Jiang al. 2024 as our baseline method, where BIOT generic biosignal learning model pre-trained on multple datasets in supervised way, and LaBraM is pre-trained on 2,500 hours data through masked EEG modeling and has learned generic representations for various EEG signals. Five other supervised methods including SPaRCNet (Jing et al.) 2023), ContraWR (Yang) ctt al. 2023b), CNN-Transformer (Peh et al. 2022), FFCL (Li et al. 2022), and ST-Transformer (Song et al. 2021 1 are also utilized our baselines. As there are multi-task methods available in EEG signal processing yet, these baselines are solely fine-tuned on each downstream dataset and cannot perform multiple tasks. We the default settings for these baselines in the BIOT paper. The batch size is 512 for TUAB, TUEV, SEED, and HMC. As the data size Workload and TUSL particularly small, the batch size of these two datasets set to 32 and 16, respectively.
18

---

Published as a conference paper at ICLR 2025
0415
-1180
2300
(a) TUAB
04767
(b) TUEV
80088 362 R
(c) SEED
01521
401567
What Ooon
tye
thi
REM
REM
BESC
0,0000
HMC
0.993
0.44
ISEP1
Quaclioe
ERS
Aaves
Workload
0.420
-0.225
00042
peer Queitio
wald-
thi
Ppiieet
Anser
0,0000
(f) TUSI.
Figure 9: The attention value on other datasets. The vertical axis denotes the Transformer layers.
19

---

Published as a conference paper at ICLR 2025
E ATTENTION VISUALIZATION
To explore the mechanism of NeuroLM, we visualize the attention scores of the answer parts in the instructions for all Transformer layers, as drawn in Figure Firstly, we observed several commonalities across datasets: For the text part, the attention tends to concentrate more in shallow layers whereas for EEG part, attention gains more in deeper layers. This pattern suggests that NeuroLM primarily processes text questions in the shallow layers and focuses EEG tokens in the dceper layers to generate answers. Interestingly, in the case of multiple-choiie questions, Neu- roLM pays close attention the options (A, B, C, etc.) between the 6th and 9th layers. Analyzing critical EEG channels for different tasks, we find that NeuroLM seems to aggregate information to Cz for most datasets. For HMC, F4 and C3 are crucial, while 02 is less effective in sleep stage classification.
F ANALYSIS OF NEURAL TOKENIZER
F.I ABLATION TEMPORAI-FRREQUERT PREDICTION
Temporal and frequency domains are two pivotal aspects of EEG signals. To investigate the impor- tance of these two domains for different downstream tasks, we study three variants by setting the reconstruction target in neural tokenizer training as only the temporal domain, only the frequency domain, and both temporal and frequency domains (original NcuroLM). Figure shows the com- parison between the three variants. Interestingly. it can be found that the temporal domain plays more crucial role on TUAB, Workload, and TUSL. On the contrary, reconstructing the frequency components obtains better performance on TUEV, SEED, and HMC, indicating that the frequency domain is of great importance for event classification,, emotion recognition, and sleep stage classifi- cation. By combining the two domains, most tasks achieve similar or higher performance, demon- strating the effectiveness of our neural tokenizer that excavates compact EEG representations for language models.
1.0
1,0
frequncy & numperd
Inqquency & tempand
frsquncy & sampuell
lrequeccy
tensorra
lamposal
0.2
0.2
0.2
0.0
Delansed BelanoedAvvtaacyy Avouacy
PU
AUROC
6.0
Baiaacal Accancy
Cnàen's Kappa
Weighted F)
0.0
Belansed Aceurazy
Cehes' Kappa
Weighted FI
3.0
1.0
fequancy & temporaa
Gequetay & teupora
froquancy & lanposi
frequncy
inquazzy Ietoonl
fequucyy
sempona
tangeal
0.2
0.2
0.2
Balanced AMsercyAssurcyyy
Coberkkarg
Winghad F1
6.0
Batrnoed Aceancy
AUKUPR
AUROC
0,0
Bolancod Acomacy
Cobert' Kappo
Weighted F1
Figure 10: Ablation study on reconstructing temporal or frequency domain in neural tokenizer.
F.2 VISUALIZATION OF EEG AND TEXT EMBEDDINGS
To evaluate the effectiveness of EEG-text embedding space alignment, we visualize the embeddings in Figure [I using t-SNE (Van der Maaten & Hinton 2008). The EEG embeddings expand outside the text space without alignment. In this case, we find that the model fails to predict the answers expect in multi-task instruction tuning, i.e., the model will output random words instead of
20

---

Published as a conference paper at ICLR 2025
options like (A), (B), or (c) in choice questions, leading near-zero on downstream tasks. This outcome appears to stem from disordered attention scores, as EEG and text embeddings remain in separate spaces. When training with alignment, the EEG space mostly aligns with text space, resulting in normal prediction in instruction tuning, proving the necessity of EEG-text alignment.
metrics
w/o alignment
w/ alignment
60
60
Text
Text
EEG
EEG
20
20
0-
0
40
40
60
60
Figure 11: Representation visualization of EEG and text by t-SNE. Left: training neural tokenizer
without alignment. Right: training neural tokenizer with alignment.
G ABLATION ON DIFFERENT PRE-TRAINING EPOCHS
We conduct ablation study tuning the pre-trained models from different epochs testify the best pre-training epoch. As shown in Table 11 we use the pre- praitre models 5, 10, 15, and 20 epochs. Bold represents the best results and underline represents the second best results. can be found that pre-training for 20 epochs obtains the most bold and underlined results. The performance of 5 epochs gets its best SEED and HMC while the model from 10 epochs achieves the best result on Workload. Overall, pre-training for more epochs can lead to good performance in different tasks.
Table 10: Results on TUAB and TUEV.
TUAB
TUEV
Pre-trained Epochs
Balanced Acc.
AUC-PR
AUROC
Balanced Acc.
Cohen's 's Kappa
Weighted F1
0.00037
0.6891 10.0040
0.7762:10.0388
0.391410.0670 1..6091
0.6975
10
0.7738-+0.0040
0..095
0.7647+0.0143
0.4693+000115
0.4625+0.0091
0.7353++0.038
15
0.7780
0.6951
0.7735:0.00122
0.45571000277
0.421610.0215
0.7131:10.0101
0.7826
0.6975-0.0081
0.7816+00.0079
+0.0048
0.7153+0.0002
Table 11: Results on SEED and HMC.
SEED
HMC
Pre-trained Epochs
Balanced Acc.
Cohen's Kappa
Weighted F1
Balanced Acc.
Cohen's Kappa
Weighted Fl
1.56411000100
0.3505
0.5679:10.0122
0.69561100136
0.6269-0.0053
0.7118:10.004
0.5583+40.0089
0.3376=0..113
0.589244.0091
0.6763+40.0054
0.706540
15
0..043
0.3264
0.5572:0.0164
0.6543+10.011
00.0113
0.6674-10.0266
20
0.5554+0.0005
0.3893-0.0117
0.5599+00.066
0.67374400000
0.6188-000057
0.7126 4.004440.0034
21

---

Published as a conference paper at ICLR 2025
Table 12: Results Workload and TUSI..
Workload
TUSL
Pre-trained Epochs
Balanced Acc.
AUC-PR
AUROC
Balanced
Cohen's Kappa
Weighted F1
0..035
0.5815=:0.0256 0.0163=0.0160
0.5342-+0.0235
0.2950::.0335
0.5241=100266
10
0.6540-0.0112
0.6123=0.0165
0.6501::.0118
0.39201:0.0600
0.888410.0876
0.5984:20.057
0..00221
0.5426=0.0177
0.5682:10.025
0.59101100629
0.3915:0.0999
0.5868:0.0536
20
0.6172=40.0113
0.5824=0.0080
0.6253=0.0160
0.673410.0436
0.5107=0.0617
0.6743=0.0394
H DISCUSSION
Limitations. NeuroLM represents the first attempt to integrate various EEG downstream tasks into a unified model, achieving promising results across multiple downstream datasets. However, has some limitations: 1) Although NeuroLM can surpass certain single-task baselines, still lags behind fate-of-the-art methods that are end-to-end trained on each downstream dataset. 2) MeuroLM is somewhat sensitive to hyperparameter settings, and may not yield satisfactory results without careful tuning. 3) With limited high-quality EEG-text pairs available, this paper only em- ploys coarse- coarse-grained alignment between EEG and language, i.e., space-wise wise alignment, which can pose challenges for LLMs in extracting useful information from EEG tokens.
Outlook. Reflecting on the outlook part of the LaBraM paper, this work explores the first and third suggested directions. Looking ahead, foresee several potential improvements: 1) Utilizing more advanced LLMs as the base models. While this paper uses GPT-2, a relatively small LLM, and still achieves promising results in the multi-task paradigm, leveraging newer, more advanced open-source LLMs such as LLaMA 3 (Dubey et al., 2024) may significantly enhance NeuroLM'ss multi-task learning capabilities. 2) Adopting the mixture-of-expers approach is another promising direction. Given the modality gap between EEG and language, using modality-specifie experts may improve multimodal learning with LLMs. 3) Developing finer-grained EEG and text alignment methods, such as describing EEG samples with predefined sentences and aligning EEG and text descriptions at the VQ training stage by adding contrastive learning loss, may further enhance performance.
22


Reviews:
[{"review_id": "c2zTdVmlTZ", "reviewer": "Reviewer_4vcv", "summary": "The paper presents NeuroLM, an innovative multi-task foundation model that bridges the gap between electroencephalogram (EEG) signals and language models. By converting EEG signals into discrete tokens aligned with text embeddings, NeuroLM enables Large Language Models (LLMs) to process EEG data effectively. The model employs a neural tokenizer trained through vector-quantized temporal-frequency prediction to encode EEG signals, and integrates these tokens into an LLM using multi-channel autoregressive pre-training. Through multi-task instruction tuning, NeuroLM adapts to various downstream EEG tasks without requiring task-specific fine-tuning. Evaluated on six diverse EEG datasets, NeuroLM demonstrates its potential to unify multiple EEG tasks within a single model.", "questions": "1. It would be highly beneficial for the research community if you could share the code for NeuroLM as supplementary material. \n\n2. The expected benefits of large-scale pretraining are not fully realized on the TUAB dataset, with only minimal improvement observed. I was anticipating that the extensive pretraining would lead to more significant performance gains. Could you provide insights into why the pretraining did not result in better performance on TUAB?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper presents an innovative method that enables handling multiple diverse EEG tasks within a single unified model without the need for task-specific fine-tuning. the authors advance the capability to perform multi-task learning in EEG signal processing, overcoming limitations of traditional models that require separate fine-tuning for each task.\n\n2. The authors conduct thorough experiments on six diverse downstream EEG tasks, including abnormal detection, event type classification, emotion recognition, sleep stage classification, cognitive workload detection, and slowing event classification. This extensive evaluation provides a robust comparison with existing state-of-the-art models, demonstrating the versatility and potential of NeuroLM across different EEG applications.\n\n3. The paper includes detailed ablation studies examining the effectiveness of various components of the proposed model, such as the EEG-text embedding alignment, multi-channel autoregressive pre-training, and the robustness to instruction variations. These analyses validate the contributions of each component, offer insights into the model's performance, and enhance the understanding of how each part contributes to the overall effectiveness.\n\n4. The authors provide clear and detailed descriptions of the model architecture, training procedures, data preprocessing steps, and experimental setups. This transparency facilitates reproducibility and allows other researchers to replicate and build upon the work more easily.", "weaknesses": "Overall the paper has a good contribution and is well structured and clear. Here are some tips for improvements: \n\n1. Even though NeuroLM's performance is currently lower compared to LaBraM on most tasks, the authors could emphasize the benefits brought by the unified instruction tuning approach. Specifically, since NeuroLM is instruction-tuned, it may exhibit better generalization to unseen tasks or prompts, enabling zero-shot or few-shot learning without additional fine-tuning. This generalizability could make NeuroLM more adaptable to new EEG tasks compared to models like LaBraM that require task-specific fine-tuning, thus presenting a significant advantage of their approach.\n\n2. The section on methodology especially the codebook and its origin from VQ-VAE,  and how it functions within NeuroLM, and the stop gradient operator justification in equation (3), could benefit from some explanation.\n\n3. For the EEG-text alignment, while an appendix mentions visualization, a more detailed analysis quantifying the impact of EEG-text alignment on model performance would strengthen the claims.\n\n4. Change the colormap for Figure 1, currently the color for NeuroLM and LaBraM are very similar and hard to distinguish.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QzzJAX2ZtV", "reviewer": "Reviewer_2GzD", "summary": "This paper introduces a multi-task foundation model called NeuroLM, which is integrated into large language models (LLMS) for electroencephalogram (EEG) signal processing. NeuroLM uses a text-aligned neural tokenizer to encode continuous EEG signals into discrete neural labels by learning through vector-quantized temporal-frequency prediction. These labels are subsequently used as input to the LLM through multi-channel autoregressive pre-training, which enables the model to learn a causal representation of EEG signals. Finally, through multi-task instruction tuning, NeuroLM is able to handle multiple downstream tasks, such as anomaly detection, event type classification, and emotion recognition, in a single model.", "questions": "1. The paper mentions that NeuroLM performs well on several EEG tasks, but is there any evaluation of how well the model generalizes across different datasets?\n2. How sensitive is the model to noisy data and how does it perform under varying degrees of EEG signal quality variation?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. NeuroLM demonstrates for the first time the ability of a model to handle multiple EEG tasks simultaneously with instruction tuning.\n2. NeuroLM is evaluated on six different downstream datasets, demonstrating its broad applicability and superior learning capabilities.\n3. NeuroLM reduces the need for repeated training through instruction adjustment, making the model more efficient.", "weaknesses": "1. While NeuroLM performs well in multi-task learning, there are still gaps in its performance on some tasks compared to state-of-the-art models specifically designed and optimized for a single task (e.g., LaBraM).\n2. Model performance fluctuates greatly across different tasks and datasets.\n3. Although the model reduces the need for a large amount of labeled data through pre-training and multi-task learning, its performance and generalization ability still rely on high-quality training data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qnXSAkqg78", "reviewer": "Reviewer_hyNm", "summary": "Introduction of **NeuroLM**: a multi-task foundational model to process EEG signals. It leverages the capabilities of LLMs, employing a unified approach encoding the signals into discrete tokens. It addresses challenges like the usual need for individual task fine-tuning. \nModel is utilizing a text-aligned neural tokenizer trained using an adversarial strategy and multi-channel autoregressive pre-training to enhance the understanding of the signals. Experimental results demonstrate a competitive performance across 6 different tasks.", "questions": "- How robust is NeuroLM across different LLM architectures?\n- What role does adversarial training play compared to alternative strategies?\n- Can you show some examples of fine-tuning that lead to SOTA performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Innovative Idea : build the first EEG multi-task foundation model. \n- Trained on large-scale EEG dataset (> 25K hours), diverse data sources enhance model's generalization ability. \n- Consider EEG signals as foreign language and expand scope of LLMs including EEG tasks\n- Instruction Tuning for Multi-Tasking: joint multi-task instruction handle various tasks in a unified manner.", "weaknesses": "- Absence of Fine-Tuning Examples: The authors position NeuroLM as a foundation model and present solid performance across six downstream tasks. However, the study does not include an exploration of fine-tuning capabilities to demonstrate NeuroLM's potential for achieving SOTA performance which could underscore its utility in practical applications.\n\n- Dependency on GPT-2: The authors only test GPT-2 so this study limits insights into the broader potential of this innovative approach. The model aims to bridge EEG and language so evaluating additional language models is necessary for a more comprehensive understanding of the model robustness and adaptability. \n\n- EEG-Text Alignment: The use of adversarial training is innovative. The absence of ablation studies limits understanding of its specific impact. It could enhance paper's clarity and support the results if it was shown how this approach contributes to the model's performance. \n\n- Complexity and Reproducibility: The pipeline's complexity may pose challenges for reproducibility. Enhanced clarity in certain sections would better support the community in replicating and extending this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZoIEuhK0qX", "reviewer": "Reviewer_gQYU", "summary": "The paper introduces NeuroLM, a universal multi-task foundation model for EEG signal processing that consists of three key stages: a text-aligned neural tokenizer, multi-channel autoregressive pre-training, and multi-task instruction tuning. Traditional EEG pre-trained models often require full fine-tuning for each downstream task, resulting in resource inefficiencies. NeuroLM addresses this issue by leveraging Large Language Models (LLMs) and treating EEG signals as a foreign language, demonstrating the potential of this multi-task learning paradigm across multiple EEG-related tasks.", "questions": "Concern 1:\nThe paper introduces an EEG-Text Embedding Space Alignment module in Figure 2, built on LaBraM’s neural tokenizer. However, I have concerns regarding the effectiveness and explanation of this alignment.\n\nFundamentally, EEG and Text have no inherent connection, especially since the EEG signals are not language-evoked. The use of a single gradient reverse layer may only distinguish EEG from text domains rather than fully align their embeddings. To clarify the module's effectiveness, could the authors:\n\n1) Provide visualizations, such as t-SNE plots, comparing EEG and Text embedding spaces before and after alignment to illustrate the degree of alignment achieved.\n2) Clarify the source of the Text embeddings and their generation method. The authors do not explain where these embeddings come from or how they are generated, leaving an important gap in understanding.\n3) Include an ablation study to quantify the impact of the EEG-Text Embedding Space Alignment module on downstream task performance, showing its necessity and contribution to the overall model.\n\nConcern 2:\nThe Theory Analysis section in 2.2 of Multi-channel Autoregressive Pre-training provides a high-level explanation by drawing an analogy to VAE, optimizing for reconstruction loss and KL divergence, but it lacks a detailed explanation of how causal relationships between EEG channels are learned or how this approach is theoretically grounded in the autoregressive framework.\n\nTo enhance the theoretical foundation of the multi-channel autoregressive approach, could the authors:\n\nInclude mathematical formulations demonstrating how this approach models dependencies between EEG channels and captures causal relationships.\n\nConcern 3:\nThe paper lacks a detailed discussion regarding instruction design and how variations in prompt formulations affect downstream task performance. While instruction tuning addresses multiple tasks, the authors do not examine how different prompts might affect task performance. Could the authors provide an analysis of how different prompts influence task performance?\n\nConcern 4:\nThe paper does not sufficiently discuss the impact of data curriculum or input order on the results. Although instruction tuning is introduced, there is no exploration of whether sequencing the data in a specific way (e.g., starting with simpler tasks or data) influences model performance across the various downstream tasks. It would be helpful if the authors could provide an analysis of how the ordering of data inputs affects model learning and task outcomes.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The approach of treating EEG signals as a form of language and integrating them with LLMs is novel and creative, offering a new direction for EEG signal processing.\n\n- NeuroLM shows potential in unifying different EEG tasks within a single model, reducing the need for task-specific fine-tuning and streamlining BCI applications.\n\n- The experiments cover six diverse datasets, demonstrating the model's versatility and robustness in multi-task settings.", "weaknesses": "My concerns mainly focus on four areas, which are detailed in the Questions section. These include the effectiveness of EEG-Text Embedding Space Alignment, the theoretical foundation of multi-channel autoregressive pre-training, the lack of discussion on instruction design's impact on downstream tasks, and the absence of an analysis on data curriculum or input order. I hope the authors can address these concerns in the rebuttal stage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)