NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals

Authors: Weibang Jiang, Yansen Wang, Bao-liang Lu, Dongsheng Li
Primary area: applications to neuroscience & cognitive science
Keywords: EEG, large language model, multi-task learning

Abstract:
Recent advancements for large-scale pre-training with neural signals such as electroencephalogram (EEG) have shown promising results, significantly boosting the development of brain-computer interfaces (BCIs) and healthcare. However, these pre-trained models often require full fine-tuning on each downstream task to achieve substantial improvements, limiting their versatility and usability, and leading to considerable resource wastage. To tackle these challenges, we propose NeuroLM, the first multi-task foundation model that leverages the capabilities of Large Language Models (LLMs) by regarding EEG signals as a foreign language, endowing the model with multi-task learning and inference capabilities. Our approach begins with learning a text-aligned neural tokenizer through vector-quantized temporal-frequency prediction, which encodes EEG signals into discrete neural tokens. These EEG tokens, generated by the frozen vector-quantized (VQ) encoder, are then fed into an LLM that learns causal EEG information via multi-channel autoregression. Consequently, NeuroLM can understand both EEG and language modalities. Finally, multi-task instruction tuning adapts NeuroLM to various downstream tasks. We are the first to demonstrate that, by specific incorporation with LLMs, NeuroLM unifies diverse EEG tasks within a single model through instruction tuning. The largest variant NeuroLM-XL has record-breaking 1.7B parameters for EEG signal processing, and is pre-trained on a large-scale corpus comprising approximately 25,000-hour EEG data. When evaluated on six diverse downstream datasets, NeuroLM showcases the huge potential of this multi-task learning paradigm.

Decision: accept (poster)
Reviewer scores: 8, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1729826314336

thread_type:
official_review

<thread_object>
creator: reviewer_gqyu

<content>
summary:
The paper introduces NeuroLM, a universal multi-task foundation model for EEG signal processing that consists of three key stages: a text-aligned neural tokenizer, multi-channel autoregressive pre-training, and multi-task instruction tuning. Traditional EEG pre-trained models often require full fine-tuning for each downstream task, resulting in resource inefficiencies. NeuroLM addresses this issue by leveraging Large Language Models (LLMs) and treating EEG signals as a foreign language, demonstrating the potential of this multi-task learning paradigm across multiple EEG-related tasks.

strengths:
- The approach of treating EEG signals as a form of language and integrating them with LLMs is novel and creative, offering a new direction for EEG signal processing.

- NeuroLM shows potential in unifying different EEG tasks within a single model, reducing the need for task-specific fine-tuning and streamlining BCI applications.

- The experiments cover six diverse datasets, demonstrating the model's versatility and robustness in multi-task settings.

weaknesses:
My concerns mainly focus on four areas, which are detailed in the Questions section. These include the effectiveness of EEG-Text Embedding Space Alignment, the theoretical foundation of multi-channel autoregressive pre-training, the lack of discussion on instruction design's impact on downstream tasks, and the absence of an analysis on data curriculum or input order. I hope the authors can address these concerns in the rebuttal stage.

questions:
Concern 1:
The paper introduces an EEG-Text Embedding Space Alignment module in Figure 2, built on LaBraM’s neural tokenizer. However, I have concerns regarding the effectiveness and explanation of this alignment.

Fundamentally, EEG and Text have no inherent connection, especially since the EEG signals are not language-evoked. The use of a single gradient reverse layer may only distinguish EEG from text domains rather than fully align their embeddings. To clarify the module's effectiveness, could the authors:

1) Provide visualizations, such as t-SNE plots, comparing EEG and Text embedding spaces before and after alignment to illustrate the degree of alignment achieved.
2) Clarify the source of the Text embeddings and their generation method. The authors do not explain where these embeddings come from or how they are generated, leaving an important gap in understanding.
3) Include an ablation study to quantify the impact of the EEG-Text Embedding Space Alignment module on downstream task performance, showing its necessity and contribution to the overall model.

Concern 2:
The Theory Analysis section in 2.2 of Multi-channel Autoregressive Pre-training provides a high-level explanation by drawing an analogy to VAE, optimizing for reconstruction loss and KL divergence, but it lacks a detailed explanation of how causal relationships between EEG channels are learned or how this approach is theoretically grounded in the autoregressive framework.

To enhance the theoretical foundation of the multi-channel autoregressive approach, could the authors:

Include mathematical formulations demonstrating how this approach models dependencies between EEG channels and captures causal relationships.

Concern 3:
The paper lacks a detailed discussion regarding instruction design and how variations in prompt formulations affect downstream task performance. While instruction tuning addresses multiple tasks, the authors do not examine how different prompts might affect task performance. Could the authors provide an analysis of how different prompts influence task performance?

Concern 4:
The paper does not sufficiently discuss the impact of data curriculum or input order on the results. Although instruction tuning is introduced, there is no exploration of whether sequencing the data in a specific way (e.g., starting with simpler tasks or data) influences model performance across the various downstream tasks. It would be helpful if the authors could provide an analysis of how the ordering of data inputs affects model learning and task outcomes.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729872854935

thread_type:
official_review

<thread_object>
creator: reviewer_hynm

<content>
summary:
Introduction of **NeuroLM**: a multi-task foundational model to process EEG signals. It leverages the capabilities of LLMs, employing a unified approach encoding the signals into discrete tokens. It addresses challenges like the usual need for individual task fine-tuning. 
Model is utilizing a text-aligned neural tokenizer trained using an adversarial strategy and multi-channel autoregressive pre-training to enhance the understanding of the signals. Experimental results demonstrate a competitive performance across 6 different tasks.

strengths:
- Innovative Idea : build the first EEG multi-task foundation model. 
- Trained on large-scale EEG dataset (> 25K hours), diverse data sources enhance model's generalization ability. 
- Consider EEG signals as foreign language and expand scope of LLMs including EEG tasks
- Instruction Tuning for Multi-Tasking: joint multi-task instruction handle various tasks in a unified manner.

weaknesses:
- Absence of Fine-Tuning Examples: The authors position NeuroLM as a foundation model and present solid performance across six downstream tasks. However, the study does not include an exploration of fine-tuning capabilities to demonstrate NeuroLM's potential for achieving SOTA performance which could underscore its utility in practical applications.

- Dependency on GPT-2: The authors only test GPT-2 so this study limits insights into the broader potential of this innovative approach. The model aims to bridge EEG and language so evaluating additional language models is necessary for a more comprehensive understanding of the model robustness and adaptability. 

- EEG-Text Alignment: The use of adversarial training is innovative. The absence of ablation studies limits understanding of its specific impact. It could enhance paper's clarity and support the results if it was shown how this approach contributes to the model's performance. 

- Complexity and Reproducibility: The pipeline's complexity may pose challenges for reproducibility. Enhanced clarity in certain sections would better support the community in replicating and extending this work.

questions:
- How robust is NeuroLM across different LLM architectures?
- What role does adversarial training play compared to alternative strategies?
- Can you show some examples of fine-tuning that lead to SOTA performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650841982

thread_type:
official_review

<thread_object>
creator: reviewer_2gzd

<content>
summary:
This paper introduces a multi-task foundation model called NeuroLM, which is integrated into large language models (LLMS) for electroencephalogram (EEG) signal processing. NeuroLM uses a text-aligned neural tokenizer to encode continuous EEG signals into discrete neural labels by learning through vector-quantized temporal-frequency prediction. These labels are subsequently used as input to the LLM through multi-channel autoregressive pre-training, which enables the model to learn a causal representation of EEG signals. Finally, through multi-task instruction tuning, NeuroLM is able to handle multiple downstream tasks, such as anomaly detection, event type classification, and emotion recognition, in a single model.

strengths:
1. NeuroLM demonstrates for the first time the ability of a model to handle multiple EEG tasks simultaneously with instruction tuning.
2. NeuroLM is evaluated on six different downstream datasets, demonstrating its broad applicability and superior learning capabilities.
3. NeuroLM reduces the need for repeated training through instruction adjustment, making the model more efficient.

weaknesses:
1. While NeuroLM performs well in multi-task learning, there are still gaps in its performance on some tasks compared to state-of-the-art models specifically designed and optimized for a single task (e.g., LaBraM).
2. Model performance fluctuates greatly across different tasks and datasets.
3. Although the model reduces the need for a large amount of labeled data through pre-training and multi-task learning, its performance and generalization ability still rely on high-quality training data.

questions:
1. The paper mentions that NeuroLM performs well on several EEG tasks, but is there any evaluation of how well the model generalizes across different datasets?
2. How sensitive is the model to noisy data and how does it perform under varying degrees of EEG signal quality variation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730678908448

thread_type:
official_review

<thread_object>
creator: reviewer_4vcv

<content>
summary:
The paper presents NeuroLM, an innovative multi-task foundation model that bridges the gap between electroencephalogram (EEG) signals and language models. By converting EEG signals into discrete tokens aligned with text embeddings, NeuroLM enables Large Language Models (LLMs) to process EEG data effectively. The model employs a neural tokenizer trained through vector-quantized temporal-frequency prediction to encode EEG signals, and integrates these tokens into an LLM using multi-channel autoregressive pre-training. Through multi-task instruction tuning, NeuroLM adapts to various downstream EEG tasks without requiring task-specific fine-tuning. Evaluated on six diverse EEG datasets, NeuroLM demonstrates its potential to unify multiple EEG tasks within a single model.

strengths:
1. The paper presents an innovative method that enables handling multiple diverse EEG tasks within a single unified model without the need for task-specific fine-tuning. the authors advance the capability to perform multi-task learning in EEG signal processing, overcoming limitations of traditional models that require separate fine-tuning for each task.

2. The authors conduct thorough experiments on six diverse downstream EEG tasks, including abnormal detection, event type classification, emotion recognition, sleep stage classification, cognitive workload detection, and slowing event classification. This extensive evaluation provides a robust comparison with existing state-of-the-art models, demonstrating the versatility and potential of NeuroLM across different EEG applications.

3. The paper includes detailed ablation studies examining the effectiveness of various components of the proposed model, such as the EEG-text embedding alignment, multi-channel autoregressive pre-training, and the robustness to instruction variations. These analyses validate the contributions of each component, offer insights into the model's performance, and enhance the understanding of how each part contributes to the overall effectiveness.

4. The authors provide clear and detailed descriptions of the model architecture, training procedures, data preprocessing steps, and experimental setups. This transparency facilitates reproducibility and allows other researchers to replicate and build upon the work more easily.

weaknesses:
Overall the paper has a good contribution and is well structured and clear. Here are some tips for improvements: 

1. Even though NeuroLM's performance is currently lower compared to LaBraM on most tasks, the authors could emphasize the benefits brought by the unified instruction tuning approach. Specifically, since NeuroLM is instruction-tuned, it may exhibit better generalization to unseen tasks or prompts, enabling zero-shot or few-shot learning without additional fine-tuning. This generalizability could make NeuroLM more adaptable to new EEG tasks compared to models like LaBraM that require task-specific fine-tuning, thus presenting a significant advantage of their approach.

2. The section on methodology especially the codebook and its origin from VQ-VAE,  and how it functions within NeuroLM, and the stop gradient operator justification in equation (3), could benefit from some explanation.

3. For the EEG-text alignment, while an appendix mentions visualization, a more detailed analysis quantifying the impact of EEG-text alignment on model performance would strengthen the claims.

4. Change the colormap for Figure 1, currently the color for NeuroLM and LaBraM are very similar and hard to distinguish.

questions:
1. It would be highly beneficial for the research community if you could share the code for NeuroLM as supplementary material. 

2. The expected benefits of large-scale pretraining are not fully realized on the TUAB dataset, with only minimal improvement observed. I was anticipating that the extensive pretraining would lead to more significant performance gains. Could you provide insights into why the pretraining did not result in better performance on TUAB?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731985095194

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global response

comment:
We would like to express our sincere gratitude to all reviewers for their insightful and constructive feedback on our work. 

### Focus and Key Contribution
Our goal is to **lay the foundation for a new multi-task paradigm in EEG processing using LLMs**, demonstrating that a unified instruction-tuning approach can effectively handle diverse EEG tasks. We're glad to find that all reviewers recognized NeuroLM’s unique contribution in this regard, particularly its ability to generalize across multiple EEG tasks within a single model. This approach enables NeuroLM to generalize across a range of tasks and we are delighted to see it highlighted by reviewer 4vcv as a promising direction for expanding the application of LLMs in EEG-based research.

Moreover, we appreciate that reviewer hyNm and gQYU pointed out some possible directions for improvement and reviewer 2GzD and hyNm provided comments on concrete targets to achieve this whole branch of work. We envision NeuroLM as a stepping stone for more advanced EEG-Language integration models, and we hope that this work will inspire future research to refine and expand on this framework. We are excited about the potential for more sophisticated alignment techniques, multi-task learning frameworks, and robust training strategies to build on NeuroLM’s foundation, contributing to broader applications of LLMs in EEG and neuroscience.

### Common Concerns and Clarifications
- **Performance of NeuroLM** Several reviewers noted areas for potential improvement and clarification. We acknowledge that NeuroLM does not achieve state-of-the-art (SOTA) performance on every task, particularly when compared to specialized, single-task models like LaBraM. We also want to emphasize that, unlike many so-called multi-task models that need specific components for each task, **NeuroLM is a true multi-task model**, meaning that the same trained model executes multiple tasks simultaneously without any additional module. This inherently introduces a trade-off problem between the performance of different tasks, as the model must generalize across diverse datasets and objectives without task-specific fine-tuning. Despite this trade-off problem in unified models, NeuroLM achieves competitive performance on multiple tasks, demonstrating the effectiveness of this paradigm. Besides, the choice of multi-channel autoregressive pre-training was a necessary compromise to ensure compatibility with the causal structure of current LLM for this study. It is well known that masked signal modeling can be a more efficient representation learning approach than next-token prediction, which contributes to LaBraM’s stronger performance. In contrast, next-token prediction appears to be better for generation and has a better scaling property. Given the causal nature of current LLMs, we aimed to find a feasible pre-training strategy that works within these constraints, with autoregressive modeling being the most compatible option for EEG tokens.
We believe that incorporating contrastive learning is a promising direction for improving EEG-text alignment. As noted in Appendix H, contrastive learning could help align EEG signals with text descriptions within the VQ encoder, inspired by frameworks like CLIP. This alignment would allow for more nuanced EEG-text integration and potentially yield significant performance gains. Introducing contrastive learning for pre-training, while promising, would add considerable complexity to the current study, and we see this as an exciting avenue for future work.
- **Ablation on EEG-text alignment** Reviewers also requested further clarification on the impact of EEG-text alignment. To assess this, we conducted ablation studies by removing the alignment loss, and the results showed that training fails entirely without it. Specifically, without alignment, the model struggled to produce meaningful predictions and outputted random codes instead of expected responses like (A), (B), or \(C), leading to near-zero scores across most metrics. This misalignment likely results from disordered attention scores, as EEG and text embeddings remain unaligned in separate spaces. We have detailed these findings in Appendix F.2, which you may have missed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732327312871

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Inquiry for post-rebuttal comments

comment:
Dear Reviewers:

Thank you again for your wisdom and valuable comments. We have provided complete explanations for all the questions. Since the rebuttal process is approaching its end, we would be glad to hear from you whether our rebuttal has addressed your concerns. Feel free to comment on our rebuttal if you have further questions or considerations.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents an innovative method for predicting on diverse EEG tasks within a single unified model without the need for task-specific fine-tuning.

Pros:
- The approach is considered original and novel
- Models are trained on a very large EEG corpus
- Results are provided on diverse set of tasks
- Paper provides an ablation study

Minor concerns:
- Code is not shared and given the complexity of the pipeline reproducibility of the results would certainly be a big challenge for the field. Yet authors promise to release code and weights upon acceptance. Please make sure that the code allows full replication of the results.
- Remaining experiments (impact of LLM, instructions etc.) should be considered but realistically in future works.

As the AC, I would strongly encourage this community to collaborate in order to facilitate benchmarks and comparisons of methods on standard tasks with minimal code changes.