Title: Multi-dataset Joint Pre-training of Emotional EEG Enables Generalizable Affective Computing
Venue: NeurIPS 2025
Accepted: yes
Keywords: EEG

pre-training

affective computing

emotion
Abstract: Task-specific pre-training is essential when task representations diverge from generic pre-training features. Existing task-general pre-training EEG models struggle with complex tasks like emotion recognition due to mismatches between task-specific features and broad pre-training approaches. This work aims to develop a task-specific multi-dataset joint pre-training framework for cross-dataset emotion recognition, tackling problems of large inter-dataset distribution shifts, inconsistent emotion category definitions, and substantial inter-subject variability. We introduce a cross-dataset covariance alignment loss to align second-order statistical properties across datasets, enabling robust generalization without the need for extensive labels or per-subject calibration. To capture the long-term dependency and complex dynamics of EEG, we propose a hybrid encoder combining a Mamba-like linear attention channel encoder and a spatiotemporal dynamics model. Our method outperforms state-of-the-art large-scale EEG models by an average of 4.57% in AUROC for few-shot emotion recognition and 11.92% in accuracy for zero-shot generalization to a new dataset. Performance scales with the increase of datasets used in pre-training. Multi-dataset joint pre-training achieves a performance gain of 8.55\% over single-dataset training. This work provides a scalable framework for task-specific pre-training and highlights its benefit in generalizable affective computing. Our code is available at https://github.com/ncclab-sustech/mdJPT_nips2025.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a framework called mdJPT for EEG-based emotion recognition that can pre-train on multiple datasets by aligning second-order statistical properties. The framework contains three key components: (1) mamba-like linear attention based EEG patchifier and spatialtemporal dynamics model, (2) cross-dataset alignment loss, (3) inter-subject alignment loss. This papers' design can be seperated into two parts: (1) EEG encoder, which has MLLA patchifier and spatialtemporal modeling, (2) domain/representation alignemtn, which has CDA and ISA. The framework is compared with three EEG foundation models on 6 datasets. Notably, it is not a traditional self-supervised framework, but requires data labeled with trials ID, each trial means a single video stimulus.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n\n1. Training on multiple datasets for a specific task is a promising approach in the EEG field, I appreciate this idea.  \n\n2. Validating the method on 6 datasets is a hard work, and I appreciate your effort in trying various combination of pre-training datasets in Fig 3.  \n\n3. Opensourcing the code is very good in EEG-based emotion recognition.\n\n## Weaknesses\n1. The problem you are trying to solve is absolutely meaningful, however, your method seems to be less motivated. You used many techniques in mdJPT, but what's the rational behind these designs? For instance, why do you use mamba-like linear attention, just because it's better than the vanilla Transformer in Table 4? Aligning the covariance of two domain is also not a new idea, but why do you use it, just because a previous work [16]? Have you tried other domain alignment strategies, like mmd loss, or other statistical properties? Please note that I am not asking for more ablation study, but explanation behind all designs. Acutally, I think there is no need to design the EEG encoder for a framework paper, e.g., adding an MLLA encoder is somewhat abrupt and distracts the research focus of the article.\n\n2. The baseline models are self-supervised framework which requires no label of data, while mdJPT requires more information about which trial does each EEG segment belong to. Although I appreciate the design to use trial information as it is a task-specific design, it brings some inconvenience and unfairness when comparing with baselines. Thus, I think it is better to discuss more about this design, rather than MLLA encoder, like comparing with more variants with mdJPT, or with some adpated baselines that can use trial information for pre-training.\n\n3. The design of the data selection for each training batch is interesting, but you didn't discuss about it. I think this part is the most essential for ISA, as the contrastive learning is very relevant to the training batch. So it might be more appropriate to move the part of how to select a training batch (line 118-121) to section 2.4. And it is important to discuss the design of training batch selection with more experiments.\n\n4. The types of emotions studied in different datasets are different but overlap, like SEED's all emotion types are contained in SEED-IV to SEED-VII, but some datasets like DEAP, which uses valence-arousal model to define emotion, is quite different from SEED series, which uses the discrete emotion model. How to transfer the knowledge of DEAP to SEED is an interesting problem too. But in Figure 3, this problem is overlooked. Using trial ID is an interesting approach, but in DEAP, different subjects may give different valence / arousal value to the same stimulus, how did you tackle this question? This is very important, because the design of mdJPT seems to assume each subject will have the same emotion when viewing the same video.\n\nIn conclusion, while I appreciate the framework, the vague motivation behinds the design and experiments that neglect key components make this version is not ready for publication. I hope you can refine it for the next conference / journal.\n\nQUESTIONS:\n1. In Figure 2. C, the color of these arrows are somewhat close and difficult to differenciate.\n\nTypo:\n1. line 48: there is an 1 outside the brackets.\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a task-specific multi-dataset pre-training framework named mdJPT for EEG-based emotion recognition. To address challenges such as distribution shifts and label inconsistencies across datasets, the authors propose a Cross-Dataset Covariance Alignment (CDA) loss, which aligns data distributions without requiring extensive labels. In addition, they design a hybrid spatiotemporal encoder to enhance EEG signal modeling. The proposed framework improves accuracy by 6.59% over previous approaches and demonstrates strong generalization across datasets.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The paper is technically sound and clearly presents the mdJPT framework, along with key contributions including the CDA loss, a spatiotemporal encoder, and the multi-dataset pre-training strategy.\n\n- The authors conduct thorough evaluations across multiple public EEG datasets, using cross-validation and comparison with competitive baselines. Ablation studies are also included to analyze the contributions of the ISA Loss, CDA Loss, and MLLA encoder.\n\n- Code is made publicly available through an anonymous repository, which supports the reproducibility of the work.\n\n\n### Weaknesses\n\n- The mdJPT framework includes several components (e.g., MLLA encoder, spatiotemporal dynamics model, CDA loss), which may lead to increased model complexity, making model implementation and deployment more challenging.\n\n- Although the framework is designed to improve generalization, it still requires a small number of subjects from the target dataset for fine-tuning, limiting its potential in zero-shot settings.\n\n- The paper does not fully utilize the 9-page limit. There is room to provide more in-depth explanations or case studies, especially illustrating how the model generalizes across different datasets, which would strengthen the overall contribution.\n\n- The Related Work section omits several relevant EEG and physiological signal foundation models. Important works such as Brant-2 [1], Brant-X [2], and BrainBERT [3] are highly relevant and should be discussed.\n\n> [1] Brant-2: Foundation model for brain signals. arXiv:2402.10251\n> \n> [2] Brant-X: A Unified Physiological Signal Alignment Framework. KDD 2024\n> \n> [3] BrainBERT: Self-supervised representation learning for intracranial recordings. arXiv:2302.14367\n\nQUESTIONS:\n1. The paper mentions that fine-tuning with a few target-domain subjects is needed. Has the model been evaluated in zero-shot settings? If so, what is the performance gap, and what might this suggest about potential bottlenecks in the current method?\n\n2. Why is valence chosen as the prediction target for DEAP, rather than arousal, dominance, or a combination? Have any experiments been conducted on the other dimensions?\n\n3. The authors state that training is conducted for 20 epochs. Could this lead to overfitting, especially with limited labeled data? No training or validation loss curves are provided—could these be added?\n\n4. While mdJPT performs well on SEED, SEED-V, and SEED-VII, its performance drops noticeably on SEEG-IV. What factors might explain this discrepancy? How is SEEG-IV different from the other datasets?\n\n5. Besides the CDA loss weight λ, are there other important hyperparameters in the model? How were these values selected? Was a hyperparameter search conducted?\n\n6. The authors are encouraged to make full use of the 9-page limit to elaborate on their methodology and findings. In particular, presenting concrete case studies or examples would enhance the reader’s understanding of how the model performs across varied datasets.\n\n7. The Related Work section would benefit from incorporating recent literature on EEG foundation models and physiological signal learning, such as Brant-2, Brant-X, and BrainBERT.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel multi-dataset joint pre-training framework (mdJPT) for EEG-based emotion recognition, incorporating a Cross-Dataset covariance Alignment (CDA) loss and a hybrid spatiotemporal encoder. Despite demonstrating a 6.59% average F1-score improvement over existing EEG foundation models in cross-dataset generalization, several limitations need to be addressed to enhance the impact and reproducibility of this work.\n\n1. The claim of \"zero-shot generalization to new emotion categories\" in Section 2.3 is not supported by the experimental results. The framework requires fine-tuning with 25% of target subjects, as described in Section 3.3, which contradicts the definition of true zero-shot learning. This limitation significantly undermines the applicability of the framework in scenarios where labeled data is scarce.\n\n2. Residual feature distribution differences persist despite the application of CDA, as noted in Section 4. The approach struggles to handle extreme heterogeneity, such as the difference between FACED’s 9-class emotions and SEED’s 3 classes. This is evidenced by the relatively low F1-score of 20.32% on FACED, as shown in Table 3. The reliance of CDA solely on covariance may overlook higher-order distribution shifts.\n\n3. The tuning of the CDA weight is critical but inadequately explained in Section 3.5. Performance varies non-monotonically, as observed in Table 5, where the F1-score decreases from 65.02% to 64.08% when the weight increases from 0.02 to 0.1. This lack of clear guidelines for selecting the weight parameter λ in Eq. 8 poses a risk to the reproducibility of the results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The CDA loss effectively mitigates inter-dataset distribution shifts by aligning second-order statistical properties without requiring labels or subject-specific calibration (Section 2.3). This addresses a critical gap in EEG emotion recognition.\n\n2.The hybrid encoder (Mamba-like linear attention + spatiotemporal dynamics) successfully captures long-term dependencies and complex EEG dynamics, outperforming vanilla transformers by >2% F1-score (Table 4).\n\nWeaknesses:\n1. The claim of \"zero-shot generalization to new emotion categories\" in Section 2.3 is not supported by the experimental results. The framework requires fine-tuning with 25% of target subjects, as described in Section 3.3, which contradicts the definition of true zero-shot learning. This limitation significantly undermines the applicability of the framework in scenarios where labeled data is scarce.\n\n2. Residual feature distribution differences persist despite the application of CDA, as noted in Section 4. The approach struggles to handle extreme heterogeneity, such as the difference between FACED’s 9-class emotions and SEED’s 3 classes. This is evidenced by the relatively low F1-score of 20.32% on FACED, as shown in Table 3. The reliance of CDA solely on covariance may overlook higher-order distribution shifts.\n\n3. The tuning of the CDA weight is critical but inadequately explained in Section 3.5. Performance varies non-monotonically, as observed in Table 5, where the F1-score decreases from 65.02% to 64.08% when the weight increases from 0.02 to 0.1. This lack of clear guidelines for selecting the weight parameter λ in Eq. 8 poses a risk to the reproducibility of the results.\n\nQUESTIONS:\n1. To evaluate the model's generalization ability, it is necessary to test zero-shot transfer learning by keeping the encoder frozen and directly assessing performance on unseen datasets. Additionally, previous claims about generalization should be reformulated to reflect the current dependency on fine-tuning, ensuring that conclusions align with reality.\n\n2. Efforts should be made to enhance distribution alignment. This can be achieved by expanding the current CDA (Contrastive Distributional Alignment) to include higher-order statistics such as maximum mean discrepancy and incorporating domain-adversarial training strategies. To verify the effectiveness of feature distribution alignment, visualization techniques like t-SNE plots should be used to display the feature distributions before and after alignment.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose mdJPT (Multi‑dataset Joint Pre‑Training), a task‑specific framework for EEG‑based emotion recognition. The core Idea is to jointly pre‑train a hybrid EEG encoder on several emotion datasets, while aligning second‑order statistics (Cross‑Dataset Alignment, CDA) and inter‑subject representations (Inter‑Subject Alignment, ISA). The architecture employs a hybrid encoder combining a lightweight Mamba-style linear attention module and a spatiotemporal dynamics block. Evaluation across six public EEG datasets using a leave-one-dataset-out protocol shows that mdJPT significantly outperforms both traditional models and recent EEG foundation models, improving average F1-scores by 6.6 percentage points.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1) Solves the problem of data scarcity by focusing on the pre-tarining with multiple datasets. Moreover, it combines task‑specific multi‑dataset pre‑training with covariance‑centric alignment; prior EEG foundation models focus on task‑agnostic breadth. \n2) The paper addresses a key challenge in EEG-based emotion recognition: poor cross-dataset generalisation. By pretraining across multiple datasets with targeted alignment strategies, mdJPT offers a meaningful step forward for real-world deployment also since combining dataset adds to data diversity in training.\n3) The integration of second-order statistical alignment (CDA) and subject-level contrastive learning (ISA) is interesting.\n4) The paper is clearly written, with well-structured sections.\n\nWeakness:\n1) The assumption that EEG emotion datasets are misaligned is taken from prior knowledge, but the paper does not directly test or visualise this. I think including visual evidence would make the motivation for their alignment techniques much more compelling.\n2) All datasets are video-evoked emotion EEG data collected in lab-settings, no dataset from other elicitation technique is used like, semi-naturalistic tasks (public speaking, debate, memory recall or others)\n3) Author should discuss the lower accuracy and poor result on dataset like FACED - 23%.\n4) While subject-independent testing is performed, the method still assumes access to labeled data from some subjects in the target dataset for fine-tuning. No zero-shot results are presented.\n5) Exact subject split ratios are missing (e.g., how many subjects used for fine-tuning vs. test?). The number of trials per subject used for training vs. testing is unclear.\n6) The number of random seeds or runs per dataset is not stated clearly, though metrics are reported as mean ± SD.\n\nQUESTIONS:\n1) It is unclear how much labeled data from the held-out (target) dataset is used for fine-tuning. The paper mentions \"fine-tuning\" but does not specify: How many subjects or trials per subject are used?\n2) Whether experiments were repeated over multiple random seeds. Whether splits were consistent across models.\n3) The paper does not specify whether the same EEG preprocessing was used for the baseline.\n4) Why does the baseline not include any task-specific cross-dataset model? The authors should add them or discuss why these baselines were chosen.\n5) Why is the EEG dataset from other elicitation methods not used for experiments? Please discuss.\n6) The authors should discuss the dataset results, and why the results vary so much.\n\nLIMITATIONS:\nYes, the paper has a limitation section, but the paper does not explicitly address the potential negative societal impacts of EEG-based emotion recognition. Further, the paper would benefit from addressing the lack of diversity in the datasets (age, cultural background, mental health status), which limits generalisability to real-world, heterogeneous populations. The authors should also consider discussing risks such as misuse in surveillance or workplace monitoring, overinterpretation of emotional states, and the ethical concerns around applying such models without informed consent. Also, given EEG hardware is bulky, their real-world applicability would be limited; this should also be added.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}