Multi-dataset Joint Pre-training of Emotional EEG Enables Generalizable Affective Computing
Authors: Qingzhu Zhang
Jiani Zhong
Li ZongSheng
Xinke Shen
Quanying Liu
Venue: NeurIPS 2025 poster ()
Keywords: EEG
pre-training
affective computing
emotion
Primary area: neuroscience_and_cognitive_science
TL;DR: Multi-dataset joint pre-training with covariance centroid alignment enables cross-dataset generalization of EEG-based emotion recognition.

Abstract: Task-specific pre-training is essential when task representations diverge from generic pre-training features. Existing task-general pre-training EEG models struggle with complex tasks like emotion recognition due to mismatches between task-specific features and broad pre-training approaches. This work aims to develop a task-specific multi-dataset joint pre-training framework for cross-dataset emotion recognition, tackling problems of large inter-dataset distribution shifts, inconsistent emotion category definitions, and substantial inter-subject variability. We introduce a cross-dataset covariance alignment loss to align second-order statistical properties across datasets, enabling robust generalization without the need for extensive labels or per-subject calibration. To capture the long-term dependency and complex dynamics of EEG, we propose a hybrid encoder combining a Mamba-like linear attention channel encoder and a spatiotemporal dynamics model. Our method outperforms state-of-the-art large-scale EEG models by an average of 4.57% in AUROC for few-shot emotion recognition and 11.92% in accuracy for zero-shot generalization to a new dataset. Performance scales with the increase of datasets used in pre-training. Multi-dataset joint pre-training achieves a performance gain of 8.55\% over single-dataset training. This work provides a scalable framework for task-specific pre-training and highlights its benefit in generalizable affective computing. Our code is available at https://github.com/ncclab-sustech/mdJPT_nips2025.

Multi-dataset Joint Pre-training of Emotional EEG Enables Generalizable Affective Computing
Qingzhu Zhang'
Jiani Zhong? Zongsheng Li Xinke Shen'" Quanying Liu!"
DDpartmeent of Biomedical Engineering, Southern University of Science and Technology. Shenzhen, China School of Data Science. University of California, San Diego, La Jolla, CA. USA
$ School Science and Engineering, The Chinese University of Hong Kong, Shenzhen, China 1 124328520mail. sustech. edu. cn, (121090846, zongshenglin @eleenngngn@llile cuhk, edu. cn {shenxk, Liugy)Osustech. eerichh.ed..coom edu.cn
Abstract
Task-specific pre-training is essential when task representations diverge from generic pre-training features. Existing task-general pre-training EEG models struggle with complex tasks like emotion recognition due to mismatches between task-specific features and broad pre-training approaches. This work aims to develop task-specific multi-datate joint pre- pre-training framework for cross-dataset emotion recognition, tackling problems of large inter-dataset distribution shifts, inconsistent emotion category definitions, and substantial inter-subject variability. We introduce a cross-dataset covariance alignment loss to align second-order statistical properties across datasets, enabling robust generalization without the need for extensive labels or per-subject calibration. To capture the long-term dependency and complex dynamics of EEG, propose hybrid encoder combining Mamba- like linear attention channel encoder and a spatiotemporal dynamics model. Our method outperforms state-of-the-art large-scale EEG models by average of 4.57% in AUROC for few-shot emotion recognition and 11.92% accuracy for zero-shot generalization new dataset. Performance scales with the increase of datasets used in pre-training. Multi-dataset joint pre-training achieves performance gain of 8.55% over single-dataset training. This work provides scalable framework for task-specific pre-training and highlights its benefit in generalizable affective computing. Our code is available htttps:: //ggtttu......// heam//cclabb-oe hcclab-sustechh mdJJP _iip22225
Introduction
In domains like neuroscience, data is often limited, heterogeneous, and intrinsically linked to specific experimental tasks. This makes task-specific pre-training particularly promising paradigm, espe- cially for modalities such as EEG. Unlike generic large-scale pre-training, which aims to learn broad, task-agnostic representations, task-specific pre-training focuses inducing representations tailored to a coherent problem class, incorporating task-relevant inductive biases and domain structure. Recent adopt generic pre- pre-training by aggregating heterogeneous datasets across multiple tasks. While effective on broad benchmarks like sleep staging or anomaly detection, such models often struggle with tasks that involve more complex and nuanced neural representations (I). This performance gap stems from severe inter-task representation mismatch and the dilution of task-relevant signals during large-scale heterogeneous pre-training.
EEG foundation models (1 2. 5 (6)
"Corresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In contrast, task-specific multi-dataset pre- training, which leverages multiple datasets tar- geting the same cognitive function, offers fo- cused alternative. E.G--based emotion recogni- tion is a natural candidate for this paradigm: it has growing availability of labeled datasets (c.g.. SEED 7., DEAP (8), FACED (9) and strong practical value affective computing and BCI applications. However, developing such frame- work faces several technical challenges: large inter-dataset distribution shifts, heterogeneous
Multi-dataset Joint Pre-training
Cross-dataset Generalization
Various emotion categories
New emotion categories
Verious EEG layout
New datasets New subjects
Training Dataset 1
target dataset
2508
Training detaset
Rbb
Training dataset
EEG montages, inconsistent emotion category definitions, and substantial inter-subject variabil- ity. Existing approaches typically rely on one- to-one dataset adaptation and subject-specifie fine-tuning (10) I1I [2] [3 [4, [5) (Table [
Figure 1: Cross-dataset generalization through multi-datase pre-training. (left) Challenges of task-specific EEG pre-training. (right) Challenges or emotion categories (Fig. DT. These limita- of inference.
limiting their generalization to unseen datasets
tions call for more principled task-specific pre- pre-training strategy that aligns across datasets without requiring extensive labels or per-subject calibration.
Inspired by the evidence that EEG signals exhibit similar second-order statistical properties (such as correlations or covariance) across datasets, which can be aligned through simple transformations (16 17), we develop a label-free, task-specific pretraining framework that learns transferable emotion representations by aligning statistical patterns, while maintaining robustness across both datasets and subjects. Our framework harmonizes cross-dataset feature structures and captures discriminative neural dynamics through tailored spatiotemporal modeling (Fig. 2 Our main contributions are as follows:
Multi-dataset joint pre- Trainrain (mdJPT): We develop scalable multi-dataset pre- training framework for EEG-based emotion recognition, called mdJPT. We validated the superior performance of mdJPT over generic EEG foundation models in cross-datasee generalization.
Cross-dataset alignment (CDA) loss: We introduce a novel CDA loss that aligns second- order feature statistics across datasets, effectively mitigating inter-dataset and inter-subject distribution shifts. This enables zero-shot generalization to new emotion categories and unseen datasets.
We design a Mamba-like linear attention (MLLA)- based encoder (18) augmented with spatial transition convolutions and dynamic attention, enabling robust extraction of emotion-related long-term EEG dynamics and inter-channel dependencies.
Hybrid spatiotemporal encoder:
2 Method
2.1
Task-specific multi-datase joint pre-training
Our work aims to develop a unified framework for task-specific multi-dataset EEG training and cross- dataset generalization (Fig. 2 We introduce two complementary alignment losses: the Covariance Distribution Alignment (CDA) loss and the Inter-Subject Alignment (ISA) loss. The CDA loss aligns global second-order statistics (i.e., inter-channel covariance structures) across subjects and datasets (Fig. 2B. The ISA loss complements the CDA loss by a more fine-grained inter-subject alignment (Fig. 2C). For the EEG encoder, we aim to design physiologically plausible and computationally efficient architecture. consists of an MLLA-basse dynamics model (Fig. (B). The former captures long-term temporal dynamics from each EEG channel, and the latter integrates information across channels and time, enabling the model to learn coordinated patterns of brain activity.
channel encoder (Fig. (D) and a spatiotemporal
Our approach is validated under two settings: (i) cross-dataset suhject-independent (few-shot) classification and (ii) sero-shot generalization. The few-shot classification consists of three stages:

---

multi- joint pre-training, classifier fine-tuning, and testing. In the first stage, an EEG encoder is pre-trained on multiple datasets using CDA and ISA loss. In the second stage, the encoder is frozen, and classifier is fine-tuned on the extracted representations from few labeled subjects in the target dataset. In the final stage, the trained model evaluated on the remaining subjects of the target dataset (Fig. A The zero-shot generalization setting has no fine-tuning stage, with the trained model directly generalized to a new dataset.
Table 1: Comparison with existing cross-dataset emotion recognition methods.
Multi-dataset joint pre-training?
Generalizable to new emotion categories?
Generalizable to new subjects w/o finetuning?
Methods
PESD (3)
E'STN 14]
JCFA 15
X
X
Imtiaz & Khan (10)
X
SCMM 12
DBDG (T]
X
mdJPT (Ours)
2.2 The EEG encoder
MLLA channel encoder. The MLLA architecture is employed for modeling long- long-range dependen- cies in EEG. Preprocessed EEG data with heterogeneous electrode layouts from different datasets are first interpolated to standardized 60-channel configuration based on the 10-20 International System (19) to ensure spatial consistency. The interpolated EEG input can be denoted as x E RCXL
Then we split EEG signals into overlapped segments. Specifically, for EEG time series of the tt channel, (()) E R1xL, we divide them into overlapping strided patches (Fig. DD), with patch length of P and the stride length of S. The sequence of patches are denoted xp RNNx where N₂ [-_ + 1]) indicates the number of patches. Patches derived from each channel are fed independently into a MILA (18) encoder. For each (c) the MILA backbone then provide results zi, (c) E N where Ki is the output dimension of MILA encoder. Each EEG segment produces output å E RCXN, xK, The MILLA encoder includes input gate, linear attention module, and a forget gate. The input gate contains linear layer and a convolution layer for temporally local processing. With strided patches inputs, the linear layer is functionally similar strided convolution. The linear attention operator can model global dependencies, with its output multiplied by the output of linea--lyer-compprised forget gate filter salient patterns.
The spatialtemporal dynamics model. EEG signals exhibit complex spatiotemporal dynamics, with temporally varying spatial co-activations across electrodes. To capture these inter-channel dependencies, we introduce a spatiotemporal dynamics model that employs local attention to highlight salient spatial-transite patterns at each time step (Fig. ( Firstly, we pass è through trainable linear spatial projector W, E RCXC mapped to latent space for covariance alignment (see section 2.3): p W,ă, where E 2CCNNiX, and p(d) E RCN,, = 1,2, .... K₁ represents slice in p. A spatial transition convolution is then employed, utilizing convolutional kernels of size C x L1 to capture spatial variation patterns across multiple time steps L1 The convolutional kernels use dilations with various temporal intervals, enabling the extraction of EEG variation patterns across different time scales, yielding latent spatiotemporal representations (1) E KxN, (K is the number of hidden dimensions and N1 is the number of time steps. Subsequently, local attention is adopted to estimate time-varying importance weights spatiotemporal representations following (20). A depthwise one-dimensional temporal convolution, an average pooling over the temporal dimension, and a pointwise convolution mixing different channels are adopted to estimate the attention weights based on latent spatiotemporal representations. The attention weights are multiplied with the spatiotemporal representations to yield the weighted latent patterns (2) E RKXN, See Appendix for details of the spatiotemporal dynamics model.
3

---

The Overall Framework
MILLA Channel Encoder
Channel Channel-indeeerdeett imput
Patchee
Delisel
Detasel 2
Dalaset
Target Detaset
APS8, 2033
28E 2
AASE S6B Whi
A&A
Trainable
Channel interplattio
Interpolation
Local
Inpur Gate
Tokena
Frozen
EEG Encoder
EEG Encoder
Global
Lieer Attention
Lineer Allention
Mamba Like
MLLA Channel Encoder
MLLA Charnel Encoder
Locall
Forget
Paramotars
E
Spatlotemporal Dynamics Model
Spatiotemporal Dynamics Model
Spaliolemporel
Dynamics Model
Muticharnal EEG
Transition Extraction
Emboodingo
Spattsl Projection
Cowerience
Centroid
Projector
Classifer
Clessifier
Cross-Dalaset Alignment
Inter-Subject
Ciaeeifiaalin Loss
Locol attention
Alignment
silertion
Spatotompord
Stage Multi-dalasel joint pre-training
Stage 2 (oplional): Cleesifier fine-tuning
Slage Testing
patterns
Cross- Dataset Alignment Loss
Inter-Subject Alignmant Loss
Coveriancee
Sub
X
Untoset
Sub
Postlive pel
Negative pair
Bub
Sub
Centroid
Aligmmett
05
ng
Dataset
Figure 2: The overall framework, model architectures, and loss designs of mdJPT. A) The overall framework. Our approach comprises three stages: multi-dataset pre-training, optional classifier fine- tuning, and testing. The EEG encoder integrates an MLLA channel encoder with spatiotemporal dynamics model. Pre-training combines CDA and inter-subject alignment (ISA) losses. The resulting model supports either zero-shot generalization or fine-tuning with a lightweight classifier on limited target data. Final evaluation performed on held-out subjects. B) The cross-dataset alignment loss. The covariance is calculated from the latent representations, and covariance centroids of subjects from different datasets are aligned. C) The inter-subject alignment loss. EEG samples from two subjects corresponding to the same time segment within trial are regarded as positive samples, and those corresponding to different trials are regarded as negative samples. contrastive loss is employed to pull together the positive samples and push apart the negative samples. D) MLLA channel encoder. channel-independent Mamba-like linear attention encoder is employed to process the time-slided patches of each EEG channel. E) Spatiotemporal dynamics model. The multichannel EEG embeddings are processed by spatial projections and transition extraction to obtain spatiotemporal EEG patterns, then local attention is employed to estimate dynamic attention weights on these patterns.
2.3 Cross-dataset alignment loss
To mitigate the discrepancy between datasets and subjects, we design the CDA Loss based on covariance centroid alignment. Specifically, we compute the subject centroid of the covariance matrices derived from the EEG latent space (p) in training batch. The loss minimizes the Euclidean distance across centroids different datasets or subjects, thereby reducing distributional discrepancies in EEG feature representations across datasets.
Each training batch contains EEG data from 2M subjects, where M is the number of datasets involved in pre-training. Um samples are extracted from each subject, where is the number of trials in dataset m. The CDA loss needs to align the covariance centroid of the 2M subjects. The covariance of the spatial projection layer's output p is calculated:

---

E(d) 5,0
p!u
(p!.)
p(o) T
.d= 1,,,.... K
(I)
N₁
(d) where ps,o E RCXM is the hidden representation from trial subject ps,v E RCC is the temporal (d)
(d)
average of ps.e. The covariance of all samples from each subject is averaged to obtain the subject-level covariance centroids:
( 6,07
-. 1,2 Ki
Umn
==1
yielding 2M subject-wise centroids (T(4) pairwise Euclidean distances between 2M centroids:
( 2M
} per dimension d. The alignment loss measures
Ld
Σ
Σ
((d)
d
100=1,2,, 1,2, K1
m/L 12 m=
where || denotes the Frobenius norm. The loss all K₁ dimensions summed, and the logarithm taken to yield the final Cross-Dataset Alignment Loss:
K Σ La+t + 11.
=
(4)
2.4 Inter-subject alignment loss
A contrastive-learinn based ISA loss is employed to mitigate inter-subject discrepancy. In the emotional EEG datasets, subjects are required to watch series of emotional video stimuli. ISA loss aims to distinguish whether EEG segments from two subjects correspond to the same stimulus or different stimuli. This approach pulls closer the representations from different subjects' similar states and pushes apart the representations of different emotion states (2I). ISA loss enables the model to learn potential emotion-relevant EEG representations without requiring explicit emotion labels, thereby overcoming the challenge of inconsistent emotion label categories across different pre-training datasets.
In a training batch, a positive pair is formed by pairing EEG segments from two different sub- jects who were exposed to the same emotional stimulus. Samples corresponding mismatch- ing emotional stimuli form the negative pairs. The contrastive loss is calculated on the two sub- jects from each dataset and summed over all datasets. We denote the samples dataset m as: ( 1, Cm....mm 8 E A, B). For sample aA .. forms a positive pair with it, and the other 2(N - samples {x = 1, mm: j o s E A, B} form negative pairs.
The contrastive loss is caleulated based the output of projector. The projector contains two convolutional layers over the temporal dimension. It receives the output of the EEG encoder (2) and projects it to h E XN,,, The similarity between two samples is defined as:
sim(h km.
(5)
1/
The normalized temperature-scaled cross-entropy loss 22 is calculated based on the sample similar- ity:
exp(sim(h 19 1BB 1)T))
m. = log
(6)
j= 1 1iz1 exp(sim(h1 mm ))T) + E exp(sim(hA) k I/T)
where [[z E {0, 1} is an indicator function. It is set to if / i. By minimizing the loss function, the model will increase the similarity between h and h, in contrast to all other possible sample pairs involving haa The ISA loss for a training batch is:
5

---

Table 2: Summary of emotion EEG datasets used for cross-dataset evaluation.
Dataset
Sampling Rate (H2)
#Subjects
#Trials
#Emo. Classes
EEG Device (#Channels)
SEED
1000
15
45
3
ESI NeuroScan System (62)
SEED-IV
1000
15
72
4
ESI NeuroScan System (62)
SEED-V
1000
16
45
5
ESI NeuroSean System (62)
SEED-VII
1000
80
7
ESS NeuroSean System (62)
FACED
250/1000
123
28
2/9
NeuSen. W33,, Neuracle (32)
DEAP
512
32
40
2
Biosemi Two system (32)
M
Um
CISA
(7)
m==1
im
km
The total loss in pre-training is defined as:
-= LISA + ALODA
(8)
3 Experiments
3.1 Datasets and experiment setup
We employed multiple EEG datasets with varying numbers of emotion categories and recording chan- nels for model pre-training (Table All datasets underwent the same preprocessing pipeline, includ- ing downsampling, filtering, ICA-based artifact removal, channel interpolation, and re-referencing. See Appendix for details.
SEED series. The SEED series contains several emotional EEG datasets in which participants watch emotional video stimuli. Participants EEG signals in response videos were recorded with the ESI NeuroScan System (62 channels, 1000 Hz). We used SEED (15 subjects, 45 trials per subject) (7)
SEED-IV (15 subjects, 72 trials per subject) (33, SEED-V (16 subjects, 45 trials per subject) (24
and SEED-VI (20 subjects, 80 trials per subject) (25) in this study.
The FACED dataset (9) records 32-channel EEG signals from 123 subjects they watched 28 video clips eliciting nine emotions (amusement, inspiration, joy, tenderness, anger, fear, disgust, sadness, and neutral). The EEG signals were recorded using wireless EEG system (NeuSen.W3) Neuracle, China) at a sampling rate of 250 or 1000 Hz. To match the number of subjects in other datasets, which allows for fair comparison of the effect of dataset augmentation on performance, we used the first 20 subjects in this dataset.
FACED.
DEAP. DEAP dataset (8) contains EEG recordings from 32 participants they watched 40 one- minute music video clips. Participants reported their arousal, valence, like/dislike, dominance, and familiarity after watching each video. For each video segment, derived binary emotion labels by performing a median split on participant ratings along the valence, arousal, and dominance dimensions.
3.2 Implementation details
We use two-layer MultiLayer Perceptron (MLP) with ReLU activation and batch normalization for emotion classification. MLP inputs features extracted from the pre-trained EEG encoder. The EEG encoder is trained using the Adam optimizer. All experiments are implemented in Python 3.12.3 using the PyTorch 2.3.1 framework and are executed on an NVIDIA GeForce RTX 3090 GPU, Details of the hyperparameters are shown in Table ST and Table S2
To obtain low-dimensional features relevant for emotion recognition, we average the output of the EEG encoder along the temporal dimension for each sample (with a window length of 5 seconds). The features from consecutive EEG samples within trial are concatenated over time and smoothed using a linear dynamical system (LDS) model (7). The smoothed features are submitted to the ML.P classifier.

---

Table 3: Performance of cross-dataset subject- innpieeeiidd classification (leave-one-dataset-ou
evaluation).
Dataset
Methods
Accuracy
Precision
Recall
FI Score
AUROC
DE baseline
62.35 4.10
62.91 3.76
62.58 + 3.77
62.49 3.69
79.38 3.15
MMM
76.24 + 2.76
7
75.90 0.75
75.79 1.67
87.73 1.51
SEED
LaBraM
71.47 + 3.98
70.71
71.03 1.37
70.84 + 1.26
86.15 + 0.96
EEGPT
71.87 2.36
71.51 +.66
71.99 2.61
71.66 2.35
85.41 1.34
mdJPT (ours)
79.65 1.24
79.84 1.17
79.62 1.20
79.45 1.32
92.98 1.00
DE baseline
45.44 1.99
45.59 1.96
45.28 + 1.97
45.10 1.84
70.03 1.88
MMM
55.61 4.12
55.22 +
55.47 + 1.46
55.30 + 2.46
75.87 1.20
SEED-IV
LaBraM
47.88 = 3.41
47.81 2.24
48.01 + 1.78
47.89 + 1.29
69.97 1.78
EEGPT
44.10 = 1.77
43,82 + 3.56
44,16 2.86
43.91 # 2.51
62.32 # 1.93
mdJPT (ours)
53.53 0.75
53.81 1.02
53.67 0.78
53.37
77.51 0.34
DE baseline
45.58 1.92
46.02 = 1.96
1.92
45.55 # 1.94
73.64 1.54
MMM
58.64 3.56
58.51 1.90
59.01 3.00
58.68 1.20
82.80 1.94
SEED-V
LaBraM
#.53
41.69
4
41.80 1.56
70.81 2.72
EEGPT
45.27 = 3.66
44.97 2.79
4
45.06 2.16
67.21 2.06
mdJPT (ours)
65.02 # 0.98
65.06 + 1.28
65,53 + 0.66
64.85 + 1.01
88.70 + 0.98
DE baseline
29.12 + 1.20
28.24 1.22
27.67 + 0.90
27.64 1.00
64.78 0.83
MMM
30.29 + 2.63
3.553
2.225
30.12 + 2.27
68.37 + 1.84
SEED-VII
LaBraM
26.85 + 2.57
2..23
4.33
26.45 3.05
66.62 1.87
EEGPT
27.81 = 2.58
27.78
28.03 2.89
27.80 2.18
59.77
mdJPT (ours)
43.93 0.69
0.88
43.03 + 0.59
43.07 + 0.70
79.27
baseline
19.97
17.73
19.16
15.81 + 1.53
60.63 + 1,38
MMM
4.112
5.005
4188
21.81 + 4.74
59.79 3.11
FACED
LaBraM
20.35
3.776
2..25
20.23
67.59 2.10
EEGPT
21.55
3..52
21.62
23.31
59.96 2.07 4.153
mdJPT (ours)
33339
2.14
3.88
DE baseline
5.154
55.60
55.54
55,43 1.13
56.72 0.62
MMM
3.443
7.22
76.80 + 65.66 + 1.53
70.50 + 1.70
77.80 11.53
DEAP
LaBraM
67.45 5.63
61,39 5.588
63.45 # 3.56
77.83 0.65
EEGPT
63.33 + 4.38
56,48 +: 6.74
64.73 + 1.26
60.32 + 2.77
74. 4.31
mdJPT (ours)
71.71 = 10.38
71.78 10.42
71.71 10.38
71.69 + 10.38
75.75
12.20
DE baseline
43.00
42.68
42,70
48.19
68.13
MMM
54.54
51.29
53.20
52.03
75.39
Average
LaBraM
45.97
44.75
45.60
45.11
73.16
EEGPT
47.32
44.36
45.96
45.04
68.14
mdJPT (ours)
56.22
56.26
56.01
55.46
79.96
3.3 Results of cross-dataset subject-independder classification
Under the cross-dataset subject-independen (few-shot) setting, our method uses a small subset of subjects from the target dataset to train an MLP classifier, which can generalize to the remaining subjects. We split the target dataset at 1:3 subject ratio for MLP training and testing. We repeated the random split 6 times and report their mean and standard deviation. A leave-one-dataset-on cross-validation employed to evaluate the model: the EEG encoder pretrained on datasets except the target set. The performance is evaluated using five metrics: accuracy, precision, recall, Fl Score, and AUROC. Our method is compared to three generic pre-trained EEG models (i.c., MMM (26), LaBraM (1) EEGPT 3)). We used the publicly available pre-trained parameters of these models. baseline method directly extracting DE features from EEG signals without pre-training also implemented. Features of the comparison methods are submitted them to an LDS smoother and an MLP classifier with the same procedures as ours.
Our method obtains the best performance across all evaluation metrics on the SEED, SEED-V. V, and SEED-V VII datasets. SEED-IV, FACED, and DEAP, also ranks within the top two across nearly all metrics (Table 3 In terms of average performance across all datasets, our method achieves the highest scores for all the metrics, improving the accuracy, precision, recall, score, and AUROC by 1.68%, 4.97%, 2.81%, 3.43%. and 4.57% (and relative improvement of 3, 3.08%, 9.69%, 5.28%,

---

6.59%, and 6.06%), respectively over the state-of-the-arg These results demonstrate the superiority of our multi-dataset joint pre-training framework, supporting the effectiveness of tusk-specialized pre-training over general-purposs pre-training approaches in emotion decoding. mdJPT is also more compact than existing pre-trained models, with less trainable parameters (1.0M, Table(STO) The model's s performance is quite stable across different random seeds (Table S9). Further results of different classification settings on the DEAP dataset and the transfer from DEAP to other datasets provided in Appendix E.8 and E.7, respectively. The confusion matrix of fine-grained emotion classification on the FACED dataset provided in Appendix E,4 (Fig. ST
3.4 Results of zero-shot generalization to a dataset
Under the zero-shot setting, the EEG encoder is pre-trained on five datasets and directly evaluated on a held-out dataset without any fine-tuning. each testing sample, we compute cosine similarities across all samples in the target dataset and identify its nearest neighbor in the representation space. prediction is correct the nearest neighbor shares the same emotion label. The overall accuracy reflects the encoder's zero-shot ability produce discriminative and dataset-invarias representations. The performance mdJPT also compared with DE baseline, LaBraM, and EEGPT. MMM not included it requires fine-tuning new datasets to optimize region-wise tokens.
Our method consistently outperforms comparison methods on all datasets by a large margin, with an improvement of 11.9% on average, and relative improvement of 40.0% over the state-of-the- art. SEED dataset, mdJPT outperforms the second best model, EEGPT, by 17.05%. the challenging FACED dataset, other models performed close to chance level (11%). with only mdPT achieving beiter-than-chanee accuracy. On the DEAP dataset, mdPP achieved an impressive accuracy of 73.3%, even outperforming the best fine-tuned model. This may be due to overfitting in fine-tuned models when trained on a small number of subjects with large individual differences.
Table 4: Performance zero-shot generalization.
Model
SEED
SEED-IV
SEED-V
SEED-VII
DEAP
FACED
DE baseline
50.54
46.86
43.32
23.58
55.40
8.88
LaBraM
48.54
45.24
39.70
21.33
67.04
10.21
EEGPT
55.42
34.02
36.97
20.96
62.78
11.64
mdJTT (Ours)
72.47
50.59
52.91
33.26
73.34
17.44
To visualize cross-dataset alignment after joint pre-training, we compare feature distributions of DE and our mdJPT model in Fig. 3 DE features form tight speccicc clusters, whereas mdJPT features show improved intermixing with lower silhouette scores (Fig. S22. indicating better cross- dataset alignment. Emotion discriminability and inter-dataset consistency are analyzed Appendices E.5 and E. respectively.
FACED
DEAP
SEED-V
DE Feature
Pretrained Feature
Figure 3: t-sne visualization of features. Compared to DE, pretrained features show better cross- dataset alignment, with features from different datasets evenly intermixed.

---

3.5 Impact of more datasets used in pre-training
To investigate the impact of using increasing number of pre-training datasets mdJPT's perfor- mance, we took SEED-V V as the target dataset and pre-trained the model on various combinations of datasets excluding SEED-V. As shown in Fig. 4 the model's performance consistently improved as more datasets were included the joint pre-training. Notably, in all combinations, adding a new dataset for joint pre-training always led better results than the previous setting without it. In particular, mdJPT achieved an 8.55% improvement (15. 14% relative improvement) when using the maximum number of training datasets compared to the best performance obtained with single dataset. This highlights the advantage of joint training across multiple datasets in enhancing the model's generalization ability unseen target data.
$34544574F4D
65.02
+3+S7+F+D
64.83
Pretrain Datasets
S34S7#F
62.25
59.71
S3+S7
59.46
S3
56.47
55.46
50
70
Accuracy (%)
Figure 4: Generalization performance of mdJPT with increasing number of datasets. Few- shot setting is used here with SEED-V dataset as testing dataset. Abbr. S3, S4, S7, F, D stand for SEED, SEED-IV, SEED-VII, FACED, and DEAP, respectively.
3.6
Ablation Study
We conduct ablation studies with SEED-V as the target dataset evaluate the contributions of the proposed components: CDA loss, ISA loss, and the MLLA encoder.
CDA loss. We use range of CDA loss weights to examine impact of cross-dataset covariance alignment on model performance. As shown in Table [ introducing CDA loss with small positive factors generally improves accuracy com- pared to the baseline without CDA (factor 0). The best performance is observed at a CDA factor of 0.02, achieving 65.02% accuracy. Performance slightly declines with larger factors, suggesting that overly strong alignment may hinder emotion-diseriminativ representation learning.
Table 5: Effecc of CDA factor
CDA factor
ACC
STD
63.52
0.64
0.005
64.98
0.59
0.010
63.81
1.51
0.020
65.02
0,98
0.050
64.70
0,80
0.075
64.10
1.19
0.100
64.08
1.18
ISA loss. When removing the ISA loss in pre-training, the model's performance drops significantly
Table (Table[6), falling even below the DE baseline. This indicates that ISA loss is not only important for aligning representations across individuals but also plays critical role in the learning of emotion- related representations. Further comparison with supervised contrastive loss and temporally unaligned sampling strategy is provided in Appendix E.2.

---

Table Ablation of ISA loss
Model
Accuracy
Precision
Recall
F1 Score
AUROC
DE baseline
45.58 + 1.92
46.02 = 1.96
45.98 = 1.92
45,55 + 1.94
73.64 + 1.54
w/o LISA
30.51 + 1.16
24.61 5,48
27.57 : 1.89
23.21 3.76
60.29 = 1.95
W LISA
62.35 4.10
62.91 3.76
62.58 3.77
62.49 + 3.69
79.38 # 3.15
MLLA encoder To assess MLLA channel encoder's effectiveness in modeling EEG temporal dynamics, we replace with a vanilla multi-head transformer (27). The transformer encoder uses two attention heads, treating the EEG data from all channels at each time step as a single token. It projects these tokens into a 128-dimensional latent space and then to a 32-dimensional output. The results shown in Table 7 demonstrate that the MILA encoder consistently outperforms the vanilla Transformer across multiple metrics, confirming its advantage in capturing the temporal patterns in EEG signals. The advantage of the spatiotemporal dynamics model over transformer layer is shown in Appendix E.1.
Table 7: Comparison of MLLA Channel Encoder with Transformer
Model
Accuracy
Precision
Recall
F1 Score
AUROC
DE baseline
45.58 + 1.92
46.02 + 1.96
45.98 = 1.92
45.55 1.94
73.64 # 1.54
Transformer
59.90 : 1.32
60.39 1.52
60.37 = 0.75
59.73 1.27
84.53 # 1.25
MILLA
62.35 + 4.10
62.91 3.76
62.58 # 3.77
62.49 3.69
79.38 + 3.15
4 Discussion and conclusion
In this study, we propose mdJPT, scalable multi-dataset pretraining framework for EBG-based emotion recognition. The model supports cross-dataset transfer and demonstrates strong generaliza- tion ability, confirming that joint pretraining across datasets can significantly enhance performance on downstream tasks. To mitigate inter-datase and inter-subject distribution shifts, we introduce
CDA loss, which aligns sample covariance across datasets and subjects. We conduct extensive experiments to validate the effectiveness of the proposed pre-training strategy. Results show that for both few-shot and zero-shot settings, the model generalizes well target datasets and achieves superior performance to existing large-scale EEG pretraining methods, especially for the most chal- lenging zero-shot setting. This highlights the broader applicability and flexibility of the proposed approach in real-world emotion recognition scenarios. Furthermore, this task-specific multi-dataset joint pre-training paradigm can offer valuable insights for other brain-computer interface tasks.
Limitations and future directions. Our study still has several limitations. First, despite the advantage over comparison methods, the performance of mdJPT on the most challenging nine- category classification of the FACED dataset is still not satisfactory, indicating residual feature distribution shifts persist. Future work is needed further improve the model's generalizability to more fine- mee-grained emotions. Second, our evaluation primarily focuses on video-induced emotion paradigms. We include an exploratory analysis to test the model's s generalizability to imagery- induced contexts the EmoBEG-MC dataset (28) (Appendix E.3). Generalizability to diverse emotion elicitation paradigms requires further validation. Third, our approach addresses signal- level heterogeneity but does not model individual differences in emotional experience, potentially overlooking nuanced affective states. Future work could incorporate personalized emotion ratings through soft contrastive learning to bridge this gap. Fourth, the generalizability of our findings is constrained by the limited demographic diversity (c.g., age, culture, and health status) of current EEG emotion datasets, and the practical deployment is further limited by the cumbersome nature of current EEG hardware. The development of wearable devices may help overcome this barrier. We also note broader societal and ethical implications, including potential misuse in emotional surveillance profiling without consent. To mitigate these risks, future efforts should establish regulatory guidelines and develop
frameworks for responsible model deployment.
The authors would like to express their gratitude to Professor Dan Zhang from the Department of Psychological and Cognitive Sciences, Tsinghua University, for his
Acknowledgements
10

---

valuable suggestions. This work was supported by the National Natural Science Foundation of China (62472206), National Key R&D Program of China (2025YFC3410000), Shenzhen Science and Technology Innovation Committee (RCYX2023122 21 1090405003, KIZID02309231 1.5221044 RCBS2023121 1090748082), Guangdong Provincial Key Laboratory of Advanced Biomaterials 1122212120100033 and the open research fund of the Guangdong Provincial Key Laboratory of Mathematical and Neural Dynamical Systems, the Center for Computational Science and Engineering at Southern University of Science and Technology.
References
[1] W.-B. Jiang, L.-M. Zhao, and B.-L.. Lu, "Large brain model for learning generic representations with tremendous eeg data in bei," arXiv preprint arXiv:2405. /876, 2024.
[2] X. Zhou, C. Liu, Z. Chen, K. Wang, Y. Ding, Z. Jia, and Q. Wen, "Brain foundation models: survey on advancements in neural signal processing and brain discovery," arXiv preprint arXiv:2503.0053 2000000000 2025.
[3] G. Wang, W. Liu, Y. He, C. Xu, L. Ma, and H. Li, "Eegpt: Pretrained transformer for universal and reliable representation of eeg signals," Advances in Neural Information Processing Systems, vol. 37, pp. 39 249-39 280, 2024.
[4] T. Yue, S. Xue, X. Gao, Y. Tang, L. Guo, J. Jiang, and J. Liu, "Eegpt: Unleashing the po- tential of eeg generalist foundation model by autoregressive pre-training," arXiv preprint 2024.
vrXiv:2410.1911
[5] C. Yang, M. Westover, and J. Sun, "Biot: Biosignal transformer for cross-data learning in the wild." Advances in Neural Information Processing Systems, vol. 36, pp. 78 240-78 260. 2023.
[6] W.-B. Jiang. Y. Wang. B.-L. Lu, and D. Li, "Neurolm: A universal multi-task foundation model for bridging the gap between language and eeg signals," arXiv preprini arXiv:2409.00101, 2024.
[7] W.-L. Zheng and B.-L. Lu, "Investigating critical frequency bands and channels for eeg-based emotion recognition with deep neural networks," IEEE Transactions on autonomous mental development, vol. 7, no. 3, pp. 175-175.77 2015.
[8] S. Koelstra, C. Muhl, M. Soleymani, J.-S. Lec, A. Yazdani, T. Ebrahimi, Pun, A. Nijholt, and I. Patras, "Deap: A database for emotion analysis; using physiological signals," IEEE transactions on affective computing, vol. 3, no. I, pp. 18-31, 2011.
[9] Chen, X. Wang, C. Huang, X. Hu, Shen, and D. Zhang, "A large finer-grained affective computing eeg dataset," Scientific Data, vol. 10, no. 1, p. 740, 2023.
[10] M. N. Imtiaz and N. Khan, "Enhanced cross-dataset electroencephalogram based emotion recognition using unsupervised domain adaptation," " Computers in Biology and Medicine, vol. 184, 109394, 2025.
[11] W. Li, S. Wang. S. Shao, and K. Huang, "Distillation-based domain generalization for cross- dataset beg-based emotion recognition," IEEE Transactions on Emerging Topics in Computa- tional Intelligence, 2024.
[12] Q Liu, W. Ye, Y. Liu, and Z. Liang, "Eeg-semm: Soft contrastive masked modeling for cross-corpus eeg-based emotion recognition," arXiv preprini arXiv:2408.09188 2024.
[13] F. Wang, Y.-C. Tian, and X. Zhou, "Cross-datasee eeg emotion recognition based pre- trained vision transformer considering emotional sensitivity diversity." Expert Systems with Applications, vol. 279, P. 127348, 2025.
[14] Y. Zhou, F. Li, Y. Li, Y. Ji, L. Zhang, Y. Chen, and H. Wang, "Enhancing cross-dataset eeg emotion recognition: A novel approach with emotional eeg style transfer network," " IEEE Transactions on Affective Computing, 2025.
[15] Q Liu, Z. Zhou, J. Wang, and Z. Liang, "Joint contrastive learning with feature alignment for cross-corpus eeg-based emotion recognition," Proceedings of the 1st International Workshop on Brain- Computer Interfaces (BCI) for Multimedia Understanding, 2024, pp. 9-17.
11

---

[16] A. Mellot, A. Collas, P. L. Rodrigues, D. Engemann, and A. Gramfort, "Harmonizing and aligning m/eeg datasets with rovariance-based techniques enhance predictive regression modeling," Imaging Neuroscience, vol. 1, pp. 1-23, 2023.
[17] A. Mellot, A. Collas, S. Chevallier, A. Gramfort, and D. A. Engemann, "Geodesic optimization for predictive shift adaptation on eeg data," Advances in Neural Information Processing Systems, vol. 37, pp. 32 828-288-38 32 855, 2024.
[18] D. Han, Z. Wang, Z. Xia, Y. Han, Y. Pu, C. Ge, J. Song, S. Song, B. Zheng, and G. Huang, "Demystify mamba in vision: A linear attention perspective," arXiv preprint wxXiv:2405.16600 2024.
[19] G. H. Klem, "The ten-twenty electrode system of the international federation. the international federation of clinical neurophysiology. Electroencephalogs Clin. Newrophysiol. Suppl., vol. 52,
pp. 3 6, 1999.
[20] X. Shen, Gan, K. Wang, S. Yang. Q. Zhang, Q. Liu, Zhang, and S. Song, "Dynamic- attention-based eeg state transition modeling for emotion recognition." IEEE Transactions Affective Computing, 2025.
[21] X. Shen, X. Liu, X. Hu, D. Zhang. and Song, "Contrastive learning of subject-invariee eeg representations for cross-subject emotion recognition," IEEE Transactions Affective Computing, vol. 14, no. 3, pp. 2496-2511, 2022.
[22] T. Chen, S. Kornllith, M. Norouzi, and G. Hinton, "A simple framework for contrastive learning of visual representations," " International conference on machine learning. PmLR, 2020, pp. 1597-1607.
[23] W. Zheng, W. Liu, Y. Lu, B. Lu, and A. Cichocki, "Emotionmeter: A multimodal framework for recognizing human emotions," " IEEE Transactions on Cybernetics, pp. 1-13, 2018.
[24] W. Liu, J.-L. Qiu, W.-L. Zheng, and B.-L. Lu, "Comparing recognition performance and robustness of multimodal deep learning models for multimodal emotion recognition," IEEE Transactions on Cognitive and Developmental Systems, 2021.
[25] W.-B. Jiang, X.-H. Liu, W.-L. Zheng, and B.-L. Lu, -ii: A multimodal dataset of six basic emotions with continuous labels for emotion recognition." IEEE Transactions on Affective Computing, pp. 1-16, 2024.
[26] K. Yi, Y. Wang, K. Ren, and D. Li, "Learning topology-agnosti eeg representations with geometry-awant modeling," Advances in Neural Information Processing Systems, vol. 36, pp. 53 8785-522 53 891, 2023.
[27] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t.. Kaiser, and I. Polosukhin, "Attention all you need," Advances in neural information processing systems, vol. 30, 2017.
[28] X. Xu, X. Shen, X. Chen, Q. Zhang, S. Wang. Y. Li. Z. Li, D. Zhang, M. Zhang, and Q. Liu, "A Multi-Context Emotional EEG Dataset for Cross-Context Emotion Decoding," Scientific Data, vol. 12, no. p. 1142, Jul. 2025. [Online]. Available: https://oiioon http://doooorg//1000 1038//41597-0055-5534922
[29] D. Kostas, S. Aroca-Ouellette, and F. Rudzicz, "Bendr: Using transformers and contrastive self-supervised learning task learn from massive amounts of eeg data." Frontiers in Human Neuroscience, vol. 15, p. 653659, 2021.
[30] P. Khosla, P. Teterwak, C. Wang, A. Sama,, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, "Supervised contrastive learning," Advances in neural information processing systems, vol. 33, pp. 18 661-18 673, 2020.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The paper contributes to multi-datasest joint-pre-traiinng of emotional EEG for generalizable affective computing.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Yes, the limitations have been discussed in Section 4 (Discussion and Conclu- sion).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiiatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer: [NA]
This work is focused on experiments and engineering, and does not include results that require theoretical proof.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The section 3 (Experiments) of provides all the necessary information required for reproduce the main results in detail.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our code is available at /nttps://anon/mntt 4open. /ene/etatuuu Covariance EEG_ Amotion-DICh The code for this research will be open-sourced, and the GitHub link is provided in the appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//ipps.cc/pullic/uuiiee
swwsssssicpyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The section 3.1-3.3 of specify all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The error bars in this paper are reasonably defined, illustrating inter-subject variability within each dataset, and all appropriate information is properly explained in the experimental results section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The section 3.3 (Implementation Details) provide the sufficient information on the computer resources.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't fidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ftttpsss//eurrpps.ccppllcll itt.
Answer: [Yes]
Justification: Ethics.
All research conducted in this paper conforms fully to the NeurIPS Code of
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The significance of this research direction is demonstrated in section 1 (Intro- duction), and its value the field is discussed in the section 4 (Discussion).
Guidelines:
16

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: The paper uses only publicly available datasets and does not involve any misuse of data models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Yes, the creators and original owners of all assets used in the paper are properly credited, and the licenses and terms of use are explicitly mentioned and fully respected.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
17

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssippsssie thcode wwwccom/aaaoo//c datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, the assets introduced in the paper are well documented, and the documentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper only publicly available datasets and does not involve crowd- sourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper uses only publicly available datasets and does not involve involve crowdsoureing nor research with human subjects.
Guidelines:
18

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This research does not involve LLMs any important, original, or non- standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy hbtp:eee//eerrrsscc/////ffeerrre for what should or should not be described.
19

---

A Related work
Cross-dataset EEG Emotion Recognition. Existing cross-dataset EEG emotion recognition mainly focuses on one-to-one dataset adaptation. Wang et al. (I3) proposed pre-trained vision transformer for cross-dataset EEG emotion recognition. They utilized cross-domain data mixup to blend source- target EEG distributions and adversarial domain alignment to minimize dataset-specificc biases. Zhou et al. (14 proposed an Emotional EEG Style Transfer Network (E*STN) for cross-dataset domain adaptation. Liu et al. (12) proposed soft contrastive masked modeling framework with learnable weights assigned to sample pairs in contrastive learning. It achieved state-of-the-art one-to-one dataset transfer on SEED, SEED-IV, and DEAP datasets, and can be transferred to datasets with different emotion categories. These methods need to be fine-tuned on each target subject, which constrains their generalization to new subjects without training data. Imtiaz and Khan (10) proposed a target data selection method to gradually select reliable target domain samples for training. They also employed test-time augmentation when the prediction confidence is low. The model is trained on labeled source domain data and unlabelled target domain data, with evaluations performed on the target domain test set. Li et al. (II) proposed distillation- based method, in which the knowledge from the target-domain model was distilled to the source-domain model. The subjects from the target dataset were divided into training and testing sets. The model was trained on the source dataset and training subjects of the target dataset, and tested on the left-out subject of the target dataset.
supervied Pre-training of EEG Models. Sulf-supe learning has achieved remarkable success in computer vision (CV) and natural language processing (NLP). Recently, many studies have adopted self-supervised learning frameworks to develop pre-trained models for EEG data (2) Kostas et al. proposed BENDR (29), a contrastive learning framework pre-trained on EEG recordings from over 10,000 people and evaluated on four downstream brain-computer interface tasks and sleep-staging task. BENDR employs a convolutional encoder to extract features from local temporal windows and utilizes random masking and contrastive learning as self-supervised pre-training strategy for EEG datasets. MMM (26) model addresses the challenges of channel discrepancies and spatial structure modeling across EEG datasets. It incorporates channel position encoding to embed 2D spatial information into representation learning. Additionally, region-level tokens are introduced to construct hierarchical spatial representation. MMM demonstrates competitive performance in emotion recognition tasks. The Biosignal Transformer (BIOT) (5) broke the time series of channel- wise biosignals into fixed-length tokens and combined them into "sentences". Then the standard Transformer can be trained with either supervised or self-supervised learning. The Large Brain Model (LaBraM) ① employed a vector- sector-quantized variational auto- encoder architecture to predict the EEG spectrum by discrete neural tokens. A Transformer model was then trained to reconstruct the masked neural tokens. LaBraM used EEG data of 2500 hours and achieved state-of-the-ard performance on tasks like abnormal detection and event type classification. EEGPT K employs dual self- supervised learning strategy by combining the masked reconstruction loss and the alignment loss with momentum encoder. It introduces joint local spatial and temporal embedding strategy to learn temporal variations and spatial co-activations of EEG. EEGPT achieved state-of-the-ar performance in downstream brain-computer interface tasks and sleep staging tasks with linear probing.
B Details of the model architecture
The spatiotemporal dynamics model. EEG signals exhibit complex spatiotemporal dynamics, with temporally varying spatial co-activations across the electrodes. To capture EEG dynamic properties, we firstly pass e through a trainable linear spatial projector W, E CRC'', .mapped to a latent space for cross-dataset covariance alignment (see section 2.3):
p W..
where E RCCNN, A spatial transition convolution is then employed, utilizing convolutional kernels of size M x L1 to capture spatial variation patterns across multiple time steps L1. The convolutional kernels use dilations with various temporal intervals, enabling the extraction of EEG variation patterns across different time scales:
( Wtr +P
(10)
where (1) E 1RK.KI W.r E K Li is the weight in tratial-transition convolution, "+" denotes the convolution operation. We used group convolution with K₂ spatial transition
20

---

convolutional kernels for each output dimension of the MLLA transformer, resulting in K2 dimensions in total. We implemented spatial transition convolution with four different dilations, with K K2/4 kernels for each dilation. The inputs are padded on the temporal dimension to ensure the outputs have the same temporal size N₁ as the inputs.
To effectively capture the evolving spatiotemporal patterns in EEG signals, we implement local attention module to assign dynamic attention weights on latent dimensions. The module contains temporal convolution operations, dimension-wise pooling, and linear mixing to model temporal dynamies and channel interactions. The temporal convolution and dimension wise pooling are formulated as follows:
A = Wat * (2) Á = AvePool(A)
(11)
(12)
where Watt E R denotes learnable temporal filters. K= = K2K2 denotes the number of filters. Here, we used group convolution with one temporal filter for each dimension. Average pooling with time step of is conducted on the temporal dimension of A. The inputs of the temporal convolution and average pooling padded to make sure the output A has the same size of h(), Then, linear mixing layer combines information across channels through a linear transformation:
K
B+ i 1,2,....F
(13)
yielding transformed features (2) RKXM Dynamic attention weights then obtained through a softmax activation function:
(2)
hit
(att)
(14)
Lj=1
producing attention weights (att)) E KRN, These dynamic weights modulate the spatiotemporal representations (1)) through Hadamard product:
(3)) (att) 3 (1)
(15)
The weighted feature representation (3( E RxN, is ultimately employed as the output of the EEG
encoder for affective state recognition.
The projector. A projector is employed between the output of the EEG encoder and the inter-subject
alignment loss to obtain a dedicated feature representation for inter-subject alignment (ISA) loss. The projector consists of an average pooling layer and two temporal convolution layers:
(4) = (oo
(16)
h ReLUU( * l2(
(17)
h WISA (5)
(18)
where h E RKx Nr. WISAI, WISA2 RR the temporal convolution filters. Group
convolution with one convolution filter for each input dimension is employed here.
C Data pre-processing
To ensure consistency and comparability across datasets, we implemented a standardized automatic preprocessing pipeline using Matlab, First, EEG signals were downsampled to 125 Hz and filtered with a 0.5-47 Hz bandpass filter. The signals were then segmented into trials based on the onset and offset of emotional video stimuli. For each channel, if the proportion of data exceeding specified multiple (m) of the median value was greater than certain percentage (n) of the trial duration, the channel was denoted as noisy. We used two sets of thresholds: m=3, n=0.4 to identify long-lasting artifacts, and m=30, n=0.01 to detect short-term large artifacts. Noisy channels identified in this manner were interpolated using their three nearest neighboring channels. Next, independent component analysis (ICA) was applied to remove artifacts caused by eye movements or muscle activity. We utilized automatic component labeling tools in EEGLab. Components labeled as eye-
21

---

or muscle-related with a confidence level exceeding 0.8, the default thresholds in EEGLab, were removed. During the initial noisy channel detection, we observed that frontal channels (such as Fpl and Fp2) were frequently identified as noisy, which interfered with the subsequent ICA-based detection of eye movement components. To address this issue, we excluded channels Fp1, Fp2. F7, and F8 from the initial noisy channel interpolation. After applying ICA, the noisy channel detection procedure was conducted again, including all channels at this time. Finally, the data were re-referenced to the common average.
D Hyperparameter Settings
The hyperparameters in model pre-training and fine-tuning shown in Table ST The hyperparam- eters the EEG encoder, including the MLLA channel encoder and the spatiotemporal dynamics model are shown in Table
Table S1: Hyperparameters of pre-training and fine-tuning.
Hyperparameters
Values
epochs
20
learning rate weight decay
0.0005
Pre-training
0.0001
ISA loss temperature
0.07
window length stride
5 seconds
seconds
Weight of CDA loss
0.02
batch size learning rate weight decay
256
0,0005
Fine-tuning
0.0022
hidden units epochs
128
25
Table S2: Hyperparameters of the BBG encoder.
Hyperparameters
Values
patch size patch stride hidden dim out dim
32
6
MLLA channel encoder
128
32
depth
2
attention head number
8
number of spatial transition convolution time length of spatial- transition convolution dilations
4
spatial-transition
3
Spatiotemporal dynamics model
[1.3,6,12]
length temporal filters
15
length of average pooling
15
length of temporal filters temperature of Softmax
3
Projector
1.0
E
Additional results
E.
The effectiveness of spatiotemporal dynamics model
To assess the effectiveness of the spatiotemporal dynamics model in modeling EEG temporal dy- namics, replace it with Transformer layer comprising multi-head self-attention (7)). The
22

---

Transformer's 's attention mechanism considers global interactions of all temporal positions to capture dependencies. Table S3 demonstrates that the spatiotemporal dynamies model outperforms the Transformer across all metrics, with an increase of 1.39% in accuracy, confirming its advantage in capturing the spatiotemporal dynamic patterns in EEG signals.
Table S3: Comparison of the spatiotemporal dynamics model with Transformer
Model
Accuracy
Precision
Recall
F1 Score
AUROC
DE baseline
45.58 + 1.92
46.02 1.96
45,98 + 1.92
45.55 1.94
73.64 # 1.54
Transformer
63.63 + 0.96
64.13 1.63
63.56 + 1.35
63.35 1.09
87.70 = 0.44
Spatiotemporal dynamics
65.02 # 0.98
65.06 + 1.28
65.53 + 0.66
64.85 1.01
88.70 = 0.98
E,2
Comparision of contrastive loss and sampling strategy
To demonstrate the effectiveness of the ISA loss over other contrastive loss, we compared it with supervised contrastive learning (supervised CL), which regards samples with the same emotion labels as positive pairs and those with different emotion labels as negative pairs in each dataset. We use SupCon loss (30) and keep all other settings the same. Our strategy outperforms supervised contrastive learning by a large margin (Table 54). indicating the effectiveness of our ISA loss for EEG alignment.
We also evaluated the superiority of the proposed sampling strategy alternatives. In the proposed temporally aligned sampling strategy, two samples in positive pair come from the same trial and start at the same timestamp. In other words, participants were watching the same video scenes in positive pair. To test the necessity of temporal aligmment of positive pairs, we draw positive samples from the same trial but with randomly selected unmatched start timestamps. Without temporal alignment, the model performance drops significantly (Table S5). This could be due to that the temporally aligned samples provide an "anchor" for mitigating the large inter-individual differences and extracting meaningful shared representations across subjects.
Table S4: Comparison of self superviid frameworks
Method
Accuracy
Precision
Recall
F1 Score
AUROC
Supervised CL
47.99 +: 2.51
47.89 + 2.22
47.86 + 2.41
47.53 + 2.33
76.28 + 1.73
ISA (ours)
65.02 + 0.98
65.06 + 1.28
65.53 + 0.66
64.85 + 1.01
88.70 + 0.98
Table S5: Comparison of alignment method
Method
Accuracy
Precision
Recall
F1 Score
AUROC
Not aligned
55.47 + 1.71
55.17 = 1.41
55.56 + 1.70
54.94 + 1.49
80.97 + 1.29
Aligned (ours)
65.02 + 0.98
65.06 1.28
65.53 0.66
64.85 1.01
88.70 +\2 0.98
E.3
Generalization to the imagery context
Most widely used EEG emotion datasets employ the video-induced paradigm. To assess the model's generalizability beyond video stimuli, we further perform experiments on the EmoEEG-MC dataset (28), multi-context dataset including imagery-induced paradigm (guided narratives with active imagination), eliciting more internally driven and sustained emotions. We evaluate on the imagery task using the few-shot setting: pre-train on six video-induced datasets, then fine-tune on 1/4 of EmoEEG-MC subjects and test on the remaining. Our method outperforms the DE baseline in the imagery-induced context (Table (66). indicating the effectiveness of pre-training in context- independent representation learning. It should be noted that the performance of transfer to imagery context remains relatively low. Future work is needed to mitigate the cross-context discrepancy.
23

---

Table S6: Accuracy on EmoEEG-MC dataset with imagery- induced emotion
Model
Accuracy
Precision
Recall
F1 Score
AUROC
DE baseline
18.38 1.12
17.91 + 1.62
18.38 + 1.12
17.87 + 1.37
54.11 + 1.36
mdJPT (ours)
20.56 0.79
20.83 + 0.66
20.56 + 0.79
19.85 = 2.11
56.14 + 0.78
E.4
Confusion matrices
To investigate the detailed confusion patterns on FACED and EmoEEGG-maagey, we calculated the confusion matrices on these datasets (Fig. ST. Each row represents true label, and each column represents predicted label. On the FACED dataset. the confusion matrix shows a diagonal dominance: within each row, the diagonal entry is the highest, meaning the model is more likely to predict the true label than any other single label. Among all categories, disgust, tenderness, and neutral have the highest classification accuracy, indicating they have more discriminative neural features. Joy tends to be confused with amusement or inspiration, indicating more similar neural representations across these fine-grained positive emotion categories. On the EmoEEG-imagee dataset, neutral and tenderness have the best recognition accuracy, indicating the cross- cross-context neural similarity for these emotions.
Cartaaia
Contasion Errttt-mmapeyy
6:7
8.17
C:5
CB5
600
GG4
A44
6.00
3.2
349
111
0015
0:5
500
6:00
049
413
0.0
114
630
X12
616
388
112
668
6:1
031
4.29
0.9
K00
010
1/8
8:14
606
610
314
X13
111
033
213
000
X32
Aur
050
831
6:1
0.0
600
011
XM
632
X11
312
675
Awyor
0.0
608
633
029
600
013
A1
631
X22
/
Figure S1: Confusion matrices of emotion classification results on FACED and EmoEEG-imagery datasets.
DE features Silhouette Scores
Latent representations Silhouette Scores
0.30
030
brutto imagery
0.22
6.06
0.22
brott0 imagury
MC20
0.2
0.50
0.20
MCE3
0.13
DEAP
0.20
0.15
0.10
DIAP
0.01
0.00
0.06
SEED
0.25
0.10
SEED
0.0
0.57
0.00
005
SEED-V
007
0.00
0.55
-0.05
-0.10
-010
Grattt--mapp
Dese
imapory
Des0
Figure S2: Average silhouette score between features of different datasets.
24

---

E,5 Feature visualization
We quantize the clustering effect between any two datasets before and after pre-training. We regard samples from one dataset as a cluster and calculated the silhouette score of different clusters. The silhouette score decreases for most dataset pairs, indicating a more blended representation across datasets (Fig. (52).
To illustrate the effect of joint pre-training on emotion discriminability of the data representations, we visualized the distribution of EEG features each dataset using the t-SNE method (Fig. (3). We compared EEG representations extracted by mdJPT and the DE features. Different colors represent different emotion categories. Although emotion labels are not used pre-training, features extracted by mdJPT exhibit higher degree of clustering within the same emotion category. This demonstrates the effectiveness of supervise learning in facilitating downstream emotion classification tasks. Notably, the model used for feature extraction on specific dataset is pre-trained on all other datasets, which never accesses data from the target dataset.
E.6 Inter-dataset consistency of emotion-related features
To validate whether our joint multi-dataset pre-training approach successfully learns emotion repre- sentations with cross-dataset invariance, we conducted comparative analyses of feature importance in emotion classification across different datasets. Specifically, we employed the Integrated Gradi- ents method quantify the contribution of EEG encoder output features emotion classification decisions. This interpretability technique enables the identification of critical feature dimensions un- derlying the classification of each emotion category. Next, we compared the similarity of importance attributions across different datasets to identify whether the same emotion category across different datasets shares similar importance attributions. Fig. S4 illustrates the importance attributions of neutral, negative/sad, and positive/happy emotions on SEED, SEED-IV, SEED-V, and SEED-V VII. We sorted the attributions of each emotion the SEED dataset in descending order. The resulting sorted indices were then applied to reorder the attributions of the corresponding emotion categories in other datasets. As shown in the figure, similar attribution patterns can be observed across the same emotion categories in different datasets.
To more clearly illustrate this correlation, we computed the Pearson correlation of emotion attributions for any emotion pairs across datasets (Fig. (S)). The attributes of identical or similar emotions across different datasets exhibit stronger correlations than different emotions. For example, feature attributes of the positive emotion on the SEED dataset exhibit a high correlation with those of the happy emotion on the SEED-IV datsaet, and attributes of the negative emotion on the SEED dataset have a high correlation with those of the sad and fear emotions on the SEED-IV dataset.
E.7 Transfer from DEAP to other datasets
To further test whether the model effectively transfer across datasets with different emotion categorization paradigms, we trained our model on DEAP and transferred it to SEED-series and FACED datasets. DEAP dataset employs dimensional emotion characterization and other datasets employ discrete emotion characterization. We found that transfer from DEAP to other dataset yielded comparable performance that of transfer from SEED dataset (Table SS). The performance is only sligntly lower than from SEED on SEED series datasets, and is even higher on FACED dataset. This indicates the model's capability in generalization new emotion categorization paradigms.
Table S7: Cross-dataset transfer performance between DEAP and SEED-series datasets
Accuracy
from DEAP
from SEED
to SEED
66.77 + 3.33
SEED-IV
46.24 + 0.87
48.52
+ 1.65
to SEED-V
54.55 + 2.13
55.85
+ 2,31
SEED-VII
36.35 + 1.04
37.33
= 1.19
to FACED
22.83 + 1.48
18.95
= 1.79
25

---

E,8 Other classification settings on the DEAP dataset
To comprehensively evaluate the model on dimensional emotion representation, we conducted binary classifications of high/low arousal and high/low dominance, as well as a four-quadrant valence-arousal classification the DEAP dataset. As shown in Table [S8] the model substantially outperformed the baseline across all tasks, demonstrating strong generalization to dimensional emotion representations and adaptability to different affective dimensions.
DE
mdJPT
tegatre
SEED
Hedral Sad
SEEDIV
SEEDV
SEEDVII
fochi
DEAP
Figure S3: t-SNE visualization of extracted feature SEED, SEED-IV, SEED V, SEED-VIL and DEAP datasets. The EEG encoder used for feature extraction was trained on all datasets other than the target dataset.
26

---

Neutral
Negative
Positive
Ingurta
040
Importance
SEED
6.0
-0.0
-0A4
200
240
400
230
406
Neutral
Sad
Happy
porttance
0.04
inporrance
0.50
0.40
0.04
SEED-IV
0.00
0.00
Dim
Dim
Dim
Happy
0:53
Importance
Inportance
0:59
SEED-V
0.00
10000
0.00
2:00
Neutral
Sad
Happy
0.003
0.000
0.03
aurda
0.05
ancc
6:0
SEED-VII
8.4
4.49
Dim
Dim
Dim
Figure $4: Comparison of feature importance attribution for neutral, negative/sad, and positive/happy emotions on SEED, SEED-IV, SEED-V, V, and SEED-VI datasets. The feature indices are sorted by the feature importance on the SEED dataset.
E.9 Results varying random seeds
We replicated the full pipeline five times with different random seeds. The performance is quite stable across the five repetitions, with variation of less than 0.85% (Table S9). The experiment is conducted on the SEED-V dataset. This result demonstrates the stability of mdJPT training.
Table S8: Other classification settings on the DEAP dataset
Evaluation
Method
Accuracy
Precision
Recall
F1 Score
AUROC
DE baseline
58.67=1.277
59.00 69.60
58.67=11.27
58.36
60.67
Arousal
mdJPT
69.36:1.54
69.36:11.54
69.27
74.19
DE baseline mdJPT
55.01 1.17
15.29:1.447
55.01=11.17
54.56:10.2
56.08
Dominance
73.52-11.61
73.6411147
73.52:11.61
73.48
79.47+1.50
DE baseline
34.03+11.06
33.2112.84
32.5941.21
30.04
57.18+1.022
Valence-arousal
mdJPT
52.9411.42
50.47-11.23
51.10-11.83
49.68:12.90
75.78
Table S9: SEED-V result with varying random seeds
Random seed
Accuracy
0
65.14 + 0.97
1
65.87 + 1.08
2
65.74 + 1.50
3
65.37 + 1.52
19260832
65.02 + 0.98
27

---

SEED-IV
SEED-V
SEED-VII
Positive
0.1
SEED
Neutral
0.0
Negative
-0.25
-0.2
Happy
Neutral
Fear
Happy
Neutral
Fear
Disqust
Hаppy
Sad
Fear
Disgust
Surprise
Anger
SEED-V
SEED-VII
Happy
0.2
SEED-V
Neutral
0.0
0.0
Fear
-0.2
Happy
Neutral
Sad
Fear
Disgust
Happy
Neutral
Fear
Disqust
Surprise
Anger
SEED-VII
Hapрy
SEED-V
Neutral
0.2
Sad
0.0
Fear
-0.
Disgust
Happy
Neutral
Sed
Fear
Disgust
Anger
Figure S5: Pearson correlation coefficient between emotion-relatee feature importance weights on different datasets
E.10
Model size comparison
To demonstrate the efficiency of our model, we compared the number of trainable parameters with several existing methods, As shown in Table S10 our model has fewer parameters of 1.0M but still outperforms state-of-the-art models, indicating more compact and efficient design.
Table S10: Comparison of parameter size across pre-trained models.
Model
Parameters
MMM
1.2M
LaBraM
5.8M
EEGPT
4.7M
mdJPT (Ours)
1.0M
28


Reviews:
[{"review_id": "P9y2LUSrQ5", "reviewer": "Reviewer_BNoM", "summary": "The authors propose mdJPT (Multi‑dataset Joint Pre‑Training), a task‑specific framework for EEG‑based emotion recognition. The core Idea is to jointly pre‑train a hybrid EEG encoder on several emotion datasets, while aligning second‑order statistics (Cross‑Dataset Alignment, CDA) and inter‑subject representations (Inter‑Subject Alignment, ISA). The architecture employs a hybrid encoder combining a lightweight Mamba-style linear attention module and a spatiotemporal dynamics block. Evaluation across six public EEG datasets using a leave-one-dataset-out protocol shows that mdJPT significantly outperforms both traditional models and recent EEG foundation models, improving average F1-scores by 6.6 percentage points.", "questions": "1) It is unclear how much labeled data from the held-out (target) dataset is used for fine-tuning. The paper mentions \"fine-tuning\" but does not specify: How many subjects or trials per subject are used?\n2) Whether experiments were repeated over multiple random seeds. Whether splits were consistent across models.\n3) The paper does not specify whether the same EEG preprocessing was used for the baseline.\n4) Why does the baseline not include any task-specific cross-dataset model? The authors should add them or discuss why these baselines were chosen.\n5) Why is the EEG dataset from other elicitation methods not used for experiments? Please discuss.\n6) The authors should discuss the dataset results, and why the results vary so much.", "limitations": "Yes, the paper has a limitation section, but the paper does not explicitly address the potential negative societal impacts of EEG-based emotion recognition. Further, the paper would benefit from addressing the lack of diversity in the datasets (age, cultural background, mental health status), which limits generalisability to real-world, heterogeneous populations. The authors should also consider discussing risks such as misuse in surveillance or workplace monitoring, overinterpretation of emotional states, and the ethical concerns around applying such models without informed consent. Also, given EEG hardware is bulky, their real-world applicability would be limited; this should also be added.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strength:\n1) Solves the problem of data scarcity by focusing on the pre-tarining with multiple datasets. Moreover, it combines task‑specific multi‑dataset pre‑training with covariance‑centric alignment; prior EEG foundation models focus on task‑agnostic breadth. \n2) The paper addresses a key challenge in EEG-based emotion recognition: poor cross-dataset generalisation. By pretraining across multiple datasets with targeted alignment strategies, mdJPT offers a meaningful step forward for real-world deployment also since combining dataset adds to data diversity in training.\n3) The integration of second-order statistical alignment (CDA) and subject-level contrastive learning (ISA) is interesting.\n4) The paper is clearly written, with well-structured sections.\n\nWeakness:\n1) The assumption that EEG emotion datasets are misaligned is taken from prior knowledge, but the paper does not directly test or visualise this. I think including visual evidence would make the motivation for their alignment techniques much more compelling.\n2) All datasets are video-evoked emotion EEG data collected in lab-settings, no dataset from other elicitation technique is used like, semi-naturalistic tasks (public speaking, debate, memory recall or others)\n3) Author should discuss the lower accuracy and poor result on dataset like FACED - 23%.\n4) While subject-independent testing is performed, the method still assumes access to labeled data from some subjects in the target dataset for fine-tuning. No zero-shot results are presented.\n5) Exact subject split ratios are missing (e.g., how many subjects used for fine-tuning vs. test?). The number of trials per subject used for training vs. testing is unclear.\n6) The number of random seeds or runs per dataset is not stated clearly, though metrics are reported as mean ± SD.", "was_revised": true, "final_justification": "The authors have addressed most points well in the rebuttal, and I appreciate their detailed response. However, I will maintain my original score, as the work would benefit from a deeper discussion on the intuition of the mdJPT method and results at the data level, particularly to support generalizability claims.", "rebuttal": "# Global response\n\nWe sincerely thank the reviewers for the thoughtful and constructive feedback. We are encouraged by the recognition of our framework's significance and competitive performance. At the same time, we acknowledge the concerns regarding our motivation, methodology, and clarity, and we are committed to addressing all points thoroughly in our revision. We also provide additional experiments and explanations to address reviewers' concerns:\n\n1. The performance of zero-shot generalization to unseen datasets. (Reviewer **BNoM**, **CDLA**, and **VGNY**)\n\n2. The performance on emotion-elicitation protocols other than video-elicitation. (Reviewer **BNoM**)\n\n3. Evaluation on more affective dimensions on the DEAP dataset and the transfer from DEAP to other datasets. (Reviewer **VGNY** and **YfAh**)\n\n4. Comparison with alternative methods, including supervised contrastive learning and MMD loss. (Reviewer **CDLA** and **YfAh**)\n\n5. The motivation behind the design of CDA loss and MLLA encoder. (Reviewer **YfAh**)\n\nBelow, we respond to each reviewer in detail.\n\n## Response to reviewer BNoM\n\nWe sincerely thank you for the thoughtful and constructive review. We are glad you found the problem setting and our multi‑dataset pre-training framework meaningful. Your comments directly helped us improve the paper’s clarity, rigor, and scope. Following the suggestions, we added a **zero‑shot** evaluation with no fine‑tuning on the target dataset and included results on the `EmoEEG‑MC imagery‑induced dataset` to test generalization **beyond video stimuli**. We add analyses on dataset misalignment and the effect of random seeds, clarify method details, and add the discussion of results and limitations.\n\n### Q1. Visualization of dataset misalignment.\n\nWe visualize t‑SNE embeddings of differential entropy (DE) features extracted from EEG signals across datasets. Data from the same dataset cluster together, while different datasets are separated. This demonstrates large cross‑dataset misalignment in EEG features. We will add the figure in the revised version.\n\n---\n\n### Q2. No dataset from elicitation techniques other than video-evoking is used, like semi-naturalistic tasks.\n\nMost widely used EEG emotion datasets are video‑induced, which emphasizes controlled, video‑elicited designs. To assess generalizability beyond video stimuli, we add experiments on `EmoEEG‑MC` [1], a multi‑context dataset published in 2025 that includes an imagery‑induced paradigm (guided narratives with active imagination), eliciting more internally driven and sustained emotions.\n\nWe evaluate on the imagery task using the same protocol: pretrain on six video‑induced datasets, then fine‑tune on 1/4 of EmoEEG‑MC subjects and test on the remaining.\n\n**Table: Comparison of different models on the EmoEEG-MC dataset with imagery-induced emotion.**\n\n| Method      | Accuracy | Precision | Recall | F1 Score | AUROC |\n| ----------- | -------- | --------- | ------ | -------- | ----- |\n| DE baseline |18.38±1.12|17.91±1.62|18.38±1.12|17.87±1.37|54.11±1.36|\n| mdJPT       |20.56±0.79|20.83±0.66|20.56±0.79|19.85±2.11|56.14±0.78|\n\nOur method outperforms other baselines in the imagery-induced setting. Comparison with other models will be provided in the discussion session. Transfer from video to imagery remains relatively low, likely due to scenario discrepancies. Future work needs to pretrain on more diverse and naturalistic tasks.\n\n\\[1] Xu X, Shen X, Chen X, et al. *A Multi‑Context Emotional EEG Dataset for Cross‑Context Emotion Decoding*. Scientific Data, 2025, 12(1): 1142.\n\n---\n\n### Q3. Authors should discuss the lower accuracy on datasets like FACED. Why do the results vary so much?\n\nPerformance varies for several reasons. First, label space differs: DEAP (2 categories of high/low valence), SEED (3 categories), SEED‑IV (4 categories), SEED‑V (5 categories), SEED‑VII (7 categories), and FACED (9 fine‑grained emotions, with more positive categories). Several positive emotions in FACED (joy, amusement, inspiration, tenderness) may be subtle in neural differences and harder to separate. Second, annotation standards differ: DEAP uses continuous affective ratings, while others use discrete labels, affecting uncertainty and separability.\n\nSensor setup and signal quality also matter. SEED uses 62 channels; DEAP and FACED use 32, reducing spatial resolution. During preprocessing, SEED‑IV showed more noise/artifacts (higher channel variance, more disruptions). A t‑SNE visualization places SEED‑IV farther from other datasets, indicating a bigger domain gap, which may explain its relatively lower performance.\n\n---\n\n### Q4. The issue of fine-tuning and the evaluation of zero-shot performance.\n\nTo evaluate our method on real zero-shot generalization tasks, we further conducted an experiment to directly apply the pre-trained model to a new dataset without fine-tuning. We compute cosine similarity across samples and define zero‑shot accuracy by whether the top‑similar pair shares the same label. Samples from the same trial are excluded to avoid leakage. The results with SEED-V as the target dataset are shown below.\n\n**Table: Comparison of different models under zero-shot settings.**\n\n|Top1 accuracy|  SEED-V| FACED | DEAP | \n| ----------- | ------|---|---|\n| DE baseline | 43.3  |8.9|55.4|\n| mdJPT       | 52.9  |17.4|73.3|\n\nmdJPT outperforms DE baseline in the zero‑shot setting. It improves by 9.6% over the DE baseline on the SEED-V dataset. On the FACED and DEAP datasets, mdJPT also outperforms the DE baseline by 8.5% and 17.9%, respectively. We will add the results of other comparison models in the discussion session.\n\nTo our knowledge, this is the first zero‑shot cross‑dataset EEG emotion experiment. Prior work typically fine‑tunes per subject. These results highlight the robustness and generalizability of our framework.\n\n---\n\n### Q5. Exact subject split ratios for fine-tuning vs. test.\n\nIn lines 185–187: “We split the target dataset at a 1:3 subject ratio for fine‑tuning and testing. A leave‑one‑dataset‑out cross‑validation is employed. Our method is pre-trained on all datasets except the target.” Concretely, 25% of target subjects are randomly selected for fine‑tuning and 75% for testing. We repeat the random split six times and report mean ± SD. We will clarify this in Section 3.1.\n\n---\n\n### Q6.1 The number of random seeds or runs per dataset.\n\nFor fine‑tuning/testing, we use six random splits (as in Q5) and report mean ± SD. During pre-training, the seed (19260832) was randomly chosen without cherry‑picking. We will replicate the full pipeline multiple times with different random seeds and report their consistency in the discussion session.\n\n---\n\n### Q6.2 Whether splits were consistent across models?\n\nYes. We strictly use the same fine‑tuning/test splits and the same seeds for all models.\n\n---\n\n### Q7. Whether the same EEG preprocessing was used for the baseline?\n\nYes. All methods share the same standard EEG preprocessing. Details are in Appendix C.\n\n---\n\n### Q8. Why does the baseline not include any task-specific cross-dataset model?\n\nOur baselines match mdJPT’s goals: (i) multi‑dataset joint pretraining, (ii) generalization to unseen categories, and (iii) subject‑independent evaluation (apply to new subjects directly). Most cross‑dataset methods are one‑to‑one adaptations, assuming the same label space and requiring target data—often per‑subject—for training/fine‑tuning (e.g., adversarial alignment). As summarized in Table 1, these pipelines do not simultaneously handle new categories and new subjects, the core challenge addressed by our framework.\n\nFor fair comparison, we use foundation‑model baselines, pretrained on multiple large datasets and evaluated on new ones, under the same lightweight classifier protocol. We will clarify this rationale in the revision.\n\n---\n\n### Q9. Limitations of generalizability, negative societal effects, and the real-world applicability.\n\nWe will add the following paragraphs to the Discussion:\n\n>“Another limitation is the lack of diversity in current EEG emotion datasets—participants are often from limited age groups (generally young people), cultural backgrounds (Probably British for DEAP and Chinese for other datasets), and mental health status (healthy), limiting generalizability to heterogeneous populations. Real‑world deployment also remains constrained by the practicality of current EEG hardware, which is often cumbersome. The ongoing development of lightweight, wearable EEG is expected to enable more practical applications.\n\n>We also acknowledge societal and ethical risks, including potential misuse in surveillance, workplace monitoring, and unauthorized emotional profiling. Such uses may lead to overinterpretation or coercion without informed consent. As the field advances, it is essential to establish clear regulatory oversight and develop privacy‑preserving frameworks.”\n\n\n\n### Q10. Some figures lack detailed captions explaining all subcomponents or labels.\n\nWe will revise the figure captions to include more details. For example, in Figure 2, we revise the caption of sub-figure A as: \"A) The overall framework. Our framework includes three stages: multi-dataset joint pre-training, classifier fine-tuning, and testing. The EEG encoder contains an MLLA channel encoder and a spatiotemporal dynamics model. During pre-training, the encoder is optimized with a combination of CDA and inter-subject alignment (ISA) losses. During fine-tuning, the encoder is frozen, and a lightweight classifier is trained on a small subset of subjects from the target dataset. Evaluation is then performed on the remaining (held-out) subjects."}, {"review_id": "L1SOtVd7FV", "reviewer": "Reviewer_CDLA", "summary": "This paper presents a novel multi-dataset joint pre-training framework (mdJPT) for EEG-based emotion recognition, incorporating a Cross-Dataset covariance Alignment (CDA) loss and a hybrid spatiotemporal encoder. Despite demonstrating a 6.59% average F1-score improvement over existing EEG foundation models in cross-dataset generalization, several limitations need to be addressed to enhance the impact and reproducibility of this work.\n\n1. The claim of \"zero-shot generalization to new emotion categories\" in Section 2.3 is not supported by the experimental results. The framework requires fine-tuning with 25% of target subjects, as described in Section 3.3, which contradicts the definition of true zero-shot learning. This limitation significantly undermines the applicability of the framework in scenarios where labeled data is scarce.\n\n2. Residual feature distribution differences persist despite the application of CDA, as noted in Section 4. The approach struggles to handle extreme heterogeneity, such as the difference between FACED’s 9-class emotions and SEED’s 3 classes. This is evidenced by the relatively low F1-score of 20.32% on FACED, as shown in Table 3. The reliance of CDA solely on covariance may overlook higher-order distribution shifts.\n\n3. The tuning of the CDA weight is critical but inadequately explained in Section 3.5. Performance varies non-monotonically, as observed in Table 5, where the F1-score decreases from 65.02% to 64.08% when the weight increases from 0.02 to 0.1. This lack of clear guidelines for selecting the weight parameter λ in Eq. 8 poses a risk to the reproducibility of the results.", "questions": "1. To evaluate the model's generalization ability, it is necessary to test zero-shot transfer learning by keeping the encoder frozen and directly assessing performance on unseen datasets. Additionally, previous claims about generalization should be reformulated to reflect the current dependency on fine-tuning, ensuring that conclusions align with reality.\n\n2. Efforts should be made to enhance distribution alignment. This can be achieved by expanding the current CDA (Contrastive Distributional Alignment) to include higher-order statistics such as maximum mean discrepancy and incorporating domain-adversarial training strategies. To verify the effectiveness of feature distribution alignment, visualization techniques like t-SNE plots should be used to display the feature distributions before and after alignment.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The CDA loss effectively mitigates inter-dataset distribution shifts by aligning second-order statistical properties without requiring labels or subject-specific calibration (Section 2.3). This addresses a critical gap in EEG emotion recognition.\n\n2.The hybrid encoder (Mamba-like linear attention + spatiotemporal dynamics) successfully captures long-term dependencies and complex EEG dynamics, outperforming vanilla transformers by >2% F1-score (Table 4).\n\nWeaknesses:\n1. The claim of \"zero-shot generalization to new emotion categories\" in Section 2.3 is not supported by the experimental results. The framework requires fine-tuning with 25% of target subjects, as described in Section 3.3, which contradicts the definition of true zero-shot learning. This limitation significantly undermines the applicability of the framework in scenarios where labeled data is scarce.\n\n2. Residual feature distribution differences persist despite the application of CDA, as noted in Section 4. The approach struggles to handle extreme heterogeneity, such as the difference between FACED’s 9-class emotions and SEED’s 3 classes. This is evidenced by the relatively low F1-score of 20.32% on FACED, as shown in Table 3. The reliance of CDA solely on covariance may overlook higher-order distribution shifts.\n\n3. The tuning of the CDA weight is critical but inadequately explained in Section 3.5. Performance varies non-monotonically, as observed in Table 5, where the F1-score decreases from 65.02% to 64.08% when the weight increases from 0.02 to 0.1. This lack of clear guidelines for selecting the weight parameter λ in Eq. 8 poses a risk to the reproducibility of the results.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the thoughtful review and for recognizing our performance gains. In the revision, we add true **zero‑shot** evaluations on unseen datasets, where mdJPT outperforms all foundation‑model baselines. We also assess higher‑order alignment via MK‑MMD (Gaussian, multi‑kernel) and find CDA remains competitive while being simpler to optimize. We further include t‑SNE and clustering‑coefficient analyses to demonstrate the pre-training effect. We appreciate your suggestions—they substantially improved the clarity and completeness of the work.\n\n### Q1. The limitation of fine-tuning on the target dataset.\n\nTo evaluate our method on real zero-shot generalization tasks, we further conducted an experiment to directly apply the pre-trained model to a new dataset without fine-tuning. We compute cosine similarity across samples and define zero‑shot accuracy by whether the top‑similar pair shares the same label. Samples from the same trial are excluded to avoid leakage. The results with SEED-V as the target dataset are shown below.\n\n**Table: Comparison of different models under zero-shot settings.**\n\n|Top1 accuracy|  SEED-V| FACED | DEAP | \n| ----------- | ------|---|---|\n| DE baseline | 43.3  | 8.9 | 55.4|\n| mdJPT       | 52.9  |17.4 | 73.3|\n\nmdJPT outperforms DE baseline in the zero‑shot setting. It improves by 9.6% over the DE baseline on the SEED-V dataset. On the FACED and DEAP datasets, mdJPT also outperforms the DE baseline by 8.5% and 17.9%, respectively. We will add the results of other comparison models in the discussion session.\n\nWe clarify the original evaluation protocol as \"few-subject fine-tuning\", and the new protocol as \"zero-shot generalization\".\n\nTo our knowledge, this is the first zero‑shot cross‑dataset EEG emotion experiment. Prior work typically fine‑tunes per subject. These results highlight the robustness and generalizability of our framework.\n\n### Q2.1 Residual feature distribution differences persist despite the application of CDA. The reliance of CDA solely on covariance may overlook higher-order distribution shifts.\n\nWe understand the reviewer's concern about the possibility of higher-order distribution shifts. We agree that relying only on covariance may miss higher‑order distribution differences. Indeed, CDA loss aligns global second-order statistics (i.e., inter-channel covariance structures) across subjects and datasets. This captures **inter-region coupling** (or functional connectivity) patterns in EEG—features that have been increasingly emphasized in recent neuroscience and EEG decoding literature [1-3]. \n\nAlthough CDA can align EEG with the second-order relations, residual distribution shifts may persist, especially under extreme heterogeneity. To test the impact of higher-order distribution, we add an experiment with **higher-order alignment** using Multiple Kernel Maximum Mean Discrepancy loss (MK-MMD, Gaussian kernel). The MMD loss with a Gaussian kernel includes all moments of the distribution. It implicitly maps inputs into an infinite-dimensional reproducing kernel Hilbert space (RKHS), where the distance between distributions captures differences across all orders of moments. Multiple kernels (five here) are introduced to reduce the reliance on the Gaussian kernel length. All other settings are kept the same as our models. The results with SEED-V as the target dataset are shown below:\n\n**Table: Comparison with MK-MMD loss.**\n\n| Method          | Accuracy | Precision | Recall | F1 Score | AUROC |\n| --------------- | -------- | --------- | ------ | -------- | ----- |\n| MK-MMD loss     |64.33±1.24|64.52±1.86|64.77±1.22|64.00±1.40|87.73±0.89|\n| CDA loss (ours) |**65.02±0.98**|**65.06±1.28**|**65.53±0.66**|**64.85±1.01**|**88.70±0.98**|\n\nCDA slightly outperforms MK-MMD in our setting, suggesting that aligning second-order statistics already captures much of the cross-dataset shift in EEG (e.g., montage/linear mixing). Higher-order criteria like MK-MMD can, in principle, model richer differences, but in our experiments they did not yield clear gains. The residual discrepancies likely arise from shifts not easily addressed by distribution matching alone (e.g., elicitation protocol mismatch or pronounced nonstationarity), so even higher-order matching may not fully close the gap. Besides, kernel-based higher-order alignment adds tuning complexity and can be harder to optimize stably. We will explore complementary solutions in future work.\n\n### Q2.2 The effectiveness of feature distribution alignment should be visualized.\n\nIn **Appendix D.2**, we visualized the latent representations on each target dataset. Although emotion labels were not used in pre-training, the features extracted by pre-trained mdJPT are visually more separable than the original DE features, especially on datasets with fewer emotions, like SEED. This demonstrates the effectiveness of pre-training in facilitating downstream emotion recognition tasks. In the revised version, we will provide the t-sne visualization of all datasets before and after pre-training, to investigate whether pre-training can effectively mitigate the data discrepancy across datasets.\n\n### Q3. The tuning of the CDA weight not inadequately explained in Section 3.5.\n\nWe agree that performance varies with λ and is not strictly monotonic. In **Table 5**, the drop at λ=0.01 coincides with a larger STD, which could be attributed to occasional training instability when ISA and CDA losses exert partially opposing gradients at this λ. In the revision, we will run a denser sweep over different λs (λ∈{0, 0.001, 0.002, 0.005, 0.0075, 0.010, 0.015, ...}) and repeat each point over multiple random seeds, providing a clear, reproducible guideline for selecting λ."}, {"review_id": "oZqUO06tBF", "reviewer": "Reviewer_VGNY", "summary": "This paper presents a task-specific multi-dataset pre-training framework named mdJPT for EEG-based emotion recognition. To address challenges such as distribution shifts and label inconsistencies across datasets, the authors propose a Cross-Dataset Covariance Alignment (CDA) loss, which aligns data distributions without requiring extensive labels. In addition, they design a hybrid spatiotemporal encoder to enhance EEG signal modeling. The proposed framework improves accuracy by 6.59% over previous approaches and demonstrates strong generalization across datasets.", "questions": "1. The paper mentions that fine-tuning with a few target-domain subjects is needed. Has the model been evaluated in zero-shot settings? If so, what is the performance gap, and what might this suggest about potential bottlenecks in the current method?\n\n2. Why is valence chosen as the prediction target for DEAP, rather than arousal, dominance, or a combination? Have any experiments been conducted on the other dimensions?\n\n3. The authors state that training is conducted for 20 epochs. Could this lead to overfitting, especially with limited labeled data? No training or validation loss curves are provided—could these be added?\n\n4. While mdJPT performs well on SEED, SEED-V, and SEED-VII, its performance drops noticeably on SEEG-IV. What factors might explain this discrepancy? How is SEEG-IV different from the other datasets?\n\n5. Besides the CDA loss weight λ, are there other important hyperparameters in the model? How were these values selected? Was a hyperparameter search conducted?\n\n6. The authors are encouraged to make full use of the 9-page limit to elaborate on their methodology and findings. In particular, presenting concrete case studies or examples would enhance the reader’s understanding of how the model performs across varied datasets.\n\n7. The Related Work section would benefit from incorporating recent literature on EEG foundation models and physiological signal learning, such as Brant-2, Brant-X, and BrainBERT.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- The paper is technically sound and clearly presents the mdJPT framework, along with key contributions including the CDA loss, a spatiotemporal encoder, and the multi-dataset pre-training strategy.\n\n- The authors conduct thorough evaluations across multiple public EEG datasets, using cross-validation and comparison with competitive baselines. Ablation studies are also included to analyze the contributions of the ISA Loss, CDA Loss, and MLLA encoder.\n\n- Code is made publicly available through an anonymous repository, which supports the reproducibility of the work.\n\n\n### Weaknesses\n\n- The mdJPT framework includes several components (e.g., MLLA encoder, spatiotemporal dynamics model, CDA loss), which may lead to increased model complexity, making model implementation and deployment more challenging.\n\n- Although the framework is designed to improve generalization, it still requires a small number of subjects from the target dataset for fine-tuning, limiting its potential in zero-shot settings.\n\n- The paper does not fully utilize the 9-page limit. There is room to provide more in-depth explanations or case studies, especially illustrating how the model generalizes across different datasets, which would strengthen the overall contribution.\n\n- The Related Work section omits several relevant EEG and physiological signal foundation models. Important works such as Brant-2 [1], Brant-X [2], and BrainBERT [3] are highly relevant and should be discussed.\n\n> [1] Brant-2: Foundation model for brain signals. arXiv:2402.10251\n> \n> [2] Brant-X: A Unified Physiological Signal Alignment Framework. KDD 2024\n> \n> [3] BrainBERT: Self-supervised representation learning for intracranial recordings. arXiv:2302.14367", "was_revised": true, "final_justification": "The main issues were solved. I think after revision, this work can reach the bottom line of acceptance, but I suggest that the AC consider this article comprehensively and combine the opinions of other reviewers to consider the decision on this article. Thank you!", "rebuttal": "We sincerely thank you for your suggestions to make the results more convincing. In revision, we (i) add **zero‑shot** evaluations on unseen datasets, where mdJPT outperforms all foundation‑model baselines and broaden evidence with DEAP arousal, dominance and VA‑quadrant prediction. Interpretability results, loss curves and hyperparameter evaluations are further added. The related work and the discussion about the results are revised. These updates more convincingly, and more comprehensively, validate the model’s generalization across datasets and label regimes.\n\n### Q1. The issue of model complexity due to multiple components.\n\nAlthough our model integrates multiple components, we have paid careful attention to efficiency during design. The total number of parameters remains relatively small (1.0M) compared to existing EEG foundation models (e.g., LaBram: 5.8M, EEGPT: 4.7M), as shown in Table S2. In addition, our architecture achieves faster inference speed, owing to the parameter-efficient designs of the MLLA encoder and spatiotemporal dynamics module. We believe this strikes a favorable balance between architectural expressiveness and practical usability.\n\n### Q2. The issue of fine-tuning on the target dataset and the evaluation in zero-shot settings.\n\nTo evaluate our method on real zero-shot generalization tasks, we further conducted an experiment to directly apply the pre-trained model to a new dataset without fine-tuning. We compute cosine similarity across samples and define zero‑shot accuracy by whether the top‑similar pair shares the same label. Samples from the same trial are excluded to avoid leakage. The results with SEED-V as the target dataset are shown below.\n\n**Table: Comparison of different models under zero-shot settings.**\n\n|Top1 accuracy|  SEED-V| FACED | DEAP | \n| ----------- | ------|---|---|\n| DE baseline | 43.3  |8.9|55.4|\n| mdJPT       | 52.9  |17.4|73.3|\n\nmdJPT outperforms DE baseline in the zero‑shot setting. It improves by 9.6% over the DE baseline on the SEED-V dataset. On the FACED and DEAP datasets, mdJPT also outperforms the DE baseline by 8.5% and 17.9%, respectively. We will add the results of other comparison models in the discussion session.\n\nTo our knowledge, this is the first zero‑shot cross‑dataset EEG emotion experiment. Prior work typically fine‑tunes per subject. These results highlight the robustness and generalizability of our framework.\n\n### Q3. Present concrete case studies or examples of how the model performs across varied datasets\n\nWe identify the most critical features for emotion classification on multiple datasets and visualize its related temporal patterns and spatial transition patterns through the integrated gradient method. The feature importances for an emotion category are correlated between different datasets (as shown in Fig. S3), indicating a shared feature space is learned across datasets. We will further include the visualization of temporal and spatial patterns of critical features in the revised version.\n\n### Q4. Introduce important EEG foundation models in Related Work.\n\nTo present a more comprehensive review of the current foundation models, we add the introduction of BrainBERT, Brant, Brant-X, Brant-2, and CBraMod in the Related Work.\n\n> \"BrainBERT implemented a masked auto-encoder on SEEG time frequency representations and improved the performance of speech decoding.\"\n\n> \"Brant-X aligns EEG with EXG (EOG/ECG/EMG) signals through contrastive learning. It pulls the representations of simultaneous EEG and EXG patches together and pushes apart those of different time segments. Brant-2 combines EEG and SEEG data from more than 15k subjects, uses temporal and frequency features for mask-prediction and forecasting in pre-training, and demonstrates its strength in downstream tasks of seizure detection/prediction, sleep stage classification, motor movement/imagery, and emotion recognition. CBraMod designed a criss-cross attention mechanism to process temporal and spatial information separately. It used more than 27,000 hours of EEG data for pre-training and was evaluated on 10 downstream tasks. These methods demonstrate the current trend of merging larger datasets for pre-training and combining the information from multiple modalities.\"\n\n### Q5. The prediction of dimensions other than valence on the DEAP dataset.\n\nIn the previous submission, we prioritized a compact cross‑dataset analysis and therefore focused on valence, the dimension most commonly evaluated in previous EEG emotion studies. To evaluate the model more comprehensively,  we now add the binary prediction of high/low arousal and high/low dominance, and a four-category classification of the valence-arousal quadrants. Our model outperforms the baseline model by a large margin, demonstrating the generalization ability of the pre-trained model to new emotion rating paradigms (emotion dimensions rather than discrete categories) and its adaptability across different affective dimensions. \n\n| Evaluation          | Method      | Accuracy | Precision | Recall | F1 Score | AUROC |\n| ------------------- | ----------- | -------- | --------- | ------ | -------- | ----- |\n| **Arousal**         | DE baseline |58.67±1.27|59.00±1.60|58.67±1.27|58.36±1.21|60.67±1.97|\n|                     | mdJPT       |69.36±1.54|69.60±1.68|69.36±1.54|69.27±1.52|74.19±0.92|\n| **Dominance**       | DE baseline |55.01 ± 1.17|55.29±1.47|55.01±1.17|54.56±0.92|56.08±1.89|\n|                     | mdJPT       |73.52 ± 1.61|73.64±1.47|73.52±1.61|73.48±1.67|79.47±1.50|\n| **Valence-arousal** | DE baseline |34.03±1.06|33.81±2.84|32.59±1.21|30.04 ± 3.18|57.18 ± 1.02|\n|                     | mdJPT       |52.94±1.42|50.47±1.23|51.10±1.83|49.68±2.90|75.78 ± 1.34|\n\n### Q6. Whether training for 20 epochs leads to overfitting? Can the loss curves be added?\n\nSpecifically, we pre-train the model for 20 epochs. As the data size in pre-training is relatively large, this does not lead to overfitting. The training and validation loss generally converges after 20 epochs. In fine-tuning, the classifier is trained with early stopping. The loss curves generally converge in a few epochs and exhibit small fluctuations since then. We will also add the loss curves for pre-training and fine-tuning in the revised version.\n\n### Q7. How to explain the performance drop on the SEED-IV dataset?\n\nTo better understand the relatively lower performance on the SEED-IV dataset, we conducted a t-SNE visualization of the learned feature representations and observed that SEED-IV samples are more distant from those of other datasets in the latent space, suggesting a larger domain gap. During preprocessing, we noticed that SEED-IV recordings tend to contain more signal fluctuations and potential artifacts, such as higher variance in channel recordings and frequent noise spikes.\n\nThese differences may partially account for the performance drop on this dataset. Compared to methods such as MMM, which first extract and smooth features before model training, mdJPT directly learns from EEG time series. This design prioritizes the preservation of fine-grained temporal information but may also make the model more exposed to noise in certain cases. Nevertheless, on datasets other than SEED-IV, mdJPT generally achieves stronger results than MMM, highlighting the benefits of learning directly from EEG time series. Furthermore, mdJPT consistently outperforms other foundation models that also operate on EEG time series (e.g., LaBram, EEGPT) on SEED-IV, demonstrating a degree of robustness even under challenging recording conditions.\n\n### Q8. Evaluation of other model hyperparameters.\n\nHyperparameters were primarily determined through established heuristics, with targeted searches conducted for several pivotal parameters. Beyond CDA weight, we found several important hyperparameters in preliminary experiments, including MLLA patch stride (which cannot be too large to enable a better extraction of local temporal features) and MLLA output dimension (needs to be at least 32 to achieve a satisfactory performance). We will report the effects of varying these hyperparameters on model performance."}, {"review_id": "IiqTk4s4nG", "reviewer": "Reviewer_YfAh", "summary": "This work presents a framework called mdJPT for EEG-based emotion recognition that can pre-train on multiple datasets by aligning second-order statistical properties. The framework contains three key components: (1) mamba-like linear attention based EEG patchifier and spatialtemporal dynamics model, (2) cross-dataset alignment loss, (3) inter-subject alignment loss. This papers' design can be seperated into two parts: (1) EEG encoder, which has MLLA patchifier and spatialtemporal modeling, (2) domain/representation alignemtn, which has CDA and ISA. The framework is compared with three EEG foundation models on 6 datasets. Notably, it is not a traditional self-supervised framework, but requires data labeled with trials ID, each trial means a single video stimulus.", "questions": "1. In Figure 2. C, the color of these arrows are somewhat close and difficult to differenciate.\n\nTypo:\n1. line 48: there is an 1 outside the brackets.", "limitations": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strength\n\n1. Training on multiple datasets for a specific task is a promising approach in the EEG field, I appreciate this idea.  \n\n2. Validating the method on 6 datasets is a hard work, and I appreciate your effort in trying various combination of pre-training datasets in Fig 3.  \n\n3. Opensourcing the code is very good in EEG-based emotion recognition.\n\n## Weaknesses\n1. The problem you are trying to solve is absolutely meaningful, however, your method seems to be less motivated. You used many techniques in mdJPT, but what's the rational behind these designs? For instance, why do you use mamba-like linear attention, just because it's better than the vanilla Transformer in Table 4? Aligning the covariance of two domain is also not a new idea, but why do you use it, just because a previous work [16]? Have you tried other domain alignment strategies, like mmd loss, or other statistical properties? Please note that I am not asking for more ablation study, but explanation behind all designs. Acutally, I think there is no need to design the EEG encoder for a framework paper, e.g., adding an MLLA encoder is somewhat abrupt and distracts the research focus of the article.\n\n2. The baseline models are self-supervised framework which requires no label of data, while mdJPT requires more information about which trial does each EEG segment belong to. Although I appreciate the design to use trial information as it is a task-specific design, it brings some inconvenience and unfairness when comparing with baselines. Thus, I think it is better to discuss more about this design, rather than MLLA encoder, like comparing with more variants with mdJPT, or with some adpated baselines that can use trial information for pre-training.\n\n3. The design of the data selection for each training batch is interesting, but you didn't discuss about it. I think this part is the most essential for ISA, as the contrastive learning is very relevant to the training batch. So it might be more appropriate to move the part of how to select a training batch (line 118-121) to section 2.4. And it is important to discuss the design of training batch selection with more experiments.\n\n4. The types of emotions studied in different datasets are different but overlap, like SEED's all emotion types are contained in SEED-IV to SEED-VII, but some datasets like DEAP, which uses valence-arousal model to define emotion, is quite different from SEED series, which uses the discrete emotion model. How to transfer the knowledge of DEAP to SEED is an interesting problem too. But in Figure 3, this problem is overlooked. Using trial ID is an interesting approach, but in DEAP, different subjects may give different valence / arousal value to the same stimulus, how did you tackle this question? This is very important, because the design of mdJPT seems to assume each subject will have the same emotion when viewing the same video.\n\nIn conclusion, while I appreciate the framework, the vague motivation behinds the design and experiments that neglect key components make this version is not ready for publication. I hope you can refine it for the next conference / journal.", "was_revised": true, "final_justification": "This paper presents an interesting framework for trial-based EEG emotion recognition task.", "rebuttal": "We sincerely thank you for your insightful suggestions. In the revision, we clarify the design rationale of EEG covariance alignment and dynamics modelling. We add experiments of supervised contrastive learning, batch construction methods (temporal alignment vs. random) and batch-size analyses. Besides, we show DEAP to other datasets transfer obtained comparable performance with SEED to others, supporting cross-rating-paradigm generalization capability of our model.\n\n---\n\n### Q1. The motivation behind the model designs.\n\nOur work aims to develop a unified framework for multi-dataset EEG training and cross-dataset generalization. All components of our design are centered around this objective. Specifically, we introduce two complementary alignment losses: the Covariance Distribution Alignment (CDA) loss and the Inter-Subject Alignment (ISA) loss. The CDA loss aligns global second-order statistics (i.e., inter-channel covariance structures) across subjects and datasets. This captures **inter-region coupling** (or functional connectivity) patterns in EEG—features that have been increasingly emphasized in recent neuroscience and EEG decoding literature [1-3]. While MMD with a Gaussian kernel can capture higher-order statistics, there is currently limited evidence in the EEG literature suggesting the functional relevance of such statistics. Moreover, MMD introduces sensitivity to kernel bandwidth and other hyperparameters. Therefore, we prioritize CDA loss as a **simple, interpretable, and stable** alignment objective in our framework.\n\nFor the model architecture, we aim to design a **physiologically plausible and computationally efficient** EEG encoder. We were not fully satisfied with standard Transformer architectures due to their limited interpretability and relatively large parameter counts. As such, we propose a tailored architecture consisting of an MLLA-based Channel Encoder and a Spatiotemporal Dynamics Module. The former captures rich temporal dynamics from each EEG channel, while the latter integrates information across channels and time, enabling the model to learn coordinated patterns of brain activity. \n\nThe MLLA component is particularly suited for modeling **long-range dependencies** in EEG due to its combination of an input gate, linear attention, and a forget gate. We apply overlapping strided patches (see Fig. 2D), where each patch is first linearly projected—functionally similar to strided convolution—to extract **local features**. These are then processed by linear attention for **global dependency** modeling, followed by a forget gate to **filter salient patterns**. In contrast, standard Transformers use the QKᵀV mechanism to assign weights to input tokens by computing dot products between queries and keys, which, in our view, lacks physiological priors to effectively capture EEG-specific characteristics. The Spatiotemporal Dynamics Module further captures inter-channel dynamics and uses local attention to emphasize important spatial-temporal patterns. \n\nWhile model architecture is not the main contribution of this work, we designed our encoder based on domain-specific understanding of EEG data and validated its effectiveness via lesion studies. We believe this principled design—though not exhaustively benchmarked against all alternatives—should not undermine the merit of our proposed framework. In the revised version, we will further explain the motivation of our framework design and discuss its relashionship with other methods.\n\n[1] Zhang Y, Chen Z S. Harnessing electroencephalography connectomes for cognitive and clinical neuroscience[J]. Nature Biomedical Engineering, 2025: 1-16.  \n[2] Li C, Tang T, Pan Y, et al. An efficient graph learning system for emotion recognition inspired by the cognitive prior graph of eeg brain network[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024, 36(4): 7130-7144.  \n[3] Wang Y, Qiu S, Ma X, et al. A prototype-based SPD matrix network for domain adaptation EEG emotion recognition[J]. Pattern Recognition, 2021, 110: 107626.\n\n---\n\n### Q2. The comparison with other self-supervised frameworks using trial information.\n\nTo demonstrate the effectiveness of our self-supervised strategy over others using trial information, we compared it with **Supervised contrastive learning (supervised CL)**, which regards samples with the same emotion labels as positive pairs and those with different emotion labels as negative pairs in each dataset. It directly optimizes based on emotion labels (but agnostic to specific emotion categories). We use SupCon loss [4] and keep all other settings the same. Our strategy outperforms supervised contrastive learning by a large margin, indicating the effectiveness of our ISA loss for EEG alignment.\n\n| Method        | Accuracy | Precision | Recall | F1 Score | AUROC |\n| ------------- | -------- | --------- | ------ | -------- | ----- |\n| Supervised CL |47.99±2.51|47.89±2.22|47.86±2.41|47.53±2.33|76.28±1.73|\n| Ours          |65.02±0.98|65.06±1.28|65.53±0.66|64.85±1.01|88.70±0.98|\n\n[4] Khosla P, Teterwak P, Wang C, et al. Supervised contrastive learning[J]. Advances in neural information processing systems, 2020, 33: 18661-18673.\n\n---\n\n### Q3. The effects of training batch selection in pre-training.\n\nIndeed, the trianing batch selection is carefully designed to learn subject-invariant representations. Sampling two temporally aligned samples as the positive pair is critical. In the preliminary experiment, we found that when the two positive samples are randomly selected from the same trial without temporal alignment, the model has a near random top1 accuracy in contrastive learning. This is possibly due to that temporally aligned samples carry more shared information, which facilitates the model to focus on meaningful shared representations across subjects and discard subject-specific noise. We will add these comparison experiments in the revised version.\n\nWe also experiment with larger batch sizes in pre-training. As previous work in contrastive learning, such as SimCLR, demonstrated that contrastive learning benefits from a larger batch size, we increase our batch size to double: Obtain two samples from each trial in one batch (one sample from each trial was obtained in the previous version). Samples that are not temporally aligned across subjects are regarded as negative samples. \n\nWe found doubling the batch size increases the leave-one-dataset-out few-subject fune-tuning accuracy on the SEED-V dataset by 2.08% and by 2.27%. This demonstrates that larger batch size also benefit EEG contrastive learning for learning effective representations. However, doubling the batch size also increases the occupation of GPU memory from approximately 20G to 40G in joint pre-training, which poses a higher demand of the hardware. Future work is needed to more comprehensively investigate the effect of batch size on downstream performance.\n\n---\n\n### Q4.1 Transfer from the DEAP dataset to SEED datasets.\n\nWe conducted experiments to train the model on the DEAP dataset and transfer it to other datasets. In the table, we compare the performance of transfer from DEAP with transfer from SEED.\n\n| Accuracy        | from DEAP | from SEED |\n| ---------------- | -------- | --------- |\n| to SEED     |66.77±3.33|/|\n| to SEED-IV  |46.24±0.87|48.52±1.65|\n| to SEED-V   |54.55±2.13|55.85±2.31|\n| to SEED-VII |36.35±1.04|37.33±1.19|\n| to FACED |22.83±1.48|18.95±1.79|\n\nThe performance of transfer from DEAP is slightly lower than transfer from SEED to other SEED-series datasets, but the performance gap is within 2.3%. Besides, the accuracy of transfer from DEAP to FACED is higher than transfer from SEED to FACED by 3.9%. These results demonstrate the effectiveness of our framework in transferring between datasets with discrepancies in emotion label paradigms. It aligns with our expectation that the model can learn generalizable emotion representations through contrastive learning, which do not rely on emotion label information.\n\n---\n\n### Q4.2 The issue of individual differences in valence/arousal experience on the DEAP dataset.\n\nWe agree that inter-subject variability is a fundamental challenge in affective EEG analysis. Such variability arises from two sources:  \n(1) Differences in emotional perception (i.e., subjects experiencing different emotions in response to the same stimulus)  \n(2) Differences in neural signal properties that are unrelated to emotion (e.g., due to montage, scalp geometry, or baseline spectral patterns)  \n\nIn this work, we focus on mitigating the second type—signal-level inter-subject variability, which poses a major obstacle to generalization across subjects and datasets. Modeling both types of variability simultaneously would significantly complicate the training process and potentially reduce robustness.  \n\nTo address this, mdJPT leverages temporally aligned EEG segments as a weak prior for capturing shared neural patterns. Although such alignment does not ensure identical emotional states across subjects, it increases the likelihood of emotion similarity compared to unaligned segments. This provides a practical and effective anchor for extracting robust emotion-related representations.  \n\nWe acknowledge the importance of accounting for subjective emotional differences, and we plan to incorporate more individualized modeling strategies (e.g., soft contrastive labels based on individualized emotion ratings) in future work."}]

Author rebuttal:

Meta-review: This paper addresses a task-specific multi-datatset joint pre-training framework for cross-dataset emotion recognition. The key idea is to jointly pre-train a hybrid EEG encoder on several emotion datasets, introducing a cross-dataset covariance alignment and inter-subject alignment. All of reviewers feel that the approach taken in this paper is sound and promising for tackling task-specific complex problems  of EEG recognition. One of critical concerns raised by reviewers is about “zero-shot generalization to new emotion categories”. Experiments on zero-shot generalization was done during the rebuttal period and reasonable empirical results resolved the concern. The authors did a good job in responding to the issues pointed by reviewers during the rebuttal period. This made us to reach consensus on this paper.
Decision: Accept (poster)