SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Although our method uses only one-tenth of the information, our method outperforms the previous state-of-the-art self-supervised learning methods, achieving a performance enhancement of  $0.44\%$  on the SEED dataset and  $0.77\%$  on the SEED-IV dataset. When compared with the supervised methods, our model exhibits a significant improvement, yielding a  $1.52\%$  increase in performance on the SEED dataset and a substantial  $14.25\%$  enhancement on the SEED-IV dataset, not to mention their setting to exclude the worst-performing session.
After pre-training, MAE and MMM get an accuracy increase of  $3.96\% / 2.87\%$  and  $1.17\% / 1.88\%$  on SEED/SEED-IV, respectively, confirming the significance of EEG pre-training. Moreover, our proposed MMM framework, though following the MAE training schema, outperforms MAE by  $3.91\% / 12.51\%$  on SEED/SEED-IV. This improvement suggests the effectiveness of our specific designs for EEG pre-training.
4.3 Transfer between Different Datasets
In this section, we examine the models' transferability across datasets with identical sensor configurations. We pre-train models on the original dataset, including SEED, SEED-IV, and their combinations. Subsequently, the fine-tuning performance of these pre-trained models is evaluated on the target datasets, SEED and SEED-IV.
As Table 2 demonstrates, all pre-trained variants outperform the corresponding baseline model trained from scratch. Interestingly, the model pre-trained on the SEED-Union by MAE only gets the second-best fine-tuning performance on both datasets. This indicates that even if the channel configuration is identical, the underlying difference in the data distribution among datasets may hinder the model without a unified representation from taking advantage of more data resources. However, while models pre-trained by MMM and MAE share the same architecture, the trend is reversed for models pre-trained by MMM. The model pre-trained with the SEED-Union dataset exhibits the
Table 2: Transfer study: Models get pre-trained on the origin dataset, (-) denotes models get trained from scratch. Then we fine-tune and test the models on the target dataset. The best results for each method and target dataset have been marked in bold.
Pre-training Dataset	Accuracy (%)
SEED	SEED-IV
MMM	MAE	MMM	MAE
-	94.59	87.89	86.04	78.22
SEED	95.15	90.96	87.25	81.09
SEED-IV	95.35	91.85	86.85	79.15
SEED-Union	95.76	90.59	87.92	80.50
best downstream performance on both datasets. This indicates that our method, built on the unified topology, can better use a larger number of data to extract more generalizable representations. For more training details, refer to the supplementary.
4.4 Transfer between Different Montages
We further investigate the transferability of our method across different sensor configurations in this section. We pre-train models using the Lite series dataset and then conduct a partial fine-tuning on the full SEED dataset, i.e., the parameters of the encoder are frozen, and only the last MLP is tuned. As reflected in Table 3, the model pre-trained by MMM on both the SEED-Lite and SEED-Union-Lite datasets outperforms the model trained from scratch. However, the model pre-trained by MMM on SEED-IV-Lite shows a performance drop. We suggest that the SEED-IV, with only 32 channels, might present a substantial data shift, making it challenging for the pre-trained model to fine-tune/partially fine-tune. However, all models pre-trained using the Mask Autoencoder (MAE) fail to match the partially fine-tuning performance of the model trained from scratch. On the other hand, the model pre-trained using the Multi-modal Multi-task (MMM) approach on the SEED-Union-Lite dataset showcases superior performance. This suggests again that the incorporation of the unified topology aids in the extraction of more generalized representations.
Table 3: Partially fine-tuning results on SEED dataset.
Pre-trained Dataset	MAE	MMM
Acc. (%)	Change	Acc. (%)	Change
SEED (Fine-tune)	91.86	-	95.29	-
-	79.82	-12.04	92.54	-2.75
SEED	76.07	-15.79	93.23	-2.06
SEED-Lite	74.37	-17.49	92.70	-2.59
SEED-IV-Lite	75.80	-16.06	91.92	-3.37
SEED-Union-Lite	75.05	-16.81	92.81	-2.48
4.5 Transferred from large-scale EEG corpus
Table 4: Large-scale pre-training: Models get pre-trained on TUEG dataset, (-) denotes models get trained from scratch. Then, we fine-tune and test the models on the SEED dataset.
Pre-training dataset	epoch	Fine-tuning Accuracy (%)
-	0	94.59
TUEG	1	94.85
TUEG	2	95.11
SEED	200	95.15
TUEG	3	95.29
In light of the preceding investigations discovering the effectiveness of our methodology across varying datasets and montages, we proceeded to deploy our approach on TUEG dataset - a large-
scale EEG corpus. Notably, although TUEG dataset collects data on a distinct channel configuration involving 21 channels in the 10-20 system, its utilization in pre-training remains beneficial. As shown in Table 4, there is a discernible increment in the downstream classification accuracy as the number of pre-training epochs increases. Remarkably, when leveraging TUEG for pre-training, our method outperforms self-pre-training within just three epochs. These findings may illuminate the path toward constructing foundation models in the EEG domain.
4.6 Impact of Region Division
Table 5: Results from variants of different region division selection
Number of Regions	Accuracy
1	93.95
4	93.41
7	93.68
17 (chaos)	93.96
17 (full attention)	94.08
62	94.53
17 (default)	94.61
In this section, we thoroughly analyze the impact of different unified topologies in MMM on the SEED dataset. We compared our default 17-region setting with several variants: 1) 1 single region for all channels; 2) 4 regions shown in Figure 5 (b); 3) 7 regions shown in Figure 5 (d); 4) 62 regions, each channel taking up an individual region; 5) 17 in chaos, where channels are divided to 17 groups without considering spatial relationships. Results are shown in Table 5. From the results, we can conclude that a proper choice of region setting contributes to a better result. Refer to the Appendix C.1 for comprehensive analysis.
In this section, we scrutinize the design of MMM for both the SEED and SEED-Lite datasets. Table 6 provides an ablation study on the contributions of different components within the MMM model.
Table 6: Ablation study on different components of MMM on the SEED dataset and SEED-Lite dataset.
Multi-dimensional Positional Encoding	Multi-level Channel Hierarchy	Multi-stage Pre-training	SEED	SEED-Lite
90.96	90.74
✓	91.46	91.31
✓	✓	95.28	94.29
✓	✓	✓	95.15	94.62
The table reveals that introducing region-wise tokens leads to the most significant enhancement in performance. This underscores the significance of capturing local and regional patterns within the EEG data for effective representation learning.
Moreover, the EEG-specific position encoding (PE) also bolsters the overall performance of the model, as indicated by a  $0.4\%$  rise in accuracy. This underlines the value of incorporating spatial information into the model, which aids in understanding the interrelationships among EEG channels and their respective locations on the scalp. Interestingly, pre-training with a multi-stage masking strategy slightly decreases performance on the full-channel SEED dataset but boosts performance on the SEED-Lite dataset. This suggests that the multi-stage mask strategy aids in enhancing the model's generalizability across various datasets. However, it may introduce a slight trade-off, potentially causing minor performance decreases on certain datasets.
Figure 3: The topographic maps of the reconstructed test data with different mask rates in the  $\gamma$  band. The row denotes different test samples from the SEED dataset. The first column is the origin data, while the rest columns stand for the topology reconstructed by MAE pre-trained model (Left) and the topology reconstructed by MMM pre-trained model (Right) with different mask rates.
Figure 4: Reconstruction losses on the test samples of SEED
4.8 EEG Reconstruction with Unified Representation
Figure 3 presents the visualization of the reconstruction quality for both the mask autoencoder and our method. By analyzing the reconstruction losses shown in Figure 4, we find that MMM demonstrates slightly inferior reconstruction results at a low mask ratio compared to MAE, which can be attributed to the amount of information they feed into reconstructors. Specifically, MAE's reconstructor can access nearly all origin tokens at a low mask ratio, while MMM's reconstructor relies solely on high-level representation, forgiving all origin tokens. However, as the mask ratio increases, MAE's performance starts to collapse while MMM maintains superior reconstruction quality. This phenomenon further substantiates that our approach can effectively model EEG data by the unified topology, yielding robust representations with rich information.
In this study, we have proposed MMM, an innovative framework for EEG pre-training, which effectively navigates the complexities associated with various sensor configurations in EEG data and well encodes the spatial information of EEG. Our model employs a self-supervised learning approach based on the masked auto-encoder, allowing it to benefit from a vast cross-dataset corpus and enhance the performance of downstream tasks. Our framework demonstrates state-of-the-art performance in emotion recognition tasks, and further evaluations substantiate the efficacy of our methodology. Looking forward, we envision extending our approach to more downstream tasks and modeling temporal and spatial information of EEG data efficiently.
[23] Y. Li, W. Zheng, Y. Zong, Z. Cui, T. Zhang, and X. Zhou, “A bi-hemisphere domain adversarial neural network model for eeg emotion recognition,” IEEE Transactions on Affective Computing, vol. 12, pp. 494–504, 2021.
[24] P. Zhong, D. Wang, and C. Miao, "Eeg-based emotion recognition using regularized graph neural networks," IEEE Transactions on Affective Computing, vol. 13, pp. 1290-1301, 2022.
[25] J. Hawkins and S. Ahmad, "Why neurons have thousands of synapses, a theory of sequence memory in neocortex," Frontiers in neural circuits, p. 23, 2016.
[26] M. Rubinov and O. Sporns, “Complex network measures of brain connectivity: uses and interpretations,” Neuroimage, vol. 52, no. 3, pp. 1059–1069, 2010.
[27] W.-L. Zheng, W. Liu, Y. Lu, B.-L. Lu, and A. Cichocki, "Emotionmeter: A multimodal framework for recognizing human emotions," IEEE Transactions on Cybernetics, vol. 49, pp. 1110-1122, 2019.
[28] A. Harati, S. Lopez, I. Obeid, J. Picone, M. Jacobson, and S. Tobochnik, "The tuh eeg corpus: A big data resource for automated eeg interpretation," in 2014 IEEE signal processing in medicine and biology symposium (SPMB). IEEE, 2014, pp. 1-5.
[29] Y. Li, L. Wang, W. Zheng, Y. Zong, L. Qi, Z. Cui, T. Zhang, and T. Song, "A novel bi-hemispheric discrepancy model for eeg emotion recognition," IEEE Transactions on Cognitive and Developmental Systems, vol. 13, no. 2, pp. 354-367, 2021.
[30] T. Song, S. Liu, W. Zheng, Y. Zong, Z. Cui, Y. Li, and X. Zhou, "Variational instance-adaptive graph for eeg emotion recognition," IEEE Transactions on Affective Computing, vol. 14, no. 1, pp. 343-356, 2023.
A DE feature extraction
The key idea of extracting DE features from EEG is calculating the accumulated power within several pre-defined frequency bands. We use  $x \in \mathbb{R}^{C \times T}$  to denote the raw EEG data, where  $C$  is the number of EEG channels and  $T$  is the number of samples (i.e.,  $N =$  time duration × sampling frequency). The procedure for processing DE features is defined as:
$$
w (t) = 0. 5 - 0. 5 \cos \left(\frac {2 \pi t}{N - 1}\right), \quad t = 0, 1, \dots , N - 1, \tag {1}
$$
$$
s _ {w} (t) = s (t) w (t), \tag {2}
$$
$$
X (\omega) = F F T \left\{s _ {w} (t) \right\}, \tag {3}
$$
$$
P (k) = | X (k) | ^ {2}, \quad k = 0, 1, \dots , \frac {N}{2}, \tag {4}
$$
$$
D E (i) = \frac {1}{n _ {i}} \sum_ {k = s t a r t _ {i}} ^ {e n d _ {i}} P (k), \quad i = 1, 2, \dots , F. \tag {5}
$$
$n_i$  is the number of samples in the  $i$ -th frequency band while  $start_i$  and  $end_i$  marks its start and end point.  $F$  is the number of frequency bands. Following medical practices, previous work extract DE feature on five frequency bands, i.e.,  $\delta$  band (1-3Hz),  $\theta$  band (4-7Hz),  $\alpha$  band (8-13Hz),  $\beta$  band (14-30Hz),  $\gamma$  band (31-50Hz). With the help of DE features, one-second-long,  $C$ -channel EEG data with hundreds of samples can be represented as a  $C \times 5$  vector, which can help us extract and aggregate temporal information and avoid inconsistency of sampling frequency in different datasets.
Here, we describe the settings used for pre-training and fine-tuning/partially fine-tuning our models on SEED series datasets.
B.1 Pre-training
We use 6 Transformer layers as encoder and symmetrical layers as decoder. The input EEG data should have 62 channels. Since the EEG data size varies greatly, we first add the channel with random noises. Then, we normalize the EEG data according to the mean and standard deviation calculated on the training set. The hidden dimension of each channel is set to 16. For multi-dimensional positional encoding, we encode x and y with 8 dimensions, respectively, and the combinations are added to the hidden vectors. For global random masking, we pick the masking ratio to 0.5. For regional masking, we pick the ratio of masking region to 0.35 for the competitive difficulty. For each 100 epochs, the previous 50 epochs take global random masking, and the rest take region-wise masking. For region division, we set 17 regions as mentioned before, i.e., 17 region tokens are appended to the input sequence and used for reconstruction.
We use 1 NVIDIA V100 GPU to train models for 600 epochs with a batch size of 512. We use AdamW to optimize the network. Learning rate and weight decay are grid-searched to the optimal.
B.2 Fine-tuning
We load the weights of the pre-trained encoder and add a randomly initialized classifier upon the region tokens. In the SEED series, each subject has three sessions, and each session contains multiple trials. For each session, the ahead trails are viewed as training samples, and the rest are viewed as testing samples. The training/test split is 9/6 for SEED and 16/8 for SEED-IV, i.e., for the SEED dataset, each subject contains 27 trials as training samples and 18 trials as testing samples. The final results come from the average of all subjects. We grid-search the learning rate and weight decay, following the same strategy in [4]. We use 1 NVIDIA V100 GPU to train the model for 100 epochs with a batch size of 32.
B.3 Partially fine-tuning / linear probing
Similar to fine-tuning, we load the pre-trained weight of the pre-trained encoder and add the classifier. However, we freeze the encoder during the backward propagation to conduct partially fine-tuning / linear probing experiments.
- STRNN [17]: The spatial-temporal recurrent neural network is based on a unified spatial-temporal dependency model that learns the information from both spatial and temporal domains of EEG signals.
- DGCNN [18]: Dynamical graph convolutional neural network learns the representations of EEG signals by graph convolution in a dynamic way for EEG-based emotion recognition.
- BiDANN [23]: Bi-hemisphere domain adversarial neural network focuses on discriminative features of EEG signals from both the right and left sides of the hemispheres of the brain for EEG-based emotion recognition.
- BiHDM [16]: Bi-Hemispheres discrepancy model investigates the asymmetric differences of the right and left hemispheres of the brain.
- R2G-STNN [15]: A region-to-global spatial-temporal neural network model learns the global and regional EEG representations in both spatial and temporal aspects of EEG signals.
- RGNN [24]: Regularized graph neural network explores the topology of EEG channels with graph convolution.
- MD-AGCN [22]: A multi-domain adaptive graph convolutional network taking full advantage of features on different domains.
- MAE [12]: Masked autoencoders as scalable self-supervised learners by reconstructing the missing patches in images for computer vision.
- MV-SSTMA [4]: A spatial-temporal masked autoencoder by reconstructing the missing channels in EEG signals.
C Analysis of Different Experiment Settings
C.1 Impact of Region Division
Table 7: Results from variants of different region division selection
Number of Regions	Accuracy
1	93.95
4	93.41
7	93.68
17 (chaos)	93.96
17 (full attention)	94.08
62	94.53
17 (default)	94.61
Our approach to handling diverse datasets with different montages maps different channel configurations to a unified topology. Our paper uses a 17-region setting based on some neurological findings. However, We would like to investigate how a different selection of regions may influence the model behavior built on it.
In this section, we thoroughly analyze the impact of different unified topologies in MMM on the SEED dataset. We compared our default 17-region setting with several variants: 1) 1 single region for all channels; 2) 4 regions shown in Figure 5 (b); 3) 7 regions shown in Figure 5 (d); 4) 62 regions,
(a)17
(b)4
(c)17 in chaos
Figure 5: Different region division
(d)7
each channel taking up an individual region; 5) 17 in chaos, where channels are divided to 17 groups without considering spatial relationship. Results are shown in Table  $7^{3}$ .
From the results, we can conclude that a proper choice of region setting contributes to a better result. The selection of the number of regions is a trade-off between modeling fine-grained spatial information and aggregating local information of a larger range. Therefore, in our experiments, a division with fewer regions can only gather coarse-grained spatial information, and more regions may fail to aggregate local information properly, which both perform worse than our default 17-region setting.
Moreover, by comparing our default 17-region setting with the chaos 17-region setting, we can see that the design of aggregating spatial information is also essential to the performance. This suggests that a good unified topology does not mean inserting an appropriate amount of extra tokens but also taking locality and native functionality into consideration when dividing regions.
C.2 Impact of Masking Ratio
In this section, we investigate the impact of the masking ratio for MMM on the SEED dataset.
The global random mask rate is set to different ratios, and for regional masks, the mask ratio is set to have  $15\%$  less than the random mask ratio for a proximal difficulty. Results are shown in Table 8
3Considering the pre-training and downstream evaluation are time-consuming, the space of grid search is smaller for experiments in the Appendix, which may result in a slightly different reported performance.
Table 8: Results from variants of different masking ratio
Random Masking Ratio	Accuracy
0.3	94.54
0.5	94.61
0.7	94.40
0.9	94.57
As we expect, a lower masking ratio results in a weak pretext task, as the model can easily reconstruct the signals given more visible channels, and thus its representations are not as useful. Interestingly, mask ratio  $= 0.9$  doesn't result in a large drop in performance, unlike [12]. This suggests that higher masking ratios may be used for EEG data, as it results in fewer tokens during the encoding state, which could absorb data within more configurations for pre-training.
C.3 Impact of Masking Strategy
Table 9: Results from variants of different masking strategy
Random Masking	Regional Masking	Downstream Accuracy	Reconstruction loss
0.3	0.7	94.49	8.26
0.5	0.5	94.61	8.07
0.7	0.3	93.67	8.12
In this section, we investigate the impact of the masking strategy for MMM on the SEED dataset. Results are shown in Table 9. The appropriate mix rate of mask strategies can bring better representations.
C.4 Impact of depth of encoders and decoders
Table 10: Results from variants of different transformer layers
Decoder
4	6	8
Encoder	4	93.86	93.99	94.55
6	94.92	94.61	94.33
8	94.17	94.52	93.95
In this section, we investigate the impact of the depth of encoder and decoder in MMM on the SEED dataset. We use encoders and decoders with different numbers of transformer layers and report the results in Table 10
To be noticed our setting in main experiment should follow the [4] for a fair comparison, which has 6 depth for encoders and 6 depth for decoders. From a horizontal perspective, the deeper decoder can help the overall auto-encoder well model the EEG data, but too many layers would take over the difficulty and understanding of the encoders. From a vertical view, the deeper encoders can bring more understanding from the pre-training data, but too many layers would bring the overfitting problem.
C.5 Impact of different components
To further scrutinize the design of different components, we conduct a simple but effective ablation study as shown in Table 11. To be noticed the decrement brought by Multi-stage pre-training has been explained in Section 4.7.
Table 11: Ablation study on different components of MMM on the SEED dataset.  $(\times)$  means the component is missing
Multi-dimensional
Positional Encoding	Multi-level
Channel Hierarchy	Multi-stage
Pre-training	SEED
X	95.12
X	91.93
X	95.28
95.15
Electroencephalogram (EEG) is a physiological signal widely used in clinical practice and brain-computer interfaces. Instead of fitting models with limited data within a task, our method exploits abundant, topology-agnostic, and cross-tasks unlabelled EEG data. Our pre-trained model can draw unified and meaningful representations that are beneficial for various downstream tasks. Neurologists and engineers could make good use of such benefits in diagnosis and BCI applications.
While our work of EEG pre-training doesn't aim to handle a specific sensitive task with negative social impacts, the technique might possibly be misused in the future with privacy concerns, e.g., reading one's mind from EEG signals without acknowledgment. However, we believe that we're moving in the right direction toward the target of building an EEG foundation model to understand EEG signals better. This is essential to brain disease diagnosis like seizures, Parkinson's disease, etc., and is life-saving.
Our method is designed to provide a unified solution to different EEG configurations. While it requires a unified modeling strategy on both spatial and temporal dimensions, our focus is more on the spatial information, and we handle the temporal information with a non-parametric solution. Our framework, however, is also capable of different kinds of temporal modeling strategies, and we believe the results can be further boosted with a more advanced technique to handle temporal information better. Also, our model is currently tested only on the emotion recognition task, and we believe the framework can be easily adapted to other EEG tasks to achieve better performances with more data available for training.