Learning Topology-Agnostic EEG Representations with Geometry-Aware Modeling
Authors: Ke Yi
Yansen Wang
Kan Ren
Dongsheng Li
Venue: NeurIPS 2023 poster (2023)
Keywords: Electroencephalogram
EEG Pre-training
EEG-based emotion recognition

Abstract: Large-scale pre-training has shown great potential to enhance models on downstream tasks in vision and language. Developing similar techniques for scalp electroencephalogram (EEG) is suitable since unlabelled data is plentiful. Meanwhile, various sampling channel selections and inherent structural and spatial information bring challenges and avenues to improve existing pre-training strategies further. In order to break boundaries between different EEG resources and facilitate cross-dataset EEG pre-training, we propose to map all kinds of channel selections to a unified topology. We further introduce MMM, a pre-training framework with Multi-dimensional position encoding, Multi-level channel hierarchy, and Multi-stage pre-training strategy built on the unified topology to obtain topology-agnostic representations. Experiments demonstrate that our approach yields impressive improvements over previous state-of-the-art techniques on emotional recognition benchmark datasets.

Learning Topology-Agnostic EEG Representations with Geometry-Aware Modeling
Ke Yi
Yansen Wang Microsoft Research Asia yansenwangemi crosoft. com
South China University of Technology cs ss_keerr@mmal.. scut. edu. en
Kan Ren
Dongsheng Li Microsoft Research Asia dongeli wnngslifiiroooott.oo com
Microsoft Research Asia kanren&mi kanren@microsoff. crosoft. com
Abstract
Large-scale pre-training has shown great potential enhance models down- stream tasks in vision and language. Developing similar techniques for scalp elec- troencephalogram (EEG) is suitable since unlabelled data plentiful. Meanwhile, various sampling channel selections and inherent structural and spatial information bring challenges and avenues to improve existing pre-training strategies further. In order to break boundaries between different EEG resources and facilitate cross- dataset EEG pre-training, we propose map all kinds of channel selections a unified topology. We further introduce MMM,, pre-training framework with Multi-dimmessonal position encoding. Multi-level channel hierarchy, Multi-stage pre- pra-training strategy built on the unified topology to obtain topology-agnen representations. Experiments demonstrate that our approach yields impressive improvements over previous state-of-the-art techniques on emotional recognition benchmark datasstts.
Introduction
Scalp electroenceephalggrapy (EEG), a physiological signal that shows the human brain's 's status and can be collected in a non-invasive and low-cost manner, is widely used in disease diagnosis, medical monitoring, rehabilitation, and some recent brain-computer interface systems 2 3. Although collecting EEG signals is relatively easy, interpreting the signals and providing meaningful labels often require huge expert efforts. Therefore, few recent works have emerged learn representations of EEG segments with self-supervised learning methods and apply the pre-trained model to several downstream tasks, including emotion recognition, sleep staging, epileptic seizure detection, and so on 4,556, 6 7 8). While there are many publicly available EEG datasets for pre-training, transferring across them very challenging because these EEG recordings can vary significantly in terms of montage (the number and the places of electrodes placed on the scalp) and sample rate. These factors will completely change the format and the EEG contents' meaning. For example, typical setting of the emotion recognition task involves 62 channels based on the 10-10 electrode system to track the fine-grained changes with very high spatial resolution, and more practical setting to monitor seizure events may only have 19 channels based on the 10-20 system. Nevertheless, previous self-supervised learning methods avoid handling this problem and thus can only work within a single dataset, which hinders the model from being trained more available resources.
'The work was conducted during Ke Yi's internship at Microsoft Research Asia. "ooreepoondence Yansen Wang and Kan Ren.
Prooeect link:
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Another challenge of EEG pre-training is to encode the structure and model the relationships among EEG signals from different locations. These signals are collected by the electrodes placed on the scalp, which can be viewed as a 2-D manifold with spatial information. This spatial information often crucial for the expert to provide precise labels. For example, lesions can be located by comparing the amplitudes and phases of abnormal discharges between the left and right brain, and changes in emotional states may be reflected by asymmetrical EEG activity [9 [0
In summary, to get a capable and powerful EEG pre-trained model, we need 1) well encoding spatial information into the representation and 2) pre-training models with EEG corpus having various sensor configurations. In this paper. we propose MMM, self- supervisdd learning framework based on the masked auto-encoder [12]. which maps EEG data with different sensor configurations into unified representations. We introduce region-wiss tokens to extract local information from the EEG channels, where region-wise tokens and origin channels form a Multi-level hierarchy learning. The well-learned region-wisg tokens form the unified representation, which can be applied in downstream tasks with rich spatial information. We demonstrate that introducing fulti-dimensional position encoding to encode geometric information of sensors benefits pre-training, and Multi-stage mask strategy with random mask and region-wise mask can enhance the hidden representation' 's robustness. To further evaluate the effectiveness of our proposed framework, we validate our approach emotion recognition tasks, which is a typical EEG downstream task and requires the model to capture the spatial information well.
We summarize our contributions as follows:
We propose to use unified topology to model the scalp and map different montages onto the same topology.
We propose a novel pre-training framework to learn unified geometry-aware EEG representations on top of the unified topology that can be generalized to different EEG channel configurations.
Our pre-trained model gains state-of-the-art performance on the emotion recognition benchmarks. More experiments demonstrated its strong ability to transfer between different datasets even with different montages.
2 Related Works
Self-supervised learning for EEG Electroencephalograms (EEGs) are commonly used to diagnose neurological, psychiatric. and sleep disorders and applications involving prahne interfaces. In the EEG domain, self-supervised learning has emerged as promising approach. In [8) the author designs two pretext tasks, temporal context prediction, and contrastive predictive coding, to perform representation learning of EEG signals and applies them to two downstream tasks: sleep staging and pathology detection. [6] introduce a set of data augmentations for EEG and extend the SimCLR framework to extract channel-wise features on time-series EEG data. The above methods focus on temporal information modeling while ignoring spatial information. 13 attempt to solve the cross- montage problem by selecting fixed channel set, while it might cause spatial information loss when faced with different configurations. 7 use GNN and RNN to model the spatiotemporal dependencies while using the generative pre- pre-training method to improve model performance further. However, their GNN-based model assumes a fixed channel set in their experiments and might naturally be worse at transferring than the attention-based model.
To the best of our knowledge, there is no method trying use cross-dataset EEG as their pre-training corpus.
EEG-basee emotion recognition Before the deep learning methods were extensively adopted in this area, spectral EEG features were commonly investigated, such as power spectral density (PSD), differential entropy (DE), and differential asymmetry (DASM), from which DE feature is proved to be the most precise and stable one in BEG-based emotion recognition tasks [I]] Several EEG-based
emotion recognition methods have been proposed based on the DE feature. [15]
16 I7] apply
recurrent neural networks to traverse signals and capture the spatial dependency between the EEG channel. However, RNN-based modeling takes the EEG channel
fixed sequence, which loses sensors' geometric information and makes transferring hard. [88 19] apply graph neural network to
model spatial relationship among EEG channel and GCN-based model achieve ofte-of-the-are in

---

BEG-based emotion recognition. However, these GCN- models are only claimed to be used in supervised training. has shown that Masked EEG modeling is an effective self-supervised pre-training paradigm for EEG data. However, this work takes EEG data as a sequence while ignoring the sensors' physical location information. Again, the above methods are all not trying to pre-training/raining on cross-dataset EEG.
3 Methodology
3.1 Task Formulation
Learning good EEG representation involves modeling from both temporal and spatial aspects. While temporal modeling is better explored the previous works, our main focus is on spatial structures. We use DE feature (00] extracted from the raw EEG to aggregate the temporal information. This simplification can bring several benefits: 1) DE feature is a well-studied feature that demonstrated to be effective for temporal modeling of EEG, especially in EEG-based emotion recognition tasks 21 2) The process of DE feature extraction doesn't 't involve learnable parameters and can exclude the eliminating impact of sample rate variations. This allows the feature to be seamlessly transferred across datasets with different time resolutions. 3) The extracted features are widely used across wide range of baseline models (22 23 24] thereby facilitating equitable comparisons among various frameworks. We describe how DE features are extracted in the Supplementary file. It is worth noting that our proposed method is also applicable raw EEG signals and other kinds of features as long as the temporal information is handled in a consistent manner across datasets.
In this way, the EEG pre-training task can be formulated as learning unified hidden representation = Encoder(x)) E RR from a dataset X' containing the extracted DE features x E X C RC.FF, which can be afterward used in downstream tasks. H is the dimension of the hidden representations. C₂ is the number of channels corresponding to a set of sensors &, = {E1, E2, Ec, used to collect the signal. F is the dimension of the DE feature, which specifically equals 5 in our case, representing the DE feature extracted on five frequency bands, i.c., å band (I-4Hz), 0 band (4-8Hz), band (8- (8-14H2), 3 band (14-31Hz), y band (31-50Hz).
Unlike vision pre-training EEG data within single task is limited, while cross-task EEG data varies significantly in the sensor configuration. Our target is to 1) pre-train general feature extractor for EEG signals on an irregular dataset X, which breaks the constraint that data have to share the same sensor config funiiuuraiion, and 2) encode the geometric information of the sensors properly which reflect the locations of the EEG signals collected.
3.2
Unified Topology
The core idea of addressing the problems is to map different sensors' configurations onto a unified topology and extract generalized representation that relies on the geometric information of sensors. Recent research in neuroscience reveals that the biological nervous system forms several functional groups in different brain regions (25). and these regions are interconnected with physical, functional, and effective connectivity (26). These functional regions provide us with a plausible basis for modeling EEG signals which provide macroscopic view of brain activities.
Drawing inspiration from the functional organization of the brain, we abstract the scalp as a unified topology that can be segmented into several regions R (RR, R2,... Rs}. Each region can be represented by a vector z R._ Consequently, the whole scalp can be represented by the collection of vectors from all regions as Z RSD where S denotes the number of regions and D denotes the
dimension of a regional representation. In this paper, we follow the previous findings [19] and divide the scalp into S 17 regions, shown in Figure (b). We leave the analysis of the influence of different region settings in the supplementary material.
Hence, the challenge of processing data with various montages can be viewed as mapping data x E with the varying sensor configuration &, into the representations of the unified topology Z Encoder(z). These representations in the same shape can thus be used to transfer among different datasets without any burdens. Moreover, because we design plausible mapping between channels and regions, the geometric information can be naturally incorporated.
3

---

Projection
0od
ood
000 000
Downstream Task
Encode
Projection
Decoder
(%, v)
Multi dime-sdmnnsonnn
000
000
000000
000
000000
Posttion Encoding
0
Unified Representation
Encoder
6
dimension
m
000
coar an m 000 m
Position Encoding
Linear Projection Flattened Nodes
GECAAN OLD DOODED RODDOCUGE O GLOD Brin DE DIDIRO
TOODOGS
Origin Region-wise
(b)
(4)
Overview of methods. Cireles and diamonds represent original nodes and region-wise nodes. unified topology. White patterns get masked/re-mant masked (b) Mapping the original EEG data into a unified topology based on channel locations. (c) Illustration of the multi sitiiddimenionnll positional encoding. (d) Attention masks. Green means the two tokens interact with attention, and white means direct interaction.
Figure 1:
Nodes the same color belong to same region before being fed into the encoder/decoder.
3.3 MMM: an EEG Pre-training Framework on Top of Unified Topology
We introduce MMM, an EEG pre-training framework based on the unified topology that follows the Masked uuto-Encoder (MAE) 12 training schema, i.c., during pre- pre-training. the model first encodes partiall--maske irregular input tokens (DE features) to unified representation with an encoder and then aims to reconstruct the masked tokens from the unified representation with decoder.
Our designs of MMM to better support EEG pre-training are 3-fold: 1) Multi-dimensional positional encoding, which injects the geometric information into tokens; Multi-level channel hierarchy, where extra tokens representing aggregated regions are added to the original set of channels. Tokens and channels interact hierarchically; 3) Multi-stage pre-training, where different masking strate- gies, global random masking and regional masking, are used alternatively to learn the hierarchical representations effectively.
3.3.1 Multi-dimensional Positional Encoding
Traditional positional encoding only considers injecting sequence order information. Nevertheless, the injecting order of channels can be quite random and doesn't 't have actual meanings, and taking channels as simple sequence will lose the spatial information for the sensors.
As aforementioned, sensors can be conceptualized as placed of each sensor E can be represented as a 2-D coordinate (x. y). We introduce a multi-dimensional positional encoding this manifold to allow the model to render its learned representations spatially position-awe aware. The multi-dimensional positional encoding is defined as:
2-D manifold, so the actual position
Tp R(x),
Vp Rly), Encode (Ip, 2i + 1) COS
Encode(Ir, 2i) sin xy
p
52 27
a
Up
Up
Encode (yp, 21) = sin
Encode (ycode(yp:3) 2i + +1) = cos
Ω
25
Ω
PE CONCAT| i), Encode i)}-
Here, (r, y) is a 2-D coordinate from the scalp, the number of sensors, and d pairs of (z, y)
corresponding to the grid of the 10-10 system form space. Function I projects x, y to their rank

---

(ajOrigin
(b)Random
(eJRegion-wise
Examples of random mask and region- wwie mask. For the region- wise mask strategy, channels in the same group will be all visible all masked.
Figure 2:
value in its original space. The i is the index of feature dimension in the encoding, and ft is large constant (normally set to 10000). The final encoding is generated by concatenating the encoding of the x and y coordinates.
The multi-dimensional positional encoding is used in two places. Firstly, it is added after the DE features from different channels are processed through a linear layer, thereby injecting geometric information into the data. Secondly, the encoding is re-applied after obtaining the representations. Position encoding is performed on the re-masked representations to inform the decoder how to distinguish re-masked tokens.
3.3.2 Multi-level Channel Hierarchy
Overall, we employ a transformer-based encoder-decoder as the backbone model of MMM under the masked auto-encoder schema. The encoder-decoder processes representations of tokens with multi-head attention for several rounds. Besides the origin nodes from existing channels, set of region-wise tokens accompanied model the regions of the unified topology.
Encoder Our encoder comprises a stack of Transformer blocks. Under the pretext of reconstruc- tion, we fraction of all tokens zero and then attempt reconstruct them with the remaining tokens. We introduce region-wist tokens $ 17x and append them to the end of the sequence. This process allows extracting local information from the original channels through the attention mechanism, as shown in Figure (a). In order to tackle potential redundancy that could arise during the interaction of region-wise tokens with nodes from other regions, assign fixed attention mask to the encoders/decoders This ensures precise messages pass between the region-wise nodes and their corresponding origin nodes, as well as amongst region-wise nodes and origin nodes. example of an attention mask can be seen in Figure I (d). The encoder maps a sequence of tokens x E (C+17)xF to Z E (C++7))x where the last 17 tokens form the hidden representation.
Decoder Our decoder is symmetrical to encoder with N Transformer blocks. Tokens from the origin EEG channels are re-masked, and the decoder can only access information contained in region-wiss wise nodes. The bottleneck design forces the encoder to model EEG well with the unified topology.
3.3.3 Multi-stage Pre-training
MMM reconstructs the inputs and predicts the DE value of EEG channels, which consists of the previous C decoder's output tokens. We use the mean square error (MSE) loss to measure the difference between the predicted DE value and the ground truth.
As a new hierarchy is introduced and only the information from region-wise nodes is used to reconstruct the origin nodes, the unified representation must be capable of both intra-region and inter-region reconstruction.
the one hand, every vector in the representation the unified topology contains aggregated information about specific region and serves to reconstruct fine-grained features within the region. On the other hand, the interactions between different regions should also be learned in case information from one region is completely missing. Therefore, to further boost the training of
5

---

the representations, we introduce the following two masking strategies accordingly and use them in turns during pre-training, as seen in Figure 2 and we call it multi-stage pre- pre-training.
Global Random Masking Channels are masked randomly and independently, and the number of masked channels is fixed. This provides the chance for the region-wise nodes to see partial information within the region and use it to reconstruct the masked intra-region channels.
Regional Masking Channels within the same region are either fully observable completely masked, and the number masked regions is fixed. In this way, we force the masked regions to learn representations from region-wise nodes of other regions since they are longer directly linked to any observable origin nodes with attention.
3.4 Apply MMM to Downstream Tasks
To apply MMM downstream tasks, such as the emotion recognition task, we extract the unified representation of the DE feature and feed it into a 2-layer MLP. As classification task, the cross- entropy loss calculated to fine-tune the model. It's noteworthy that the unified representation, which is a hidden vector, can serve in several types of downstream applications.
Experiments
In this section, we first introduce the datasets we used and the overall settings of our experiments. Then, we compared our model with series of competitive baselines under different settings to demonstrate the strong modeling and transferring ability between different datasets our proposed MMM framework. We also include ablation studies investigate the contributions of the designs separately and showcase the effectiveness of the unified representation for EEG modeling.
4.1 Datasets and Experiment Setup
SEED SEED [ is a discrete EEG emotion dataset elicited by videos. The SEED dataset contains EEG signals of fifteen subjects (eight females and seven males) recorded by 62 EEG sensors while watching fifteen Chinese film clips for three types of emotions. For each subject, the EEG signal recording process is repeated in three different periods corresponding to three sessions. Each session includes the EEG signals while watching the fifteen film clips corresponding to fifteen trials.
SEED-IV SEED-IV dataset [27 is another version of the SEED dataset. It comprises EEG data of fifteen subjects (eight females and seven males, not overlapping with SEED) recorded in 62 channels. Each subject has three sessions, and each session contains 24 trials.
SEED-Union We combine the SEED and SEED-IV datasets to form the SEED-Union dataset, which is only used for pre-training. Its train set consists of the training samples in SEED and SEED-IV and so as the test set. There is no label in SEED-Union since it's only for the reconstruction pre-training task.
Lite series We evenly select 32 channels from the 62 channels in SEED and SEED-IV datasets to form the SEED-Lite and SEED-IV-Lite datasets. The selected 32 channels are evenly distributed on the scalp. Then, we combine the SEED-Lite and SEED-IV-Lite datasets to form the SÈiDD-nnon-Lite dataset, which is also only used for pre-training.
TUEG TUEG 28 dataset consists of clinical recordings using mostly conventional recording configuration (monopolar electrodes in a 10-20 configuration, sample rate varying from 250Hz to 1024Hz) of 14,987 people for 27, 27,063 hours in total. The subjects were 51% female, and ages ranged from under year old to over 90.
Experiment setup Previous baselines on SEED series datasets all use public DE features, while SEED and SEED-IV provide DE features with different STFT window sizes. It's more reasonable to pre-train the model on the DE features with the same STFT window size. Unless specifically mentioned, the experiments of our model are conducted on the 1-second DE features on both SEED and SEED-IV, Refer to the Appendix B.2 for baseline descriptions and training details.

---

4.2 Subject-dependent Classification
We compare our models different settings on SEED and SEED-IV with established baselines, and the results are shown in Table
Table 1: Comparison of our method with baseline methods on SEED and SEED-IV dataset on the subject- dependent classification task. Window size is the length of features to construct the input. For example, 4sx 5 means every DE feature aggregates 4-second model imput. dagger symbol (1) represents the exclusion of worst worst-performing session out three for each subject when calculating the final average accuracy.
EEG signals, and five consecutive DE features used the
Window Size
Accuracy (%) Std (%)
Method
Pre-trained
SEED
SEED-IV
SEED
SEED-IV
STRNN
X
Isx9
99.50/7.63
DGCNN
X
1sx1
4sx
90.40 8.49
69.88 16.29 f
BiDANN 23
X
1sx1
4sx1
92.38 7.004
70.29 / 12.63
BiHDM 29
X
Isx1
48x
93.12/6.06
74.35 / 14.09
R2G-STNN 15
Isx9
9
RGNN 24
1sx1
4sx1
94.24/
79.37 / 10.54 f
MD-AGCN 2
1sx5
4sx5
94.81 /4.52
87.63 5.77
MV-SSTMA
1sx
10
4sx
95.32 3.05
92.82 / 5.03
V-IAG 30
X
Isx1
95.64 / 5.08
HDGCN
X
Isx1
96.40 /.54
MAE 12
X
Isx1
4sx1
87.89 8.06
78.22 14.21
MAE
Isx1
4sx1
91.85 7.72
81.09/ / 10.96
MMM (ours)
x
Isx1
1sx1
94.59 / 6.49
86.04 / 10.04
MMM (ours)
Isx]
1sx1
95.76 5.53
87.92 9.24
MMM (ours)
1sx1
1sx1
97.80 3.28
91.78 / 7.05
MMM (ours)
4sx
93.62 / 7.74
Although method uses only one-tenth of the information, method outperforms the previous state-of-the-ard self-supervised learning methods, achieving a performance enhancement of 0.44% on the SEED dataset and 0.77% on the SEED-IV dataset. When compared with the supervised methods, our model exhibits significant improvement, yielding a 1.52% increase in performance on the SEED dataset and substantial 14.25% enhancement the SEED-IV dataset, mention their setting to exclude the worst-performing session.
After pre-training, MAE and MMM get accuracy increase of 3.96% and 1.17% on SEED/SEED respectively, confirming the significance of EEG pre-training. Moreover, our proposed MMM framework, though following the MAE training schema, outperforms MAE by 3.91 22/11 % on SEED/SEED-IV. This improvement suggests the effectiveness of our specific designs for EEG pre-training.
4.3 Transfer between Different Datasets
In this section, we examine the models' transferability across datasets with identical sensor configura- tions. We pre-train models on the original dataset, including SEED, SEED-IV, and their combinations. Subsequently, the fine-tuning performance of these pre-trained models is evaluated the target datasets, SEED and SEED-IV.
As Table demonstrates, all pre-trained variants outperform the corresponding baseline model trained from scratch. Interestingly, the model pre-trained on the SEED-Union by MAE only gets the second-best fine-tuning performance on both datasets. This indicates that even if the channel configuration is identical, the underlying difference the data distribution among datasets may hinder the model without unified representation from taking advantage of more data resources. However, while models pre-trained by MMM and MAE share the same architecture, the trend reversed for models pre-trained by MMM. The model pre- pre-trained with the SEED-Union dataset exhibits the
7

---

Table 2: Transfer study: Models get pre-trained the origin dataset, (-) denotes models get trained from scratch. Then we fine-tune and test the models on the larget dataset. The results each method and target dataset have been marked in bold.
Accuracy (%)
Pre-training Dataset
SEED
SEED-IV
MMM
MAE
MMM
MAE
94.59
87.89
86.04
78.22
SEED SEED-IV SEED-Union
95.15
90.96
87.25
81.09
95.35
91.85
86.85
79.15
95.76
90.59
87.92
80.50
best downstream performance both datasets. This indicates that method, built the unified topology, can better use a larger number of data to extract more generalizable representations. For more training details, refer the supplementary.
4.4 Transfer between Different Montages
We further investigate the transferability of our method across different sensor configurations this section. We pre-train models using the Lite series dataset and then conduct partial fine-tuning the full SEED dataset, i.c., the parameters the encoder are frozen, and only the last MLP is tuned. As reflected in Table the model pre-trained by MMM both the SEED-Lite and SEED SEED-Union- Union-nionn datasets outperforms the model trained from scratch. However, the model pre-trained by MMM on SEED-IV-Lite shows performance drop. We suggest that the SEED-IV, with only 32 channels, might present a substantial data shift, making it challenging for the pre-trained model to fine- aine-une/parttillyy fine-tune. However, all models pre-trained using the Mask Autoencoder (MAE) fail match the partially fine-tuning performance of the model trained from scratch. On the other hand, the model pre-trained using the Multi-modal Multi-task (MMM) approach on the SEED-Union-LE showcases superior performance. This suggests again that the incorporation of the unified topology aids in the extraction more generalized representations.
dataset
Table 3: Partially fine-tuning results SEED dataset.
MAE
MMM
Pre-trained Dataset
Acc. (%)
Change
Acc. (%)
Change
SEED (Finc-tunc)
91.86
95.29
79.82
-12.04
92.54
-2.75
76.07
-15.79
93.23
-2.06
SEED-Lite
74.37
-17.49
92.70
-2.59
SEED-IV-Lite
75.80
-16.06
91.92
-3.37
SEED-Union-LLe
75.05
-16.81
92.81
-2.48
4.5 Transferred from large-scale EEG corpus
Table 4: Large-scale pre-training: Models get pre-trained on TUEG dataset, (-) denotes models get trained from scratch. Then, we fine-tune and test the models the SEED datasct.
Pre-training dataset
epoch
Fine-tuning Accuracy (%)
94.59
TUEG
94.85
TUEG
95.11
SEED
200
95.15
TUEG
95.29
In light of the preceding investigations discovering the effectiveness of our methodology across varying datasets and montages, proceeded to deploy approach on TUEG dataset large-

---

scale EEG corpus. Notably, although TUEG dataset collects data on a distinct channel configuration involving 21 channels in the 10-20 system, its utilization pre-training remains beneficial. As shown in T there is discernible increment in the downstream classification accuracy as the number of pre-training epochs increases. Remarkably, when leveraging TUEG for pre-training, our method outperforms self-pre-training within just three epochs. These findings may illuminate the path toward constructing foundation models in the EEG domain.
4.6 Impact of Region Division
Table 5: Results from variants of different region division selection
Number of Regions
Accuracy
93.95
93.41
7
93.68
(chaos)
93,96
17 (full attention)
94.08
94.53
17 (default)
94.61
In this section, we thoroughly analyze the impact of different unified topologies in MMM on the
SEED dataset. We compared default 17-region setting with several variants. 1) single region for all channels; 2) 4 regions shown in Figure (b); 3) 7 regions shown Figure (d); 4) 62 regions, each channel taking up an individual region: 5) 17 in chaos, where channels are divided
groups without considering spatial relationships. Results are shown in Table 5| From the results, we can conclude that proper choice of region setting contributes to better result. Refer to the Appendix C.I for comprehensive analysis.
4.7
Ablation Study
In this section, we scrutinize the design of MMM for both the SEED and SEED-Lite datasets. Table provides an ablation study on the contributions of different components within the MMM model.
6
Table Ablation study on different components MMM on the SEED dataset and SEED-Lite dataset.
Multi-dimensional Positional Encoding
Multi-level Channel Hierarchy
Multi-stage Pre-training
SEED
SEED-Lite
90.96
90.74
91.46
91.31
95.28
94.29
95.15
94.62
The table reveals that introducing region-wise tokens leads to the most significant enhancement in performance. This underscores the significance capturing local and regional patterns within the EEG data for effective representation learning.
Moreover, the EEG-specific position encoding (PE) also bolsters the overall performance of the model, as indicated by 0.4% rise in accuracy. This underlines the value of incorporating spatial information into the model, which aids in understanding the interrelationships among EEG channels and their respective locations on the scalp. Interestingly, pre-training with a multi-stage masking strategy slightly decreases performance on full-channel SEED dataset but boosts performance on the SEED-Lit dataset. This suggests that the multi-stage mask strategy aids in enhancing the model's generalizability across various datasets. However, may introduce slight trade-off, potentially causing minor performance decreases certain datasets.

---

Original
Mask 10%
Mask 30%
Mask
Mask
Figure 3: The topographic maps of the reconstructed test data with different mask rates in the y band. The row denotes different test samples from the SEED dataset. The first column the origin data, while the rest columns stand for the topology reconstructed by MAE pre-trained model (Left) and the topology reconstructed by MMM pre-trained model (Right) with different mask rates.
MMM
MAE
0.0015
0.7
MSE
R2
0.0010
MMM
MAE
9.3
0.9
0.7
09
Mask Ratio
Mask Ratio
Figure Reconstruction losses on the test samples SEED
4.8 EEG Reconstruction with Unified Representation
Figure presents the visualization the reconstruction quality for both the mask autoencoder and method. By analyzing the reconstruction losses shown in Figure 4 find that MMM demonstrates slightly inferior reconstruction results at a low mask ratio compared to MAE, which be attributed the amount of information they feed into reconstructors. Specifically, MAE's reconstructor can access nearly all origin tokens at low mask ratio, while MMM's reconstructor relies solely on high-level representation, forgoing all origin tokens. However, as the mask ratio increases, MAE's performance starts to collapse while MMM maintains superior reconstruction quality. This phenomenon further substantiates that our approach can effectively model EEG data by the unified topology, yielding robust representations with rich information.
5 Conclusion
In this study, we have proposed MMM, an innovative framework for EEG pre-training, which effectively navigates the complexities associated with various sensor configurations in EBG data and well encodes the spatial information of EEG. Our model employs a self-supervised learning approach based on the masked auto-encoder, allowing it to benefit from vast cross-dataset corpus and enhance the performance downstream tasks. Our framework demonstrates state-of-the-at performance in emotion recognition tasks, and further evaluations substantiate the efficacy of our methodology. Looking forward, we envision extending our approach to more downstream tasks and modeling temporal and spatial information of EEG data efficiently.
10

---

References
[1] Y. Varatharajah, M. J. Chong, K. V. Saboo, B. M. Berry, B. H. Brinkmann, G. A. Worrell, and R. K. Iyer, "Feg-graph: A factor-graph-hast model for capturing spatial, temporal, and observational relationships in electreencephalogranss" in NIPS, 2017.
[2] S. Fazli, C. Grozea, M. Danóezy, B. Blankertz, F. Popescu, and K.-R. Müller, "Subject independent ceg-based bci decoding," in NIPS, 2009.
[3] Y. Pan, J.-L. Chou, and Wei, "Matt: manifold attention network for eeg decoding," vol. abs/2210.01986 2022.
[4] R. Li, Y. Wang, W.-I. 7heng, and B. Lu, "A multi-view spectral-spatial-eempooe masked autoencoder for decoding emotions with self-supervised learning." Proceedings of the 30th ACM International Conference on Multimedia, 2022.
[5] H. J. Banville, O. Chehab, A. Hyvärinen, D. A. Engemann, and A. Gramfort, "Uncovering the structure of clinical eeg signals with self-supervised learning," Journal of Neural Engineering, vol. 18, 2020.
[6] M. Mohsenvand, M. R. Izadi, and P. Maes, "Contrastive representation learning for electroenceepaalogaam classification," in NeurlPS, 2020.
[7] S. Tang, J. Dunnmon, K. Saab, X. Zhang. Q. Huang. F. Dubost, D. Rubin, and Lee- Messer, -Sper-super graph neural networks for improved dlectrennephallgrrphicc scizure analysis," in International Conference on Learning Representations, 2022. [Online]. Available: https///paneevieie.w///tmmmmnmmd hixxki
[8] X. Jiang. J. Zhao, B. Du, and Yuan, "Self-supervised contrastive learning for eeg-based sleep staging," in 202/ International Joint Conferencee on Neural Networks (IJCNN) 2021, pp. 1-8.
[9] R. J. Davidson and N. A. Fox, "Asymmetrical brain activity discriminates between positive and negative affective stimuli human infants," " Science, vol. 218, no. 4578, pp. 1235-1237, 1982.
[10] Y.-P. Lin, Y.-H. Yang. and T.-P. Jung. "Fusion of electnoencephhloograhie dynamics and musical contents for estimating emotional responses in music listening." Frontiers in neuroscience, vol. 8, p. 94, 2014.
[11] W.-L. Zheng and B.-L. Lu, "Investigating critical frequency bands and channels for baseb emotion recognition with deep neural networks," IEEE Transactions on Autonomous Mental Development, vol. 7, pp. 162-175, 2015.
[12] K. Hc, X. Chen, S. Xie, L. P. Doll'ar ar, and R. B. Girshick, "Masked autoencoders are scalable vision learners," 2022 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15 979-
15 988, 2021.
[13] J. Y. Cheng, H. Goh, K. Dogrusoz, O. Tuzel, and Azemi, "Suhject-aware contrastive learning for biosignals." arXiv preprint arXiv:2007.0455 2020.
[14] D. Kostas, S. Aroca-Quellette, and F. Rudzicz, "Bendr: using transformers and contrastive self-supervised learning task to learn from massive amounts of ceg data," Frontiers Human Neuroscience, vol. 15, p. 653659, 2021.
[15] Y. Li. W. Zheng, L. Wang. Y. Zong. and Z. Cui, "From regional to global brain: A novel hierarchical spatial-temporal neural network model for ceg emotion recognition," " IEFF Transactions on Affective Computing, vol. 13, pp. 568-578, 2022.
[16] Y. Li, L. Wang, W. Zheng, Y. Zong, L. Qi, Z. Cui, T. Zhang, and T. Song, "A novel bi- hi-hemispheric discrepancy model for eeg emotion recognition," " IEEE Transactions Cognitive and Developmental Systems, vol. 13, pp. 2022..
2021.
[17] T. Zhang, W. Zheng, Cui, Y. Zong, and Y. Li, "Spatial temporal recurrent neural network for emotion recognition." " IEEE Transactions Cybernetics, vol. 49, pp. 839 847, 2017.
[18] T. Song, W. Zheng, P. Song, and Z. Cui, "Eeg emotion recognition using dynamical graph convolutional neural networks," IEEE Transactions on Affective Computing, vol. 11, pp. 532-541, 2020.
[19] M. Ye, L. P. Chen, and Zhang, "Hicrarchical dynamic graph convolutional network with interpretabillity for ceg-basco based emotion recognition." JEEE transactions on neural networks and learning systems, vol. PP, 2023.
[20] I.-C. Shi, Y. Jiao, and B.-I. Lu, DDifferential entropy feature for eeg-hased vigilance estimation."" " 2013 35th Annual International Conference of the IEEE Engineering Medicine and Biology Suciety (EMBC). pp. 6627-6630, 2013.
[21] R.-N. Duan, J.-Y. Zhu, and B.-L. Lu, "Differential entropy feature for eeg-based emotion classification." 2013 6th International JEEE/EMBS Conference on Neural Engineering (ÑER), pp. 81 14, 2013.
[22] R. Li, Y. Wang, and B. Lu, "A multi-domain adaptive graph convolutional network for eeg- based emotion recognition," Proceedings of the 29th ACM International Conference Multimedia, 2021.
11

---

[23] Y. Li, W. Zheng, Y. Zong, Z. Cui. T. Zhang, and X. Zhou, "A bi-hemisphere domain adversarial neural network model for eeg emotion recognition." IEEE Transactions Affective Computing. vol. 12, pp. 494-504, 2021.
[24] P. Zhong. D. Wang, and C. Miao, "Beg-based emotion recognition using regularized graph neural networks," IEEE Transactions Affective Computing, vol. 13, pp. 1290- 1301, 2022.
[25] J. Hawkins and S. Ahmad, "Why neurons have thousands of synapses, a theory of sequence memory in neocortex," Frontiers in neural circuits, 23, 2016.
[26] Rubinov and O. Sporns, "Complex network measures of brain connectivity: uses and interpretations" Neuroimage, vol. 52, no. 3, pp. 1059-10600 1069, 2010.
[27] W.-L. Zheng, W. Liu, Y. Lu, B.-L. Lu, and A. Cichocki, "Emotionmeter: A multimodal framework for recognizing human emotions," TEEE Transactions on Cyhernetics, vol. 49, pp. 1110-1122, 2019.
[28] A. Harati, S. Lopez, I. Ohcid, J. Picone, M. Jacobson, and S. Tobochnik, "The tuh ceg corpus: A big data resource for automated ceg interpretation," in 2014 IEEE signal processing in medicine and biology
symposium (SPMB). IEEE, 2014, pp. 1-5.
[29] Y. Li, L.. Wang, W. Zheng, Y. Zong. L. Qi. Z. Cui, T. Zhang, and T. Song, "A novel bi-hemispheric discrepancy model for eeg emotion recognition," JEEE Transactions Cognitive and Developmental Systems, vol. 13, 2. pp. 354 367, 2021.
[30] T. Song, S. Liu, W. Zheng, Y. Zong. Z. Cui, Y. and X. Zhou, "Variational instance-adaptive graph for eeg emotion recognition." IEEE Trunsactions Affective Computing, vol. 14, no. 1, pp. 343-356, 2023.
12

---

A DE feature extraction
The key idea of extracting DE features from BEG is calculating the accumulated power within several pre-defined frequency bands. We use x E RCC to denote the raw EEG data, where C' is the number of EEG channels and T is the number of samples (i.e., N time duration x sampling frequency). The procedure for processing DE features is defined as:
2nt
)) 1- 0, 1
w(t) = 0.5 0.5 cos
N N-T)
(1)
(t) = s(t)w(t),
(2)
X (w) = [FF((s(())
N
P(k) = (X())),, k k= 0, 0,1,
(4)
2
1 endi
Σ DE(1) = Th kestart
P(k), 1, F.
(5)
n; is the number of samples in the i-th frequency band while stort; and end; marks its start and end point. F is the number of frequency bands. Following medical practices, previous work extract DE feature on five frequency bands, i.e., ō band (1-3Hz), 0 band (4-7Hz), a band (8-13Hz), 3 band (14-30Hz), 7 band (31-50Hz). With the help of DE features, one-second-long. C-channel EEG data with hundreds of samples can be represented as a C' x 5 vector, which can help us extract and aggregate temporal information and avoid inconsistency of sampling frequency in different datasets.
B Training Details
Here, we describe the settings used for pre-training and fine-tuning/partially fine-tuning our models on SEED series datasets.
B.1 Pre-training
We use Transformer layers as encoder and symmetrical layers as decoder. The input EEG data should have 62 channels. Since the EEG data size varies greatly, we first add the channel with random noises. Then, we normalize the EEG data according to the mean and standard deviation calculated on the training set. The hidden dimension of each channel is set to 16. For multi-dimensional positional encoding, we encode x and y with dimensions, respectively, and the combinations are added the hidden vectors. For global random masking, we pick the masking ratio to 0.5. For regional masking, we pick the ratio of masking region to 0.35 for the competitive difficulty. For each 100 epochs, the previous 50 epochs take global random masking, and the rest take region- wise masking. For region division, we set 17 regions as mentioned before, i.c., 17 region tokens are appended to the input sequence and used for reconstruction.
We use NVIDIA V100 GPU train models for 600 epochs with batch size of 512. We use AdamW to optimize the network. Learning rate and weight decay are grid-searched to the optimal.
B.2
Fine-tuning
We load the weights of the pre-trained encoder and add randomly initialized classifier upon the region tokens. In the SEED series, each subject has three sessions, and each session contains multiple trials. For each session, the ahead trails are viewed as training samples, and the rest are viewed as testing samples. The training/test split is 9/6 for SEED and 16/8 for SEED-IV, i.e., for the SEED dataset, each subject contains trials as training samples and trials as testing samples. The final results come from the average of all subjects. We grid-search the learning rate and weight decay, following the same strategy in We use NVIDIA V100 GPU to train the model for 100 epochs with batch size of 32.
13

---

B.3 Partially fine-tuning / linear probing
Similar to fine-tuning, we load the pre-trained weight of the pre-trained encoder and add the classifier. However, freeze the encoder during the backward propagation conduct partially fine-tuning linear probing experiments.
B.4 Baselines
STRNN [7]: The spatial-temporal recurrent neural network is based on unified spatial-temporal dependency model that learns the information from both spatial and temporal domains of EEG signals.
DGCNN [8] Dynamical graph convolutional neural network learns the representations of EEG signals by graph convolution a dynamic way for EEG-based emotion recognition.
BiDANN 23
Bi-hemispheres domain adversarial neural network focuses discriminative features of EEG signals from both the right and left sides of the hemispheres of the brain for EEG-based emotion recognition.
Bi-Hemispheres discrepancy model investigates the asymmetric differences of the right and left hemispheres of the brain.
BiHDM T
R2G-STNN [5] A region-tooglober ppatial-tempore neural network model learns the global and regional EEG representations in both spatial and temporal aspects of EEG signals.
RGNN 24 Regularized graph neural network explores the topology of EEG channels with graph convolution.
MD-AGCN [22]:
A multi-domain adaptive graph convolutional network taking full advantage of features on different domains.
MAE 12 Masked autoencoders scalable self-supervised learners by reconstructing the missing patches in images for computer vision.
MV-SSTMA 4] A spatial-temporal masked autoencoder by reconstructing the missing channels in EEG signals.
C Analysis of Different Experiment Settings
C.1 Impact of Region Division
Table 7: Results from variants of different region division selection
Number of Regions
Accuracy
93.95
93.41
7
93.68
17 (chaos)
93.96
17 (full attention)
94.08
94.53
17 (default)
94.61
Our approach to handling diverse datasets with different montages maps different channel configura- tions to unified topology. Our paper uses a 17-region setting based on some neurological findings. However, We would like to investigate how different selection of regions may influence the model behavior built on it.
In this section, we thoroughly analyze the impact of different unified topologies in MMM on the SEED dataset. compared our default 17-region setting with several variants: 1) single region for all channels; 2) 4 regions shown in Figure (b); 3) 7 regions shown in Figure 5 (d); 4) 62 regions,
14

---

(a)17
(b)4
(c)17 in chaos
(d)7
Figure 5: Different region division
each channel taking up an individual region; 5) 17 in chaos, where channels are divided to groups without considering spatial relationship. Results are shown in Table i
From the results, we can conclude that a proper choice of region setting contributes to a better result. The selection of the number of regions is a trade-off between modeling fine-grained spatial information and aggregating local information of larger range. Therefore, in our experiments, a division with fewer regions can only gather coarse- -aariied spatial information, and more regions may fail to aggregate local information properly, which both perform worse than our default 17-region setting.
Moreover, by comparing our default 17-region setting with the chaos 17-region setting, we can that the design of aggregating spatial information is also essential to the performance. This suggests that a good unified topology does not mean inserting an appropriate amount of extra tokens but also taking locality and native functionality into consideration when dividing regions.
C.2 Impact of Masking Ratio
In this section, we investigate the impact of the masking ratio for MMM on the SEED dataset.
The global random mask rate is set to different ratios, and for regional masks, the mask ratio is set to have 15% less than the random mask ratio for a proximal difficulty. Results are shown in Table g
'Conideeiin the pre-training and downstream evaluation are time- consuminsu,nng the space of grid search is smaller for experiments in the Appendix, which may result in slightly different reported performance.
15

---

Table 8: Results from variants of different masking ratio
Random Masking Ratio
Accuracy
0.3
94.54
0.5
94.61
0.7
94,40
0.9
94.57
As we expect, lower masking ratio results in weak pretext task, as the model can easily reconstruct the signals given more visible channels, and thus its representations mask ratio = 0.9 doesn't result in a large drop in performance, unlike [12]. This suggests that higher masking ratios may be used for EEG data, it results in fewer tokens during the encoding state, which could absorb data within more configurations for pre-training.
not useful. Interestingly,
C.3 Impact of Masking Strategy
Table 9: Results from variants of different masking strategy
Random Masking
Regional Masking
Downstream Accuracy
Reconstruction loss
0.3
0.7
94,49
8.26
0.5
0.5
94.61
8.07
0.7
0.3
93.67
8. 3.12
In this section, investigate the impact of the masking strategy for MMM the SEED dataset. Results are shown in Table 9 tions.
The appropriate mix rate of mask strategies can bring better representa-
C.4 Impact of depth of encoders and decoders
Table 10: Results from variants of different transformer layers
Decoder
6
8
Encoder
93.86
93.99
94.55
94.92
94.61
94.33
8
94.17
94.52
93.95
In this section, we investigate the impact of the depth of encoder and decoder in MMM on the SEED dataset. We use encoders and decoders with different numbers of transformer layers and report the results in Table [T]
To be noticed setting in main experiment should follow the 4 for a fair comparison, which has 6 depth for encoders and 6 depth for decoders. From a horizontal perspective, the deeper decoder can help the overall auto-encoder well model the EEG data, but too many layers would take over the difficulty and understanding of the encoders. From a vertical view, the deeper encoders can bring more understanding from the pre-training data, but many layers would bring the overfitting problem.
C.5 Impact different components
To further scrutinize the design of different components, we conduct simple but effective ablation study as shown Table TT To be noticed the decrement brought by Multi-stage pre-training has been explained in Section
16

---

Table 11: Ablation study on different components of MMM on the SEED dataset. missing
means the component is
Multi-dimensional Positional Encoding
Multi-level Channel Hierarchy
Multi-stage Pre-training
SEED
X
95.12
91.93
95.28
95.15
D Broader Impact
Electroencephaloorrm (EEG) is physiological signal widely used in clinical practice and brain- computer interfaces. Instead of fitting models with limited data within a task, our method exploits abundant, applogy-agnostin and cross-tasks unlabelled EEG data. Our pre-trained model can draw unified and meaningful representations that are beneficial for various downstream tasks. Neurologists and engineers could make good use of such benefits in diagnosis and BCI applications.
While our work of EEG pre-training doesn't aim to handle specific sensitive task with negative social impacts, the technique might possibly be misused in the future with privacy concerns, e.g.. reading one's 's mind from EEG signals without acknowledgment. However, we believe that we'r re moving in the right direction toward the target of building EEG foundation model to understand EEG signals better. This is essential to brain disease diagnosis like seizures, Parkinson's 's disease, etc., and is life life-saving.
E
Limitation
Our method is designed to provide a unified solution to different EEG configurations. While requires unified modeling strategy both spatial and temporal dimensions, our focus more on the spatial information, and we handle the temporal information with parametriic solution. Our framework, however, also capable of different kinds of temporal modeling strategies, and we believe the results can be further boosted with more ad advanced technique to handle temporal information better. Also, our model is currently tested only the emotion recognition task, and we believe the framework can be easily adapted to other EEG tasks to achieve better performances with more data available for training.
17


Reviews:
[{"review_id": "JRZmVXSOZM", "reviewer": "Reviewer_rN6e", "summary": "This paper explores the application of large-scale pre-training techniques to scalp electroencephalogram (EEG) data. They leverage the abundance of unlabeled EEG data and address challenges related to sampling channel selection, structural information, and spatial information. To enable cross-dataset EEG pre-training, they propose a unified topology that maps different channel selections. They introduce MMM, a pre-training framework with multi-dimensional position encoding, multi-level channel hierarchy, and a multi-stage pre-training strategy based on the unified topology. Experimental results demonstrate significant improvements over previous state-of-the-art methods on benchmark datasets for emotional recognition.", "questions": "- I am curious on whether this pretraining method can work on different downstream tasks, such as EEG to text translation. Have the authors considered this? \n- The improvements/decrease in performance of adding Multi-stage Pre-training is seen for SEED and SEED-Lite datasets. How computationally expensive is the multi-stage pre-training? I am trying to see if this tradeoff is worth the slight, possible improvement it may give. \n- For Figure 3, the topological maps created by MAE seems to be quite different from the ground truth. Additionally, the topological maps created by MMM seems to be slightly better than MAE but still pretty different from the ground truth. Figure 4 also seems to present MMM having a better loss reconstruction than MAE, however, both seem to converge at a mask ratio of 0.5. I also want to note that Table 2 in the supplementary material, the random masking ratios seems to not have much difference for the performance. Have the authors considered trying different loss functions to observe distinct behaviors in loss? \n- Please correct me if I am wrong, but it seems like the paper does not clarify whether the results in the paper are from the averaged EEG signals across all patients or an average of the results for each individual patient. Would the authors be able to kindly clarify a bit more?", "limitations": "The authors adequately addressed the limitations of the paper.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The authors present a novel pretraining method for cross-dataset pretraining on EEG signals, which proves to be an effective method for improvements for emotion recognition. This framework seems to be promising in providing researchers in the EEG space with larger datasets for pretraining.", "weaknesses": "- Although the authors present a pretraining method for cross-dataset EEG signals, they only combine the SEED and SEED-IV datasets. It would be extremely enlightening if the authors combined N > 2 datasets to see if their pretraining framework generalizes across them. \n- I am a bit confused on how the emotion recognition accuracy can be so high while the loss seems to not converge well and topological maps (Figure 3) seem to not be reconstructed well. Would the authors be able to clarify this phenomenon?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments and suggestions that truly enhanced the quality of our paper. \n1. **Cross-task challenge (W1, Q1)**: We have handled the cross-montage problem. But it is still non-trivial to do broader cross-task transitions because of challenges such as inconsistent time scope, task-specific feature pre-processing, etc. Some of these challenges are beyond the scope of this paper. \n  Despite this, we still managed to pre-train our model on a large-scale EEG Corpus, the TUH EEG Corpus (TUEG) [1], designed for several medical tasks, including seizure detection, slow wave detection, etc., and fine-tune it on the SEED dataset. To be noticed, we maintain a similar experiment setting for the pre-training and fine-tuning, except for pre-training with 21 channels in a 10-20 system. The fine-tuning result on SEED (62 channels) is shown in the following table. Even though there are huge gaps in sampling devices, subjects' physiological status, and so on, we outperform the model pre-trained on the downstream dataset by bringing massive pre-training data (~1.7T).\n  This not only suggests that our pre-training framework can generalize among >2 datasets but also show its effectiveness in leveraging large-scale resources with different channel configurations to boost the performance further.\n  \n  | Pre-training dataset (pre-training epoch) | Fine-tuning Accuracy on SEED (%) |\n  |-----|-----|\n  | random initialization | 94.59 |\n  | TUEG (1 epoch) | 94.85 |\n  | TUEG (2 epoch) | 95.11 |\n  | SEED | 95.15 |\n  | TUEG (3 epoch) | 95.29|\n\n\n2. **Regarding Figure 3 and Figure 4, Reconstruction convergence (W2, Q3)**: To clarify, reconstruction error (Figure 3, 4) is used to measure the reconstruction quality during the *pre-training* stage, and the emotional recognition accuracy is used for downstream task performance after *fine-tuning*. Despite expected correlations between them, the pre-training score does not directly account for the downstream task performance. \n  As for the convergence problem, Figure 4 demonstrates the reconstruction loss at different experiment settings during the pre-training stage. Combined with Figure 3, we are showing MMM has a more robust EEG understanding ability than MAE by verifying their abilities to reconstruct corrupted EEG segments of different mask ratios, and all the reported results are from converged checkpoints. The reviewer might use 'not converge well' to describe the high loss. We further use the R2 score to demonstrate the reconstruction quality in Figure 3 of the attached PDF file in the global response. Moreover, because of the possible gap between pre-training and downstream tasks, we assume that over-chasing low reconstruction loss would not bring better downstream performance.\n\n3. **More downstream tasks (Q1)**: Good point! Our method should work on different downstream tasks with different channel configurations. However, it is not yet trivial to do broader cross-task transitions because of the challenges such as inconsistent time scope, feature pre-processing, etc. Take the EEG-to-text translation task as an example. Previous work [4] uses eight frequency bands to extract EEG features on ZuCo [5, 6] dataset with inconsistent lengths between EEG segments and textual sentences, while our experiments on SEED use five frequency bands with fixed lengths of time per sample. \n  Nevertheless, such EEG tasks are still in the scope of EEG understanding, which will be a good future work with the help of our unified hierarchical representation. \n\n4. **Details about Multi-stage pre-training (Q2)**: Multi-stage pre-training is a strategy that alternatively uses different masks in different iterations, and the global random mask and region-wise mask have the same computation cost. Since the total number of training iterations is kept the same during pre-training, there is no trade-off for additional computations.\n\n5. **Regarding Table 2 in the supplementary material (Q3)**: The result of Table 2 in the supplementary material helps confirm our assumption that the different pre-training setting (mask ratio) does not bring too much difference in the downstream task (See Response 2: Reconstruction convergence), i.e., with the help of region-wise tokens, our framework is more robust to high mask ratios. \nAnd we will consider your suggestion of bringing more loss functions in future work.\n\n6. **Way to get results (Q4)**: We follow the experiment setting of the previous work [2, 3], where the results are from the average results of all patients. We will clarify the details of the dataset in the revised version. \n\n[1] Harati A, Lopez S, Obeid I, et al. The TUH EEG CORPUS: A big data resource for automated EEG interpretation[C]//2014 IEEE signal processing in medicine and biology symposium (SPMB). IEEE, 2014: 1-5.\n\n[2] T. Song, W. Zheng, P. Song and Z. Cui, \"EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks,\" in IEEE Transactions on Affective Computing, vol. 11, no. 3, pp. 532-541, 1 July-Sept. 2020, doi: 10.1109/TAFFC.2018.2817622.\n\n[3] Rui Li, Yiting Wang, Wei-Long Zheng, and Bao-Liang Lu. 2022. A Multi-view Spectral-Spatial-Temporal Masked Autoencoder for Decoding Emotions with Self-supervised Learning. In Proceedings of the 30th ACM International Conference on Multimedia (MM '22). Association for Computing Machinery, New York, NY, USA, 6–14.\n\n[4] Wang Z, Ji H. Open vocabulary electroencephalography-to-text decoding and zero-shot sentiment classification[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2022, 36(5): 5350-5358.\n\n[5] Hollenstein N, Rotsztejn J, Troendle M, et al. ZuCo, a simultaneous EEG and eye-tracking resource for natural sentence reading[J]. Scientific data, 2018, 5(1): 1-13.\n\n[6] Hollenstein N, Troendle M, Zhang C, et al. ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation[C]//Proceedings of the 12th Language Resources and Evaluation Conference. 2020: 138-146."}, {"review_id": "iZu55Yx60B", "reviewer": "Reviewer_tt92", "summary": "This work seeks to improve classification tasks over EEG brain data using pre-training. There exist many EEG datasets, but not all datasets use the same montage format. To leverage all the datasets together, this work presents an approach for learning generic representations of EEG data. This approach proceeds in two stages. In the first stage, EEG data is given as input to a Transformer-based autoencoder. The bottleneck for this autoencoder is a small set of region-nodes, which are meant to represent the regions of the brain. In the second stage, the representations for these region nodes are used for fine-tuning on a classification task. The benefit of this method is that the region representation is agnostic to montage format.", "questions": "- In general, how do you think this method would perform in the case where you have two datasets, but the set of regions present in each dataset is different? For example, dataset A might contain recordings for regions 1, 2, and 3. And Region B might only contain recordings on regions 4, 5, and 6. \n- Can you say briefly what the main difference between MAE and MMM is?\n- On line 151, it mentions the (x,y) coordinates of the nodes. What are these coordinates relative to? Every scalp is different, so were projections made to a common scalp?\n- How is the Lite series created? How are the 32 channels selected? Are they the same 32 channels for every subject?\n- On line 233, why does it say that the method only uses one-tenth of the information?\n- I'm not sure how to understand the experiments in section 4.2? Are these the results when training is restricted to a single subject? Which subject?\n- Why are there two seemingly identical models in Table 1? I'm looking at the third-from-last and second-from-last rows?\n- In table 1, what is the standard deviation taken over?\n- I'm not sure how to read the text in section 4.4 and table 3. What does the \"SEED (Fine-tune)\" row denote? Aren't the other models also finetuned on SEED? \n- On line 261 it says that \"We pre-train models using the Lite series dataset\", but then table 3 seems to suggest that different pre-training datasets are used. For what models was the Lite series used?\n- The text in 4.4 makes comparisons to the \"model trained from scratch\" but all the rows in the table show deltas with respect to \"SEED (Fine-tune).\" I suggest showing the deltas with respect to the model trained from scratch.\n- On line 265, it says that the performance drop is due to the fact that SEED-IV-Lite only has 32 channels. But isn't this the case for SEED-Union-Lite dataset, which shows an increase in performance?", "limitations": "Limitations and potential social impacts are not discussed. But for this work, I think this is alright.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- This work sets out to address an important question: how do we handle the fact that there is a lot of EEG data out there, but not all of the same montage format? Can we pre-train over all this data at once?\n- The presented method is a reasonable first approach to this problem.", "weaknesses": "- My main concerns are clarity and the significance of the results\n- The aim of this work is to show the ability to transfer between different datasets, but as described in 4.1, all the datasets considered seem to have a very similar source. The most different datasets, the Lite datasets, are created by extracting data from the other datasets being considered. It would strengthen this work if a wider variety of sources was considered. Or perhaps it could be better explained why the present datasets are more different than they might appear.\n- The results presented in 4.5 show inconsistent and small improvement when transferring between montages.\n- I have some doubts about the underlying theory in this work. It's claimed that there is a hierarchy in the representation, because independent region nodes are given as input to the model. But there is nothing about the architecture that constrains region information to pass through the region nodes. \n- There are many small typos throughout. For example, sentence 1 of the abstract should probably read: \"Large-scale pre-training has shown great potential to enhance models on downstream tasks in vision and language\". Sometimes these typos impede clarity. For example, line 176 refers to a \"symmetry stack\", and it is unclear whether this denotes a specific type of transformer, or if it just means that the decoder is symmetrical to the encoder. Lines 38 and 39 have similar typos that make reading difficult. I definitely don't mean to pick on the English writing, and overall the sentences are mostly clear, but I think the clarity would be greatly improved if it was given a close reading for typos.\n- The ablation study doesn't contain an experiment where the positional encoding is ablated. It would be useful to see this, since it's a key part of the method.\n- Similarly, no experiment is done where Multi-level channel hierarchy is the only component missing\n- I have other concerns about clarity (see questions below)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable suggestions, which help us enhance the clarity of this paper and make it more understandable for the wider community. We reviewed your suggestions, and our manuscript to ensure all typos, vague descriptions, and unclear settings we can find (W1, W5, Q4, Q6, Q8) are properly addressed in the revised version. Please kindly refer to the global rebuttal for more details. \n\nFor individual concerns, we respond as follows:\n\n1. **Wider variety of sources (W2)**: SEED and SEED-IV contain different subjects and stimulating materials, which are non-trivial within the EEG area. SEED and SEED-Lite have different montages. Although they're designed for the same task, as we know, there is no relevant work on these two kinds of transition. \nDespite challenges, we pre-train our model on a large-scale EEG dataset, the TUH EEG Corpus [2]. We maintain a similar experiment setting for the pre-training and fine-tuning, except for pre-training with 21 channels. Please kindly refer to the global response for details and results.\n\n2. **Inconsistency and small improvements in section 4.5 (W3)**: The ablation study is based on the setting, pre-training on SEED / SEED-Lite, and fine-tuning on SEED. For inconsistency, it suggests that the multi-stage mask strategy aids in enhancing the model's generalizability across various datasets (Lines 282-285). As for the number, a 3.88% downstream accuracy increment is shown for MMM pre-trained on SEED-lite with full components to the one with no component. We beg to differ in that it is not a small improvement.\n\n3. **Hierarchy and constraints (W4): Hierarchy consists of region-wise nodes**, EEG channels, and their communications. We have constrained the region-wise node not to interact with EEG channels out of its region. Compared with constraints on cross-region channel interaction, the scheme in the paper can model the EEG signal more efficiently without smashing the hierarchical structure.\n\n4. **Typos (W5)**: Thanks again for pointing out the typos. The phrase \"symmetry stack\" means \"symmetrical to the encoder\".\n\n5. **Comprehensive Ablations (W6, W7)**: We validate the effectiveness of positional encoding by comparing the second row in Table 4. A similar comparison can be made between the second and third rows for multi-level channel hierarchy.\nAdding multi-stage pre-training without a multi-level channel hierarchy is not very sensible. However, we conduct additional experiments for rigor, as shown in the table.\n  \n  |Multi-dimensional PE | Multi-level Channel Hierarchy| Multi-stage Pre-training| Accuracy on SEED (%) |\n  |-|-|-|-|\n  |1|0|1|91.93|\n  |0|1|1|95.12|\n  |1|1|1|95.15|\n\n6. **Non-overlapping case (Q1)**: It is right to worry that our method cannot handle this extreme case without overlapped channels. But most datasets have channels uniformly distributed on the scalp, such as TUEG [2], Sleep-edf [3], etc. And the majority use subsets of the 10-20/10-10 international system [1]. Even though the extreme case happens, we could practically introduce the C dataset, which overlaps A and B.\n\n7. **Main difference between MAE and MMM (Q2)**: Table 4 has shown that the main difference between MAE and MMM comes from a multi-level channel hierarchy, i.e., 1) replacing the class token with region-wise tokens for a spatial-aware representation (Line 168-169). 2) reconstruction with region-wise tokens instead of the visible channels to enforce the capability of the representation (Line 176-178).\n\n8. **Coordinates for position encoding (Q3)**: (x,y) coordinates are relative to the channels' position in the 10-10 international system [1] (Line 152). Thanks to the unified system, most EEG-related research does not worry about different scalps. \n\n9. **Creating Lite series (Q4)**: Lite series is created by evenly selecting 32 channels from 62 channels. The selected channels remain the same across subjects. \n\n10. **One-tenth information (Q5)**: MV-SSTMA uses ten-timestep information, while we only use one timestep (shown in Table 1 \"Window Size\").\n\n11. **Subject-dependent Classification (Q6, Q8)**: We follow the previous work [4, 5] to perform subject-dependent classification. Each subject has 15 recorded trials with emotion labels. The first 9 are for training, and the rest are for testing. Each subject has its own training and test sets, and the reported results are the average for each subject. The standard deviation is taken over the result from each subject.\n\n12. **Seemingly identical models (Q7)**: The difference is the dagger symbol (†), representing excluding the worst-performing session out of three for each subject when calculating the accuracy (refer to the caption of Table 1). This brings fair comparisons following different previous settings, i.e., results with/without the dagger symbols are directly comparable.\n\n13. **Explanation on Section 4.4 (Q9~11)**: Section 4.4 compares models with different aspects: 1) method (MMM or MAE). 2) fine-tuning setting (fully or partially fine-tuning). 3) pre-training dataset (none, SEED, Lite series). The \"SEED (Fine-tune)\" row denotes the model pre-trained and fine-tuned on SEED. The rest denotes the model partially fine-tuned. i.e., the parameters of the encoder are frozen, and only the last MLP is tuned (Line 262).\nWe will take your advice to show the deltas w.r.t model trained from scratch for better presentation.\n\n14. **Line 265 (Q12)**: SEED-Union-Lite combines SEED-Lite and SEED-IV-Lite (Line 223-224) and more data bring better performance.\n\n[1] Homan R W, 1988. The 10-20 electrode system and cerebral location\n\n[2] Harati A et al., 2014. The TUH EEG CORPUS: A big data resource for automated EEG interpretation\n\n[3] Kemp B et al., 2018. Sleep-edf database expanded\n\n[4] Song T et al., 2020. EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks\n\n[5] Li R et al., 2022. A Multi-view Spectral-Spatial-Temporal Masked Autoencoder for Decoding Emotions with Self-supervised Learning."}, {"review_id": "gz8fkp7btl", "reviewer": "Reviewer_JXz6", "summary": "The paper proposed an innovative approach to the pre-training of models for the EEG data. Large-scale pre-training which demonstrated great potential in CV and NLP requires a substantial amount of data. While EEG data is relatively easy to collect, their interpretation and labelling often require substantial expert efforts. Although the integration of various datasets can address this concern, the differing electrode configurations among these datasets can result in domain shifts and dimension mismatches.\n\nThe paper introduces a framework named MMM, which stands for Multi-dimensional position encoding, Multi-level channel hierarchy, and Multi-stage pre-training strategy. The novelty lies in the mapping of all EEG channel selections onto a unified topology, enabling the development of a pre-training framework to learn unified, geometry-aware EEG representations that can generalize across different EEG channel configurations.\n\nIn their approach, the authors encode the spatial information into the representation and develop a method that allows pre-training with an EEG corpus having various sensor configurations. This is achieved through the concept of region-wise tokens, which form a multi-level hierarchy learning from the EEG channels. These tokens eventually form a unified representation that can be applied to downstream tasks.\n\nAdditionally, the paper proposes a multi-dimensional position encoding technique to encapsulate geometric sensor information and a multi-stage mask strategy involving random and region-wise masks designed to enhance the robustness of hidden representations.\n\nThe authors validate their framework against a wide range of state-of-the-art methods on EEG emotion recognition tasks. Their experimental results indicate that the proposed method not only achieves state-of-the-art performance but also exhibits a strong ability to transfer across different datasets, even those with differing montages. The author also conducted an ablation study, underscoring the validity and robustness of each proposed component, contributing to the MMM framework's overall effectiveness.", "questions": "1. For the Reconstruction experiment, is the mask uses global random masking or regional masking? \n\n2. Will the code be made available?", "limitations": "please refer to the weaknesses ", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This manuscript makes an innovative contribution to the field by addressing the crucial issue of sensor configuration heterogeneity, which is a significant barrier to enabling large-scale pretraining using multiple EEG datasets. The novelty of the approach opens a plethora of opportunities for future research in EEG decoding and is commendable.\n\n2. The presentation is excellent, with cohesiveness that makes the paper feel complete and thorough. All elements, from figures to text, complement each other well, creating a pleasant reading experience. The figures are particularly well designed, effectively illustrating the complex method in an understandable way. The choice of colour scheme enhances the visual appeal and readability of the work.\n\n3. The manuscript's strength also lies in the clear articulation of motivation and contributions in the introduction. The narrative throughout the methodology section is seamless, explaining the proposed method with great clarity and depth.\n\n4. The rigorous experimental validation of the findings is highly appreciated. The authors included comprehensive experiments covering classification tasks with various state-of-the-art methods based on RNN, CNN, and GNN. The work's transferability across different datasets and electrode configurations is also well demonstrated. In addition, the authors performed a detailed ablation study to validate the significance of each proposed component, further strengthening the work.\n\n5. The inclusion of qualitative results, such as the topographic plot of the reconstructed data, is valuable. The results demonstrated the superior performance of the proposed MMM method in reconstructing missing channels at high mask ratios. \n\n6. Overall, this paper makes a significant contribution to the field and is likely to inspire future work. The quality of presentation, the novelty of the approach, and rigorous experimentation all stand out as notable strengths of the work.", "weaknesses": "1. While the overall design and information presentation in Figure 1 is commendable, it might be beneficial to consider further optimizing the text sizes for enhanced readability. There are noticeable empty spaces that could potentially be utilized to enlarge the text, improving the figure's overall clarity and effectiveness.\n\n2.  The manuscript could benefit from additional clarification regarding the reconstruction experiment, particularly in terms of the masking method employed. Are global random masking or regional masking methods used, or perhaps a combination of both? Exploring potential performance differences between the two masking methods would be an insightful addition. Furthermore, it would be beneficial to provide a comparison of the Mean Squared Error (MSE) between different masking methods to better understand the relative performance of the proposed method.\n\n3. When considering the transferability experiments across different datasets and montages, it might be prudent to include the state-of-the-art methods that were employed in the regular classification task. These methods can be trained on the common channels between different datasets, aligning with conventional practices in the literature. Additionally, incorporating established EEG transfer learning techniques as comparisons for these experiments could offer a more comprehensive view. This inclusion will make the results more comparable to existing literature and potentially broaden the generalizability of the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments and suggestions that truly enhanced the quality of our paper. We are cheerful that we have provided you with a pleasant reading experience, and our contribution is well recognized.\n\nResponses to your concerns are presented as follows:\n1. **Text size of Figure 1. (W1)**: Thanks for your suggestion, Figure 1 will be revised in the next version, and a preview is available in the attached PDF file.\n2. **Details of multi-stage mask strategy (W2, Q1)**: The global random and regional masking methods are used alternatively during pre-training, whose details are in the supplementary (Line 20-23). We will clarify this setting in the manuscript for higher readability.\nMoreover, thanks for the suggestion on adding the ablation study that uses MSE to measure different masking strategies while we are now using downstream task performances. (Supplementary Material section 3.3). The results are shown in the following table and confirm our statement that the appropriate mix rate of mask strategies can bring better representations (Line 90 in Supplementary Material). We will add these experiments to the revised version of our supplementary file.\n    | Ratio of Random Masking |Ratio of Regional Masking| Mean Square Error($\\times10^{-4}$) |\n    |----------|----------|----------|\n    |0.3|0.7|8.26|\n    |0.5|0.5|8.07|\n    |0.7|0.3|8.12|\n    \n3. **Baseline trained on the common channels (W3)**: It is inspired to train models on the common channels to avoid inconsistent channel configurations. Previous work [1] has investigated the effectiveness of focusing on common channels. We also conduct more experiments with MAE, MMM, and DGCNN, as shown in the following table. To simplify the table, the [Pre-train, Fine-tune, Test] columns denote the number of channels used in the corresponding stage. The results show that the information loss caused by focusing on the common channels can not be ignored. With more tasks, the common channel would become less, and more information loss would occur. That is one of the motivations for our method, which can handle different channel configurations.\n    | Method |Pre-train|Fine-tine/train |Test| Accuracy (%)|\n    |----------|----------|----------|----------|----------|\n    |MMM|/|62|62|93.76|\n    |MMM|62|62|62|94.61|\n    |MMM|32|62|62|93.97|\n    |MAE|/|62|62|87.89|\n    |MAE|/|32|32|83.25|\n    |DGCNN|/|62|62|90.04|\n    |DGCNN|/|32|32|83.25|\n\n4. **Code availability (Q2)**: We will publish both the code and the pre-trained model upon acceptance.\n\n[1] Kostas D, Aroca-Ouellette S, Rudzicz F. BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data[J]. Frontiers in Human Neuroscience, 2021, 15: 653659."}, {"review_id": "iYYVmJsY5n", "reviewer": "Reviewer_JKwr", "summary": "This paper aims to provide new architecture that perform pre-training for scalp EEG to utilize the large-scale unlabelled data. However, one of the challenge of performing such pre-training is the different sampling channels selection & inherent structural and spatial information across different EEG datasets. Thus, this paper aims to break such boundaries by mapping different channel selection setup into a unified topology/spatial map of EEG electrodes. Based on this inspiration, the paper proposed MMM as a pre-training strategy with multi-dimensional position encoding and multi-level channel hierarchy based on a unified topology to extract representation from EEG. The paper used many experiments on emotional detection to validate their proposed method.", "questions": "The experiments focused on emotional detection as the major task. While in EEG, sleep staging datasets contain much more abundant unlabelled (or labelled, in some cases) publicly available datasets that could be used for pretraining. Have the authors considered that dataset, as the proposed method is supposed to work well w cross-dataset EEG? ", "limitations": "The authors should discuss more limitations of the work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper aims to address a very important problem. The difference of channels selections for scalp EEG is one critical issue/limitation for current studies.\n2. The proposed method is intuitive but novel, the multi-scale channel hierarchy is carefully and nicely designed. It is a valid/meaningful contribution to the community.\n3. The proposed method achieves adequate performance improvements over the benchmark.\n4. The writing and organization is clear. The paper is enjoyable to read.", "weaknesses": "Minor concerns below:\n\n1. Aside from \"montage (the number and the places of electrodes placed on the scalp) and sample rate\" as mentioned in line 24-25, another issue that prevents methods from being transferrable to another dataset for scalp EEG might be the type of electrodes. e.g. wet electrodes v.s. dry electrodes, and the corresponding domain shifts of collected signals. While this is not the major issue to be tackled in this work, it'd be great if the authors can mention/discuss this factor (and other possible factors) in either the introduction or the conclusion section.\n\n2. The related work section is not 100% complete. Some arguments are not accurate and could use some edits.\n- Aside from [6] and [7], https://arxiv.org/abs/2007.04871 [Subject-Aware Contrastive Learning for Biosignals] also explored spatial augmentations for EEG data.\n- \"To the best of our knowledge, there is no method trying to use cross-dataset EEG as their pre-training corpus.\", in [BENDR: Using Transformers and a Contrastive Self-Supervised Learning Task to Learn From Massive Amounts of EEG Data] section 2.1.1 \"This also means that these data should include multiple different recording hardware and configurations.\" Although they eventually used TUEG dataset as the pre-training dataset (which is defaulted to be the same configuration, but due to the TUH data collection pipeline the actual data contain many variations), so I am not sure if this should be counted as \"cross-dataset pre-training\", this should be discussed.\n- Overall, the section \"Self-supervised learning for EEG\" should include more related works. The authors should spend more efforts doing the literature search on this front.\n\n3. The experimental details could be more comprehensive, some of the details should be included within the main text.\n- Is the train/validation/test split randomly created or based on different subjects? Correct me if I am wrong, but I did not find related details in both the main text and the appendix. Is there k-fold cross-validation for different subjects? SEED is a relatively small dataset, so I'd expect such a dataset manipulation would not be too computationally expensive.\n\n4. Minor writing issues. e.g. line 98 \"It's\" --> \"It is\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments and suggestions, which are valuable for enhancing our paper. We feel excited that our paper provided a pleasant reading experience and that our contributions are well recognized. \nResponses to your concerns and questions are hereby presented:\n1. **Discussion on more transferring issues (W1)**: Good points! It is inspired to consider the type of electrodes. Besides that, inconsistent time scope and feature pre-processing are all factors that prevent transfer between tasks. These are all the core challenges in front of the EEG pre-training area, and we will add the discussion to the limitation section in the revised version. While these challenges are beyond the scope of the paper, we hope this work can provide a powerful tool to tackle the cross-montage problem and inspire the related community.\n\n2. **Additional related work about spatial modeling with EEG (W2.1)**: Thanks for the suggestions. [Subject-Aware Contrastive Learning for Biosignals] incorporates time, frequency, and spatial augmentations on EEG data while using them on contrastive learning. The method also works on a fixed channel set, not faced with cross-montage problems. We will cite it with a few discussions.\n\n3. **Additional related work about self-supervised training with EEG (W2.2)**: Thanks for the suggestions. BENDER work on cross-dataset pre-training by selecting a 19-channel configuration, based on which they pad missing channels to 0 and drop surplus channels during the data preparation stage. Unlike our work, they have not yet tackled the cross-montage problem from the model's perspective. \n  When faced with 62-channel data (SEED series in our experiment), BENDER will only keep the 19 channels in the 10-20 system and ignores the rest. This will lead to huge information loss, and our method is working on utilizing full spatial information faced with different channel configurations. We will cite BENDER and add a few discussions in the revised version. \n\n3. **Experiment settings (W3)**: We follow the experiment setting of the previous work [2, 3], where the results are from the average results for each patient. It is noticed that all patients share the same number of samples, and samples in the first nine trials are treated as training samples, while samples in the last six trials are treated as test samples. The detail of the dataset can be referred to SEED [4], and we will clarify the details of the dataset in the revised version.\n\n4. **Text suggestions (W4)**: Good catch! We went through the whole manuscript and revised all the typos we found.\n\n5. **Larger dataset for pretraining (Q1)**: We have handled the cross-montage problem. But it is still non-trivial to do broader cross-task transitions because of challenges mentioned in Response **Discussion on more transferring issues**, such as inconsistent time scope, task-specific feature pre-processing, etc. Some of these challenges are beyond the scope of this paper. Despite the challenges, we still manage to pre-train our model on a large-scale EEG dataset, the TUEG dataset [1]. To be noticed, we maintain a similar experiment setting for the pre-training and fine-tuning, except for pre-training with 21 channels in the 10-20 international system. The fine-tuning result on SEED is shown in the following table. Even though there are huge gaps in sampling devices, subjects' physiological status, and so on, we outperform the model pre-trained on the downstream dataset by bringing massive pre-training data.\n\n    | Pre-training dataset  (pre-training epoch)| Fine-tune Accuracy on SEED(%) |\n    |-----|-----|\n    | random initialization | 94.59 |\n    | TUEG (1 epoch) | 94.85 |\n    | TUEG (2 epoch) | 95.11 |\n    | SEED | 95.15 |\n    | TUEG (3 epoch) | 95.29 |\n\n[1] Harati A, Lopez S, Obeid I, et al. The TUH EEG CORPUS: A big data resource for automated EEG interpretation[C]//2014 IEEE signal processing in medicine and biology symposium (SPMB). IEEE, 2014: 1-5.\n\n[2] T. Song, W. Zheng, P. Song and Z. Cui, \"EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks,\" in IEEE Transactions on Affective Computing, vol. 11, no. 3, pp. 532-541, 1 July-Sept. 2020, doi: 10.1109/TAFFC.2018.2817622.\n\n[3] Rui Li, Yiting Wang, Wei-Long Zheng, and Bao-Liang Lu. 2022. A Multi-view Spectral-Spatial-Temporal Masked Autoencoder for Decoding Emotions with Self-supervised Learning. In Proceedings of the 30th ACM International Conference on Multimedia (MM '22). Association for Computing Machinery, New York, NY, USA, 6–14.\n\n[4] Zheng W L, Lu B L. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks[J]. IEEE Transactions on autonomous mental development, 2015, 7(3): 162-175."}, {"review_id": "puNBW0p2y6", "reviewer": "Reviewer_v1Fw", "summary": "The work proposes a cross-dataset cross-electrode montage deep-learning pretraining method. As a model, they use a transformer that uses the electrode coordinates as positional encodings. The transformer model then processes differential entropy features per electrode as tokens with attention across all electrodes. Additionally, it has tokens for 17 predefined brain regions with attention restricted to attention between a region and electrodes within that region as well as between regions. The transformer is pretrained using a masked autoencoder framework, where the transformer predicts differential-entropy values of randomly masked-out electrodes. Masked-out electrodes are either chosen completely randomly or all electrodes of randomly chosen brain regions are masked out. The work evaluates both cross-dataset and cross-montage experiments, showing improvements from their chosen pretraining scheme.\n", "questions": "For attention, this was a bit confusing to me:\n““As a new hierarchy is introduced and only the information from region-wise nodes is used to reconstruct the origin nodes, the unified representation must be capable of both intra-region and inter-region reconstruction. “”\nAs far as I understand Fig 1.d) there is attention from all electrodes to all electrodes, from all regions to all regions, and from any region to all electrodes of that region? Sentence above made it sound like maybe first part (all electrodes to all electrodes) doesn’t exist? Also please write this explicitly in the text in 3.3.2, so it is more clear.\n\nRegarding the positional encoding, how does it relate to the positional encoding from the VIT Vision transformer? Please do mention also in the text where it is different or similar.\n\nIn Figure 3, all scalp plots of the original column look very similar to me. Could one find a more diverse set of examples?", "limitations": "Ablation study as written above might be more helpful to understand contribution of different parts.\n", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The approach has a clear motivation to do cross-montage pretraining, which is a potentially very useful capability. The method is straightforward and mostly easy to understand. Evaluation setup is interesting. Ablation studies are also useful.", "weaknesses": "I did not see how many random seeds are used to obtain the results. A lot of times accuracy differences are relatively small and may be impacted by random noise from the training process. Ideally, results should be obtained by averaging results from multiple seeds and this should be reported in the paper.\n\nAnother ablation where you add 17 artificial tokens, but they are not region-specific, so they have full attention to all electrodes and all other artificial tokens would be necessary to distinguish improvements from model capacity and region-specificness.\n\nTable 2 Pre-trained dataset should be pre-training dataset\n\nA lot of the fonts/text in figures is hard to read, especially Figure 1 d), but also others.\n\n“subjects(8 f” -> missing space on p.6 l 215\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments and suggestions, which are valuable for enhancing our paper. The following are responses to individual concerns:\n1. **Random seeds (W1)**: We follow the previous work [1,2] to perform subject-dependent classification with five random seeds to reduce the effects of random noise. We will clarify this in the revised version.\n\n2. **Comprehensive ablation study on region-specificness (W2)**: Thanks for your suggestion! Although we have investigated the region-specificness in spatial modeling by shuffling the attention between region-wise tokens and all electrodes (Table 1 in Supplementary file, 17 Chaos and 17 default), it is inspired that we can add 17 artificial tokens without any region specification to get another ablation. The result is shown in the following table (full attention). Our design still outperforms the one with full attention, further validating the effectiveness of region-specificness.\n    | Attention setting| Fine-tune Accuracy on SEED(%) |\n    |----------|----------|\n    | ours | 94.61 |\n    | chaos | 93.96 |\n    | full attention | 94.08 |\n\n3. **Text and figure suggestions (W3, W4, W5)**: Thanks for the suggestions about the text and figures. We went through our manuscript and revised all the typos we could find accordingly. Besides, we polished our figures for better readability, and you can find them in the attached PDF file. \n\n4. **Information Flow and Attention configuration (Q1)**: Our framework generally follows the encoder-decoder / encoder-classifier schema, where the multi-channel EEG data are first encoded by the encoder, then *only the region-wise nodes* are used for reconstruction / downstream task with the decoder and the classifier. The quoted sentence *\"As a new hierarchy is introduced, and only the information from region-wise nodes is used to reconstruct the origin nodes, the unified representation must be capable of both intra-region and inter-region reconstruction.\"* is used to explain how we design such a bottleneck, where we only use the unified representation from encoders to reconstruct EEG channels in decoders. As for the attention configuration, the first part (all electrodes to all electrodes) exists in the forward process of both encoder and decoder layers, though the origin nodes have been re-masked to zeros before feeding to the decoder. We will make these more explicit in the revised version.\n\n5. **Difference between positional encoding in ViT and ours (Q2)**: ViT incorporates three kinds of positional encoding: 1-D, 2-D, and relative. Multi-dimensional Position encoding is similar to 2-D position encoding, while the (x, y) denotes different things in these two concepts. The former use (x, y) to denote the physical location of EEG electrodes on the scalp, while the latter use (x, y) to denote a location in grids. We will clarify it in the revised version.\n\n6. **More diverse examples (Q3)**: We will provide a diverse set of examples in the revised version, and a preview can be referred to Figure 2 in the attached PDF file.\n\n[1] T. Song, W. Zheng, P. Song and Z. Cui, \"EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks,\" in IEEE Transactions on Affective Computing, vol. 11, no. 3, pp. 532-541, 1 July-Sept. 2020, doi: 10.1109/TAFFC.2018.2817622.\n\n[2] Rui Li, Yiting Wang, Wei-Long Zheng, and Bao-Liang Lu. 2022. A Multi-view Spectral-Spatial-Temporal Masked Autoencoder for Decoding Emotions with Self-supervised Learning. In Proceedings of the 30th ACM International Conference on Multimedia (MM '22). Association for Computing Machinery, New York, NY, USA, 6–14."}]

Author rebuttal:
Global response:
We're cheerful that the reviewers found our method novel (Reviewer rN6e, JXz6, JKwr), well motivated and reasonable (Reviewer tt92, JXz6, JKwr, v1Fw), and important (all reviewers). We're also delighted that reviewers feel reading our manuscript is pleasant (JXz6), joyful (JKwr), and easy-to-understand (v1Fw).

A major concern most reviewers share is the absence of a wider variety of sources. Even though we have handled the cross-montage problem, it is still non-trivial to do broader cross-task transitions because of some challenges, such as the type of electrodes(Reviewer JKwr), inconsistent time scope, task-specific feature pre-processing, etc. Some of these challenges are beyond the scope of this paper. Nevertheless, we still manage to pre-train our model on a large-scale EEG Corpus, the TUH EEG Corpus (TUEG) [1], and maintain the same pre-training/fine-tuning setting as in the main paper, except for pre-training models on 21 channels of EEG data. The results are shown in the following table, which further validate the effectiveness of our method on cross-montage problems.
  | Pre-training dataset (pre-training epoch) | Fine-tuning Accuracy on SEED (%) |
  |-----|-----|
  | random initialization | 94.59 |
  | TUEG (1 epoch) | 94.85 |
  | TUEG (2 epoch) | 95.11 |
  | SEED | 95.15 |
  | TUEG (3 epoch) | 95.29|

Another concern is the clarification of the paper. We thank for reviewers for their feedback. We are trying to address the concerns and polish our paper in the revised version to make our work better understandable to the wider community. Specifically:
1. We fix all the typos we found throughout the manuscript, including space before brackets, expansion of abbreviations, misspellings, etc. 
2. Figure 1 now has a better text font size. Refer to Figure 1 in the attached PDF file.
3. Figure 3 now has more diverse examples. Refer to Figure 2 in the attached PDF file.
4. We add more explicit details of our experiment setting.
5. We add discussions on the remaining challenges for cross-task EEG training to the limitation section.

Finally, we would like to express our great appreciation and excitement that several reviewers recognize our promising potential to inspire the EEG community for further exploration. We want to emphasize that our work contributes to the community not only by proposing a particular model working well with the EEG emotion recognition task but also by providing a paradigm of self-supervised learning which can tackle cross-montage problems as the first step of large-scale EEG pre-training. We believe that it is worth publishing to stimulate further discussion.

[1] Harati A, Lopez S, Obeid I, et al. The TUH EEG CORPUS: A big data resource for automated EEG interpretation[C]//2014 IEEE signal processing in medicine and biology symposium (SPMB). IEEE, 2014: 1-5.

Meta-review: Large-scale pretraining for EEG could significantly improve the accuracy and reliability of EEG. From the ML point of view, what it takes to perform well in this structured and somewhat idiosyncratic domain could lead to other advances. To that end, reviewers asked for various ablations to demonstrate which aspects of this work matter and for additional cross-dataset experiments. Authors provided such in the rebuttal satisfying the reviewers.

Reviewers were unanimous in their recommendation that this work should be accepted. Multiple reviewers would have liked to see the method applied to other decoding tasks. But all were satisfied with the current experiments. Cross-dataset demonstrations of the method were particularly convincing as were extensive comparisons against current methods.

EEG is a popular method to investigate the brain and additional tools to enhance it will find a wide audience. What is presented here is not the BERT-equivalent for EEG which the community can widely adopt, but it is a step in that direction.
Decision: Accept (poster)