SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The ERP Core dataset is predominantly time-locked with data centered around either stimulus or response<sup>3</sup>. We further evaluate the models "as is" on two other modalities of EEG data from PhysioNet [18] and with already established state of the art model results: the EEG Motor Movement/Imagery Dataset (EEGMMI) [54] and the Sleep-EDF Expanded (SleepEDFx) database [29]. The EEGMMI dataset is cue time-locked and consists of recordings from 109 subjects performing various motor imagery (MI) tasks. We applied our model to the standard L/R/0/F MI task using 3s epoch windows, closely following the approach of Wang et al. [67]. The SleepEDFx is included to explicitly probe the model integrity on data that is not time-locked to an external stimulus and contains 153 polysomnography studies from 78 subjects. Given its limited EEG channels, we adapted our methodology by considering a single EEG channel and applied a short-time Fourier transform to fit the data to the same setup as used for ERP Core. We maintained the conventional 30s time series windows commonly used in sleep stage literature. Details on the data preprocessing and model integration for these datasets can be found in the supplementary materials.
Model comparisons The proposed CLSP-AE approach is systematically compared against a mix of learning strategies comprising conventional AE-based representation learning, contrastive learning and representations obtained by supervised learning. To optimally compare these learning strategies all models are based on the same model structure given in Figure 1. Models will be denoted by which losses they are trained on, or which external methods they use. Here CSLP-AE will denote the contrastive split-latent permutation over both subjects and latents  $(L_{LP}(S;\cdot ,\cdot),L_{LP}(\mathcal{T};\cdot ,\cdot),L_{CLIP}(\mathcal{T};\cdot ,\cdot))$  and  $L_{CLIP}(\mathcal{T};\cdot ,\cdot))$ , SLP-AE the corresponding model without the contrastive loss components  $(L_{LP}(S;\cdot ,\cdot)$ , and  $L_{LP}(\mathcal{T};\cdot ,\cdot))$ , CL the contrastive loss in both of the split latent spaces  $(L_{CLIP}(S;\cdot ,\cdot)$  and  $L_{CLIP}(\mathcal{T};\cdot ,\cdot))$ . Cosine similarity will be used as the similarity metric in the contrastive loss. AE will denote the standard auto-encoder with mean-squared error reconstruction loss on the reconstructions of non-permuted split-latents. C-AE will denote the combination of reconstruction loss and contrastive learning, such that contrastive learning is applied in both spaces and the standard autoencoder reconstruction loss is applied to non-permuted reconstructions. CE will denote the supervised learning cross-entropy loss jointly trained in the subject and task latent spaces using the corresponding true labels in a supervised manner. Similarly,  $\mathrm{CE(t)}$  will denote cross-entropy only trained on the task labels in the task latent space. This is to substitute and compare with a supervised deep learning model, contrary to the self-supervised methods described here. For completeness, we also included the common spatial pattern (CSP) method [3, 32] using the multi-class generalization from Grosse-Wentrup and Buss [20] which is a supervised method for extracting discriminative features from EEG data. These features are then used to map unseen data into the same "CSP space". For the EEGMMI and SleepEDFx datasets we respectively compared the task and sleep stage classification performance to [12, 67, 73] and [14, 39, 45-47, 79].
<sup>1</sup>ERP Core info: https://erpinfo.org/erp-core. Data available at: https://osf.io/thsqg/ <sup>2</sup>See the full ERP Core procedure here: https://github.com/lucklab/ERP_CORE/blob/master/ERN/ERN%20Analysis%20Procedures.pdf
The total loss for models with multiple losses is the sum of losses with equal weighting. This is further detailed in the supplementary materials and loss curves are shown in the appendix.
Hyperparameters were chosen based on evaluation during development, and the most critical hyperparameters (the number of blocks and the size of the latent space) were verified on the evaluation set in a grid search. See supplementary material for details on the grid search across both latent dimension and number of blocks. The test set was used only for final evaluation.
Subject and task characterization Non-linear classifiers were trained to classify the subject and task labels of split-latents from the test set to quantify the disentanglement and generalization of the latent spaces. This was performed as two five (5)-fold cross-validations (CVs) over the subject and task latents respectively stratified on the subject and task labels on the test data (unseen subjects). Since there is high class-imbalance in the task labels, undersampling was performed for each class to match the number of samples in the least represented class. The undersampling was performed on the training split of each fold, and the test split was left untouched. Balanced accuracy [5] was used due to the high class-imbalance. The subject CV was used to evaluate the subject classification accuracy  $(\mathrm{S}_{-}\mathrm{acc}\%)$  and task-on-subject classification accuracy  $(\mathrm{T} - \mathrm{S}_{-}\mathrm{acc}\%)$ , while the task CV was used to evaluate the task classification accuracy  $(\mathrm{T}_{-}\mathrm{acc}\%)$  and subject-on-task classification accuracy  $(\mathrm{S} - \mathrm{T}_{-}\mathrm{acc}\%)$ . The task-on-subject classification accuracy was evaluated by training a classifier on the subject latents to predict the task of the corresponding input sample, and vice versa for the subject-on-task classification accuracy.
For each fold, an end-to-end tree boosted system (XGBoost) [6] was trained on the training split, and subsequently evaluated on the test split. Finally, the results were averaged over the five (5) folds. All classifications were performed on a single-trial level. A K-nearest neighbors (KNN) classifier and an Extra Trees classifier [16] were also trained and evaluated. See supplementary material for these.
ERPs from zero-shot EEG conversions An ERP conversion loss was measured on the test set. First, a ground-truth ERP was found for each subject and for each ERP component (task) by averaging over all samples belonging to the same subject and task only in the EEG channel respective to which Kappenman et al. [28] found the given ERP component most prominent. Let  $\hat{x}_{(s,t)}^{\mathrm{ERP}}$  denote such an ERP for subject  $s$  and task  $t$ . If disentanglement was successful then the conversion method described in Section 2.1 and illustrated in Figure 2 should be able to reconstruct the ERP from a given subject and task latent pair. Thereby it should be possible to sample an amount of subject and task latent pairs encoded on the test set, and convert the ERP from these pairs. Let S.s. and D.s. denote sampling task latents from the same or different target subject  $\sigma$  respectively, and let S.t. and D.t. denote sampling subject latents from the same or different target task  $\gamma$  respectively. For a conversion to be valid subject latents must all come from samples with target subject  $s_k = \sigma$  and task latents from samples with  $t_k = \gamma$ . S.s., D.s., S.t., and D.t. are then used as additional conditions for sampling. We then consider all combinations (S.s., S.t.), (D.s., D.t.), (D.s., S.t.), (S.s., D.t.) for different conversion abstractions.  $N$  pairs of subject and task latents corresponding to the targets are drawn using the specific considered combination of conditions. The EEG is reconstructed for the  $k$ 'th sample to obtain  $\hat{x}_k^{(\sigma,\gamma)} \in \mathbb{R}^T$  where  $T$  is the number of time-samples. The converted ERP (C-ERP) is measured by averaging over each of these converted EEG signals:  $\hat{x}_{(\sigma,\gamma)}^{\mathrm{C-ERP}} = \frac{1}{N}\sum_{k=1}^{N}\hat{x}_k^{(\sigma,\gamma)}$ . The ERP conversion loss is calculated as the mean squared error (MSE) between the converted ERP and the per-subject per-task ERP:  $L_{\mathrm{C-ERP-MSE}}(\sigma,\gamma) = \frac{1}{T}\|\hat{x}_{(\sigma,\gamma)}^{\mathrm{ERP}} - \hat{x}_{(\sigma,\gamma)}^{\mathrm{C-ERP}}\|_2^2$ . An illustration of this procedure and examples of these ERPs are available in supplementary material.
An ERP was measured for each subject and task combination and was compared with the corresponding ERP from converted EEGs. The reported ERP conversion MSE is the average over all target subject and target task combinations. The number of samples  $(N)$  was set to 2000 for all methods. See supplementary material for an analysis of how this number affects the ERP conversion MSE.
A more detailed summary of the data, pre-processing, hyperparameters and training for all models, architectures and methods can be found in the supplementary material.
4 Results and Discussion
Table 1 shows the results of the task and subject classification as well as EEG conversion. The standalone CL, CE and  $\mathrm{CE(t)}$  models do not have a decoder, and therefore do not have a reconstruction loss. The standard error of the mean is reported for all classification accuracies and ERP conversion MSEs over the five  $(n = 5)$  repeats of each model.
Table 1: Single-trial balanced subject classification accuracy (S.acc%), task-on-subject classification accuracy (T-S.acc%), task classification accuracy (T.acc%), subject-on-task classification accuracy (S-T.acc%), and zero-shot same-subject same-task (S.s., S.t.), different-subject different-task (D.s, D.t.), different-subject same-task (D.s., S.t.), and same-subject different-task (S.s., D.t.) ERP conversion MSE. All ERP conversion MSE values have scales of  $10^{-11}\mathrm{V}^2$ . Confusion matrices are provided in the supplementary material.
Model	S.acc%	T-I-S.acc%	T.acc%	S-I-T.acc%	(S.s., S.t.)	(D.s, D.t.)	(D.s., S.t.)	(S.s., D.t.)
CSLP-AE	80.32 ± 0.28	45.41 ± 0.37	48.48 ± 0.34	79.64 ± 0.37	4.21 ± 0.12	20.06 ± 0.10	5.80 ± 0.15	6.65 ± 0.23
SLP-AE	74.63 ± 0.74	47.23 ± 0.31	47.00 ± 0.32	74.70 ± 0.73	3.82 ± 0.04	19.92 ± 0.10	6.12 ± 0.09	5.02 ± 0.08
C-AE	79.42 ± 0.48	37.34 ± 0.45	46.59 ± 0.23	73.27 ± 0.25	4.28 ± 0.06	20.28 ± 0.07	11.33 ± 0.47	10.64 ± 0.30
AE	60.68 ± 0.16	31.62 ± 0.27	31.43 ± 0.28	61.08 ± 0.38	3.54 ± 0.12	20.82 ± 0.07	11.20 ± 0.32	10.74 ± 0.48
CL	78.82 ± 0.46	37.65 ± 0.54	45.36 ± 0.37	71.70 ± 0.55	-	-	-	-
CE	79.25 ± 0.37	35.52 ± 0.38	45.22 ± 0.23	64.73 ± 0.44	-	-	-	-
CE(t)	-	-	45.80 ± 0.24	44.27 ± 0.59	-	-	-	-
CSP	-	-	35.22 ± 0.11	69.89 ± 0.10	-	-	-	-
From the results on ERP Core (Table 1), the best subject and task classifications were obtained using the proposed CSLP-AE whereas the second best performant models for subject classification and task classification were C-AE and SLP-AE respectively. We further observe a substantial performance increase in the subject and task classification of the SLP-AE when compared to the conventional AE whereas SLP-AE even provides higher task accuracies than conventional supervised training (i.e., CL, CE,  $\mathrm{CE(t)}$ , and CSP). Importantly, AE and C-AE exhibit poor conversion performance and can only reconstruct good ERPs in the standard autoencoder regime considering same subject and same task (S.s., S.t.). We observe that the SLP-AE and CSLP-AE both perform well in conversion when either the task or subject latents come from other samples (D.s., S.t.) and (S.s., D.t.) achieving reconstruction errors similar to (S.s., S.t.). Examples of converted ERPs from (D.s., S.t.) and (S.s., D.t.) are shown in Figure 3b.
Experiments on the EEGMMI dataset (Table 2) showed similar results with the CSLP-AE model outperforming both the C-AE and SLP-AE models, and achieving on par performance with the current state-of-the-art model: CSLP-AE task accuracy of  $64.28 \pm 0.16\%$  compared to  $65.07\%$  achieved by EEGNet[67]. However, the SLP-AE model performed considerably worse than the CSLP-AE and C-AE models which performed similarly on the SleepEDFx dataset. Split-latent permutation, therefore, does not seem to increase task classification accuracy on non-time-locked data.
Table 2: Task classification accuracy on PhysioNet [18] EEG Motor Movement/Imagery Dataset [54] (EEGMMI) and Sleep-EDF Expanded Dataset [29] (SleepEDFx)
Model	EEGMMI	SleepEDFx
CSLP-AE (ours)	64.28 ± 0.16	75.16 ± 0.95
C-AE (ours)	61.89 ± 0.41	75.16 ± 0.86
SLP-AE (ours)	57.93 ± 0.56	70.59 ± 1.18
EEGNet (Wang et al. [67])	65.07	-
f-CTrans (Xie et al. [73])	64.22	-
CNN (Dose et al. [12])	58.59	-
XSleepNet2 (Phan et al. [46])	-	84.0
Zhu et al. [79]	-	82.8
SeqSleepNet (Phan et al. [45])	-	82.6
SleepTransformer (Phan et al. [47])	-	81.4
AttnSleep (Eldele et al. [14])	-	81.3
SleepEEGNet (Mousavi et al. [39])	-	80.0
The CSLP-AE model achieved a sleep stage classification accuracy of  $75.16 \pm 0.96\%$  which is notably lower than the state-of-the-art model XSleepNet2 [46] with  $84.0\%$  accuracy. With an accuracy of  $75.16 \pm 0.96\%$ , the model demonstrates a capability beyond mere chance, effectively characterizing sleep stages outside the constraints of the time-locked paradigm. We presently restricted the model to Fourier-compressed representations of the data but we expect performance could be increased using encoders with larger receptive fields such as WaveNet [9, 40, 56] on the raw EEG waveform data.
$t$ -distributed stochastic neighbor embedding [1, 31, 65] ( $t$ -SNE) plots are provided in Figure 3a for the SLP-AE, C-AE and CSLP-AE models on the ERP Core dataset (further details of the  $t$ -SNE and additional plots are provided in the supplementary). From the figure we observe that the SLP-AE
Figure 3: (a)  $t$ -SNE plots of split-latents as encoded on the test set (unseen subjects) of the ERP Core dataset, colored by true labels. Rows show subject and task latent spaces, respectively, while columns indicate model type (SLP-AE, C-AE, and CSLP-AE respectively). For task latent space colorized by subject, and vice versa, see supplementary material. (b) Converted ERPs from the same three models for a random target subject and target paradigm. All latents used in conversion were from unseen subjects on the test set, i.e. unseen to unseen conversion. The FCz channel was chosen according to which channel Kappenman et al. [28] found to most prominently show the ERP of the given paradigm. For more conversion examples see supplementary material.
trained subject and task spaces are topologically similar. We further observe that the addition of the split-latent permutation loss to the contrastive learning model had little effect on the topology of the task latent space but some effect on the subject latent space. As the observed topology of the latent space did not undergo significant changes, the improvement in conversion performance can primarily be attributed to the capabilities of the decoder and the information flow within it to implicitly account for the permutation-invariance. This points to the importance of accounting for structural encoding.
The different ERP conversion abstractions described in Section 3 are increasingly more difficult to convert from. (S.s., S.t.) allows structural information through both latent spaces, while (D.s., S.t.) and (S.s., D.t.) allow structural information through one latent space since one of the latents will come from the same input sample. (D.s., D.t.) is a hard problem requiring conversion with no retention of any structural information from the specific sample for the decoder to rely on, i.e. conversion must be done using only abstract subject and task embeddings from which structure must arise.
Models trained with latent-permutation learns this structural encoding in the latent spaces since the decoder can rely on at least one of the latent spaces in the latent-swap procedure (Figure 2) to provide the structural information of the EEG signal. This structural information, although indistinguishable in different latent spaces, coincidentally is highly correlated with both subject and task content of the signal. This structural encoding is most notable in the SLP-AE latent space as shown in Figure 3a allowing the classifier to perform well on all classification tasks. However, the SLP-AE model obtains worse performance on subject classification compared to the other deep learning models. Interestingly, it had identical performance of subject classification on either subject or task latents, and similarly for task classification (S.acc% ≈ S-T.acc% and T.acc% ≈ T-S.acc% for SLP-AE in Table 1), which further accentuates their topological similarities seen in Figure 3a.
Contrastive learning in the latent space itself has nothing to do with decoding the structure of the data for reconstruction. The encoder is simply trained to learn an embedding of the subject and task content of the signal. The decoder must, therefore, do the heavy work of extracting this structural information itself. This exposes a property which is applicable to the latent-permutation method but not contrastive learning. When the stand-alone latent-permutation method is used (SLP-AE), the decoder is allowed a reliability in the latent spaces.
A lot of the EEG signal is structural information, therefore, there might be more to gain from minimizing the permutation-invariance to structural information rather than encoding the subject and task content directly. Thereby, the decoder can learn a permutation-invariant structural encoding of the signal in both latent spaces, which allows the decoder to rely on this information irrespective of the permutation or swap - since only one latent space (see Figure 2) is swapped at a time. Thus, the latent-permutation only trained model (SLP-AE) does not learn explicitly disentangled representations of the subject and task content of the signal, but rather a duplicated latent space permutation-invariant
structural encoding which is highly correlated with both subject and task content of the signal. This interestingly is also the goal of the standard autoencoder. The latent-permutation method allows the model to learn an encoding explicitly in the latent space instead of implicitly in the encoder/decoder networks. This might be a powerful tool since it further constricts the bottleneck in the auto-encoder sense, although at the cost of suboptimally encoding identical latent spaces. This can be, and is, remedied by using both contrastive learning and latent-permutation in conjunction.
Having completely disentangled latent spaces is a local minimum in the latent-permutation method and allows for the ideal swap in Figure 2a. The CSLP-AE model is able to keep the latent spaces disentangled while also providing the structural encoding information required for the conversion method to work. This is evident from Table 1 which shows that the stand-alone latent-permutation model (SLP-AE) achieves about half  $(\approx 51.7\%)$  the MSE error of the C-AE model on the ERP conversion tasks using samples from different tasks or subjects (D.s, S.t. and S.s., D.t.), and the CSLP-AE model retains this performance. We propose the latent-permutation method as a replacement for the standard auto-encoder reconstruction loss to be used in conjunction with contrastive learning and the batch construction method to provide disentangled latent spaces which also allows for the structural encoding information to flow through and ease zero-shot conversion.
The latent-permutation method does not increase performance on the (D.s., D.t.) conversion task. Arguably the difference in performance between the stand-alone latent-permutation and contrastive learning methods (on D.s, S.t. and S.s., D.t.) is due to this structural encoding property. An analysis is provided in the appendix providing a generalization of the latent-permutation method onto different subject/different task conversion to circumvent the structural encoding reliability. Further research could explore different avenues for keeping structural information while also disentangling subject and task latents, or providing a source for this structural encoding using generative methods or distributional power to generate structure, e.g. through the use of a VAEs [30].
In this paper, a novel split-latent permutation framework was introduced for disentangling subject and task content in EEG signals and enabling single-trial zero-shot conversion. By combining the proposed split-latent permutation framework with contrastive learning, we achieved better performance compared to standard deep learning methods on the standardized ERP Core dataset. The experimental results demonstrated a significant  $51.7\%$  improvement in ERP conversion loss on unseen to unseen subject conversions. The method also achieved high single-trial subject classification accuracy  $(80.32 \pm 0.28\%)$  and single-trial task classification accuracy  $(48.48 \pm 0.34\%)$  on unseen subjects.
Limitations The conversion results in this paper are limited to the single dataset on which they were trained, and may not generalize to other datasets with different experimental conditions. The ERP Core dataset is meant to standardize ERP measurements with more data to come in the future from other laboratories which might alleviate this limitation. Furthermore, the tasks used in the experiments are all seen during training. This limits the scope of the results to the seen tasks and no conclusions can be drawn about the generalization of the model to unseen tasks.
Broader Impact Zero-shot conversion of EEG signals, similar to other deep fake methods, can be used for malicious purposes. Methods discussed in this paper intrude on one of the most sacred places yet to be exploited and confused by technology: the human mind. Malicious use of this technology includes the ability to decode thoughts and intentions from EEG signals, and the ability to create fake EEG signals to confuse verification systems using EEG signals as a biometric identifier [11, 37, 51]. Care must be taken when developing and deploying such technology to ensure that it is not used for malicious purposes. However, it can also be used to improve the quality of life for people with and without disabilities. It provides a base for generalizing EEG signal representations across subjects and tasks, which can be used to improve the performance of EEG-based BCI systems, especially on a single-trial level as attestable by the results in Table 1. Similarly, it could provide a base for novel analysis strategies, such as predicting drug or stimulus reactions in a healthy versus diseased brain, or as biomarkers of brain disorders.
Funding in direct support of this work: Lundbeck Foundation grant R347-2020-2439.