CSLP-AE: A Contrastive Split-Latent Permutation Autoencoder Framework for Zero-Shot Electroencephalography Signal Conversion
Authors: Anders Vestergaard Nørskov
Alexander Neergaard Zahid
Morten Mørup
Venue: NeurIPS 2023 poster (2023)
Keywords: zero-shot conversion
representations learning
contrastive learning
electroencephalography
autoencoder
subject variability
permutation invariant training
TL;DR: The study presents a new method, CSLP-AE, to effectively separate and analyze brain activity (content) and individual differences (style) in EEG data, enhancing task and subject conversion, even with previously unseen subjects.

Abstract: Electroencephalography (EEG) is a prominent non-invasive neuroimaging technique providing insights into brain function. Unfortunately, EEG data exhibit a high degree of noise and variability across subjects hampering generalizable signal extraction. Therefore, a key aim in EEG analysis is to extract the underlying neural activation (content) as well as to account for the individual subject variability (style). We hypothesize that the ability to convert EEG signals between tasks and subjects requires the extraction of latent representations accounting for content and style. Inspired by recent advancements in voice conversion technologies, we propose a novel contrastive split-latent permutation autoencoder (CSLP-AE) framework that directly optimizes for EEG conversion. Importantly, the latent representations are guided using contrastive learning to promote the latent splits to explicitly represent subject (style) and task (content). We contrast CSLP-AE to conventional supervised, unsupervised (AE), and self-supervised (contrastive learning) training and find that the proposed approach provides favorable generalizable characterizations of subject and task. Importantly, the procedure also enables zero-shot conversion between unseen subjects. While the present work only considers conversion of EEG, the proposed CSLP-AE provides a general framework for signal conversion and extraction of content (task activation) and style (subject variability) components of general interest for the modeling and analysis of biological signals.

CSLP-AE: A Contrastive Split-Latent Permutation Autoencoder Framework for Zero-Shot Eleetroencephalogrraphy Signal Conversion
Anders Vestergaard Nørskov
Alexander Neergaard Zahid
Morten Mørup
Department of Applied Mathematics and Computer Science
Technical University of Denmark
andersxa@gmail.cm com {aneol, mmor) @dtu.
Abstract
Electroencephalograapy (EEG) is prominent non-invasive neuroimaging tech- nique providing insights into brain function. Unfortunately. EEG data exhibit a high degree of noise and variability across subjects hampering generalizable signal extraction. Therefore, a key aim EEG analysis is to extract the underlying neural activation (content) well to account for the individual subject variability (style). We hypothesize that the ability to convert EEG signals between tasks and subjects requires the extraction of latent representations accounting for content and style. Inspired by recent advancements in voice conversion technologies, we propose a novel contrastive split-latent permutation autoencoder (CSLP-AE) framework that directly optimizes for EEG conversion. Importantly, the latent representations are guided using contrastive learning to promote the latent splits explicitly represent subject (style) and task (content). We contrast CSLP-AE to conventional super- vised, unsupervised (AE), and self-supervised (contrastive learning) training and find that the proposed approach provides favorable generalizable characterizations of subject and task. Importantly, the procedure also enables zero-shot conversion between unseen subjects. While the present work only considers conversion of EEG, the proposed CSLP-AE provides general framework for signal conversion and extraction of content (task activation) and style (subject variability) components general interest for the modeling and analysis sis of biological signals.
Introduction
Electroenncephalograayy (EEG) is non-invasive method for recording brain activity, commonly used neuroscience research analyze event-related potentials (ERPs) and gain insights into cognitive processes and brain function (28). However, EEG signals are often noisy, contain artifacts, and exhibit
high sensitivity to subject variability 13 43 making challenging to analyze and interpret the data, In particular, the inherent subject variability is well known to confound recovery of the task content of the signal, and a study by Gibson et al. [7 demonstrated that across-subject variation in EEG variability and signal strength was more significant than across-task variation. A major challenge modeling EEG data is thus to remove the intrinsic subject variability in order to recover generalizable patterns of the underlying neural patterns of activation.
Supervised methods explicitly classifying tasks can potentially filter subject variability and recover generalizable patterns of neural activity. However, explicitly disentangling subject and task content EEG signals valuable not only for classifying task content but also for characterizing the subject variability which ultimately can provide biomarkers of individual variability. As such, rather than
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

focusing only on the experimental effects and considering inter-subject variability as noise it should
be treated as an important signal enabling the understanding of individual differences 24 55].
Deep learning-basee feature learning has become prevalent in EEG data analysis. As such, auto-
encoders have shown promise in learning transferable and feature-rich representations 36 60 70 and
have been applied to various downstream tasks, including brain-computer interfacing (BCI) 35 44
78), clinical epilepsy and dementia detection [99
38 77,, sleep stage classification [34 63 emotion
recognition 126 611. affective state detection 53 74 and monitoring mental workload/fatigue
75 76
with promising results. For systematic review on deep learning- based EEG analysis, see
e.g. Roy et al.
[52].
The task of disentangling content (signal) from style (individual variability) is well-known aim in voice conversion technologies. A study on speech representation learning achieved promising results in disentangling speaker and speech content using speaker-conditioned auto-encoders [9, while Chou et al. [0] proposed using instance normalization to enforce speaker and content separation in the latent space. In Qian et al. (48) zero-shot voice conversion was proposed using a simple autoencoder conditioned on a pretrained speaker embedding model and exploring bottleneck constriction to obtain content and style disentanglement with good results. Wu and Lee (71) and Wu et al. 772 disregarded the pretrained speaker embedding model by generating content latents based on a combination of instance normalization and vector quantization of an encoded signal, while the speaker latents were generated based on the difference between content codes and the encoded input signal.
Disentangling subject and task content has been shown to enhance model generalization in emotional recognition and speech processing tasks 4 50 57.. Bollens et al. 4 used two explicit latent spaces in a factorized hierarchical variational autoencoder (FHVAE) model high-level and low-level features of EEG data and found that the model was able to disentangle subject and task content. They also found that high-level features were more subject-specific and low-Ievel features were more task-specific. Rayatdoost et al. (50) and Özdenizci et al. 42 explored adversarial training to promote latent representations that were subject invariant. Recently, self-supervised learning has gained substantial attention due to its strong performance in representation learning enabling deep learning models to efficiently extract compact representations useful for downstream tasks. This includes the use of (pre- )training on auxiliary tasks [62] as well as contrastive learning methodologies guiding the latent representations 225) Shen et 57 used contrastive learning between EEG signal representations of the same task and different subjects to learn subject-invariant representations and found that the learned representations were more robust to subject variability and improved generalizability. For a survey of self-supervised learning in the context of medical imaging, see also (58)
Inspired by recent advances in voice conversion, we propose a novel contrastive split-latent permuta- tion autoencoder (CSLP-AE) framework that directly optimizes for EEG conversion. In particular, we hypothesize that the auxiliary task of optimizing EEG signal conversion between tasks and subjects requires the learning of latent representations explicitly accounting for content and style. We further use contrastive learning to guide the latent splits to respectively represent subject (style) and task (content). The evaluation of the proposed method is conducted on a recent, standardized ERP dataset, ERP Core (28), which includes data from the same subjects across a wide range of standardized paradigms making it especially suitable for signal conversion across multiple tasks and subjects. We contrast CSLP-AE to conventional supervised, unsupervised (AE (23 ), and self-supervised training (contrastive learning
49 59 69).
2
Methodology
The aim of this paper is to develop modeling framework for performing generalizable (i.e., zero- shot) conversion of EEG data considering unseen subjects. The procedure should enable conversion of EEG from one subject to another as well as one task to another task. In this context, "tasks" refer to the specific ERP components present in the EEG data, such as face or car perception, word judgement of related or unrelated word pairs, perception of standard or deviant auditory stimuli, etc. [2].
The standard autoencoder (AE) model consists of an encoder, denoted as Ex( : X + Z, and a decoder, denoted as D& (Z): 2 + X. The encoder maps the input data to a latent space, while the decoder reconstructs the input from the latent space. The encoder and decoder are parameterized by 0 and ó respectively.
2

---

To enable the model to perform conversion, it needs to be conditioned on the target subject and/or task. The latent space uppears be the suitable place for conditioning. as it is compact representation often referred to as the bottleneck of the model. However, since the latent space is shared across subject and task representations, partitioning into specific subject and task streams is non-trivial.
To address this challenge, split-latent space is explored, which explicitly divides the latent space into subject and task disentangled representations. This is achieved by introducing a split in the model design within the encoder. The split-latents can be obtained by encoding the input data as follows: Ex(X) = ===(0) (=())=²()), where z represents the subject latent and (T)) represents the task latent, such that (x) X' (S, T). Note that the (S) and (T) denominations
not inherent from the model architecture but are necessary distinctions for use in the model loss functions. These split-latents can then be joined and decoded using similar split within the decoder. By feeding the split-latents into the decoder, the input can be reconstructed as X Dz(2(0), 2 such that (z((() xTT) : (S, T) + X. The concept of split-latent space has been explored by researchers within speech 48 and in the EEG domain 4 using separate encoders for each latent space. We presently employ a shared encoder to reduce the number of parameters used. However, separate encoders or even pre-trained encoders, specifically, on subjects could help kick-start the training process or work as conditioning with frozen weights. We leave this to further studies.
While voice conversion [8 68 and other voice synthesis problems 440 566 based on autoencoders generally use models with expansive receptive fields, e.g. as in WaveNet [40), other studies have found similar performance in voice conversion using simpler architectures, such as CNNs [10 27 33) Voice conversion usually done over a large number of samples with exceptionally high sampling rate compared to EEG data. ERP Core uses sampling frequency of 256 Hz (downsampled from 1024 Hz) over an epoch window of second, which only yields a time resolution of 256 samples. Therefore, large receptive field is unnecessary, and strided convolution to reduce time-resolution is used instead. The proposed EEG auto-encoder model with split latent space is illustrated in Figure]
ConvBlock
Enceder
EEG Auto-enczdder with Spril Lalent Space
Decoder
Convid, k= C: 30 266
EEG
Reporelructed EEG
Convic, km1
C: 258 - 30
Convid, kee, p=1 + ReLU InstanosNom
ConvBlock
ConvBlack
ConvBlock
Encoder
Decoder
Transposeooov
StridedComv
ConvBlack
Convid ket + ReLU InstamoeNomm
Splt Bottlonack
Transfomner ayans:
4
Split Bottlenack
ConvBlock
ConvBlock
Convid, ket
ConnBlock
Cenceb
ConvBlock
ConvBlock
ConwBlock
ConvBlack
Figure 1: The proposed EEG auto-encoder model with split latent space. The encoder and decoder are mirrored deep convolutional neural networks using one- dimensionn convolutions. Each part the auto-encoder consists of ConvBlocks which are made up of three convolutions with residual connections (as in Hc ct al. (2),, rectified lincar unit (RcL.)) activation [15] and instance normalization 64 (similar to Chou ct al. [0)). The encoder applies a ConvBlock together with strided convolution (stridc=2), to reduce the time-resolution by half, four times. The decoder mirrored and uses transposed convolutions (Iractionally strided, stride=1/2) upscale the time-resolution by a factor of two with each block. Both the encoder and decoder models use a transformer [66] with four layers on each side of the bottleneck to confer atlention. Finally, on the encoder side, a split is made into subject and task latent spaces. The decoder takes these split-latenn as inputs and joins them again in the bottleneck in order to reconstruct the input. lo is the kernel size and is the padding both sides.
3

---

2.1 Split-latent permutation
To guide the autoencoder in Figure in disentangling task and subject we propose the use of latent- permutation, which is self-super approach that guides the latent space by ensuring consistency in the subject and/or task encodings between permutations. This can be seen as a direct loss relating to the conversion method described in this section.
To achieve zero-shot conversion the respective subject and task information need to be extracted from the input data, and with an explicit split of the latent space, the conversion becomes straightforward and practical. Depending on the desired conversion task, such as converting from subject U subject V or from task M to task N, the corresponding split-latents are simply swapped with that of the target subject or task. This conversion method is illustrated Figure 2a.
Given a pair of input samples (X X where i is the batch index, the two pairs of split- split-latents are
defined from Eg splitting the latent space in two parts yielding (z/(8,), z 2(TTa)) and (z(5.,) 2 =(TTb)). A latent penmutation is performed which swaps two of the latents in a given latent space L E {S, T before reconstruction. comprehensive glossary symbols and abbreviations provided in the appendix.
Consider the latent space L = T. The pair of input samples (X f x are sampled to both belong the same task 4, The task latents are swapped between the pairs such that the reconstructed EEG data decoded by D& becomes
(x() = (z/5,0)
x(T =D0==D>(2 (=( (S,)) 222 (T.b))
(1)
x(T (=( (TTa)))
here colorized according to their corresponding input sample X or X This is the same-task autent-permutatioon since pairs belong to the same task class. A corresponding swap can be done for the subject latent space S with pairs belonging to the same subject, s;. The task and subject latent space permutations are illustrated in Figure 2p and Figure 2 respectively.
The latent-pen permutation loss is defined as the sum over the pair of reconstruction losses between the two samples and their corresponding reconstructions with split-latents from latent space L swapped:
LLP(C; X
Σ (IX N
1
X! ((Ca)
112 +
I
X x
Consider the scenario where L = T again using the reconstructions from Eq. In Eq. (3) the L2-norm is calculated between the input data X and the reconstruction of the latent- latent-permutatiinn permutation (z,(5.), 2 <(TTi)). According the smoothness meta-prior proposed by Bengio et al. (2 a pair of task latents should be invariant to local perturbations in the input. Seen in the context of latent- permutation, such local perturbation could be equivalent two task representations of the same class t, from different input samples. When the latent space is locally smooth (i.e. encoder is consistent) then this term approximates the autoencoder reconstruction loss, i.e. if x; (z(2 (S,a) x (T.a) =s (2((5a)) z ((T,b)) where (2 (S.0), 2, (T()) is the standard reconstruction when both Eg and D are consistent. This is expanded upon in the supplementary material.
(T,a) z;
(T,b) then
)
Notably, such autoencoding enables flow structural information specific for the given input sample and its reconstruction beyond subject and task content. In the supplementary material, we explore setup where the output sample is from a different subject and task than the input sample.
In contrast to AC [48). which achieves disentanglement by adding an explicit speaker latent and redueing the capacity of the content encoder through smaller latent dimension, the proposed split-latent permutation directly optimizes for conversion. It is not guaranteed that it will explicitly separate task and subject information in their respective spaces. When the capacities of the latents are sufficiently large, both subject and task content can potentially be encoded together, and the decoder can learn to extract the corresponding subject and task content from these identically encoded splits. To address this issue and avoid relying solely on disentanglement achieved through careful bottleneck tuning, we explore contrastive learning to suffice the smoothness criteria, disentangle and specialize each latent space. Limiting the capacity of the latent space is explored in the supplementary material.

---

Ideal convension
Späit Latent Permutation
Split Latent Permutation subject-poinn
task-painss
MSE
MSB
X
X
TT
M
M
(5,60
_
X
Tray
(7,
(a)
(b)
(c)
Figure 2: (a): Split-latent conversion example. Two samples X and X", where X corresponds to subject U and task M and X" corresponds to subject V and task N respectively. for each sample. this example, to convert from task to task the subject (U) latent is kept while the task (M) latent simply swapped with the corresponding latent from the other sample which encodes task N. The split-latents are then decnded to idcally obtain sample from the data distribution given subject U and task N. This is the ideal case where the split-latents
encoded yielding pair of split latents
(b) and (c): Illustration of the object class-dependant swap. The input samples encoded yielding their corresponding split-latents (z(8,)), 2(5,8), (TT,a), ((T,))). Depending on the object class (lasks in (b), subjects in (c)), the split-latents are swapped between the pairs of latents. These are then decoded yielding the reconstructed ERG data. The same-task pairs both have blue (M) task-latents since these are intentionally sampled from the same class. The intention the perettpermutation method is expressed when the model is consistent in its latent representation of the class, and when the split latent spaces are sufficiently disentangled in the ideal case. In such cases the swap will have negligible impact on the conversion, i.e. if they encode the same (subject or task) information.
perfectly disentangled and independent.
2.2 Contrastive Learning
Contrastive learning is a self-supervised learning method which aims learn representation the data by maximizing the similarity between positive pairs and minimizing it between negative pairs 28 achieving local smoothness around classes. We apply contrastive learning to each split-latent space. This is done by utilizing batch construction technique to sample pairs for both subjects and tasks separately, and then applying the contrastive loss to each split-latent space according to denomination, thus specializing the given latent space for either subject task embeddings.
Specifically, consider the multi-class N-pair loss [59] which a deep metric learning method [69] utilizing special batch construction technique. The batch construction techmique involves sampling two samples from each class, and then constructing pairs of samples from the batch. This batch construction is similarly required for the latent- permutation in Section 2.1 We use the batch construction method combined with the InfoNCE generalization from Oord al. [4]] with the temperature parameter as in Chen et al. 7 The full generalization is equivalent to the CLIP-loss from Radford et al. [49] in which we minimize the symmetric cross-entropy loss of the temperature-scaled similarity matrix based on the NT-Xent loss
(7). Let Z RCcK and Z gCxK be latent representation matrices the K pairs of samples, then LNT-Xcot and LCUP are defined as
exp(sim( =(//T)
LNT.Xom (e:xom((;2 2L; Z" kk) log K
(4)
D 1 (ist exp(sim (z/i 2/T)
LeLw(C; C: A B
[NNTXem
(L; (x; B kk) INTXXe((2; 8 22 (1))
(5)
K
where [e] E {0. 1) } is the indicator function yielding 1 iff the condition holds true. sim(2/, = is similarity metric. denotes from which latent space the pairs have corresponding labels, e.g. for the task latent space T, the k thth pair has the same task l which is different from the other tasks in the same batch, i.e. I # la .... lx. T.uLPP (T; therefore is contrastive loss across tasks, while LS is across subjects.
5

---

3 Experimental Setup
Data We consider the ERP Core datase from Kappenman et al. [28] providing a standardized ERP dataset containing data from 40 subjects across six different tasks based on seven widely used ERP components: N170 (Face Perception Paradigm), MMN (Mismatch Negativity, Passive Auditory Oddball Paradigm), N2pe (Simple Visual Search Paradigm), N400 (Word Pair Judgement Paradigm), P3 (Active Visual Oddball Paradigm), and LRP and ERN (Lateralized Readiness Potential and Error-related Negativity. Flankers Paradigm). We only consider data processed by Script #1 up until ICA preparation For more information on the data and paradigms, see Kappenman et al. [28]
Only the epoch windows around the time-locking event described in Kappenman et al. 28 used as epochs, therefore, for each paradigm there are two available "tasks" each with different resulting ERP. The two classes assigned for each event per paradigm are: N170; faces/cars, MMN; deviants/standards, N2pc; contralateraliisilattrs N400; unrelated/related, P3; rare/frequent, and ERN and LRP: incorrect/correct. The dataset was split across subjects into a training set of 70%, an evaluation set of 10%, and a test set of 20% of the subjects respectively. The exact splits are available in the supplementary material.
The ERP Core dataset is predominantly time-locked with data centered around either stimulus or response") We further evaluate the models "as is"
other modalities of EEG data from PhysioNet 18 and with already established state of the art model results: the EEG Motor Move- ment/Imagery Dataset (EEGMMI) [54] and the Sleep-EDF Expanded (SleepEDFx) database (29). The EEGMMI dataset is cue time-locked and consists of recordings from 109 subjects performing various motor imagery (MI) tasks. We applied our model to the standard L/R/0/F MI task using 38 epoch windows, closely following the approach of Wang et al. [67]. The SleepEDFx is included to explicitly probe the model integrity on data that is not time-locked to an external stimulus and contains 153 polysomnography studies from 78 subjects. Given its limited EEG channels, we adapted our methodology by considering a single EEG channel and applied short-time Fourier transform to fit the data to the same setup as used for ERP Core. We maintained the conventional 30s time series windows commonly used in sleep stage literature. Details on the data preprocessing and model integration for these datasets can be found in the supplementary materials.
Model comparisons The proposed CLSP-AE approach is systematically compared against mix of learning strategies comprising conventional AE-based representation learning, contrastive learning and representations obtained by supervised learning. To optimally compare these learn- ing strategies all models are based on the same model structure given in Figure [] Models will be denoted by which losses they are trained on, which external methods they use. Here CSLP-AE will denote the contrastive split-latent permutation over both subjects and latents and (T; ; )), SLP-AE the corresponding model with- (Trp(S; T.p((T:..). (. (cuup(S;,,) out the contrastive loss components (S,p(S;;,) -, ). and (T::;..;)))) +)), CL the contrastive loss in both of the split latent spaces similarity metric in the contrastive loss. AE will denote the standard auto-encoder with mean-squaree error reconstruction loss on the reconstructions of non-permuted split-latents. C-AE will denote the combination of reconstruction loss and contrastive learning, such that contrastive learning is applied in both spaces and the standard autoencoder reconstruction loss is applied to non- non-permuted reconstructions. CE will denote the supervised learning cross- cross-entropy loss jointly trained in the subject and task latent spaces using the corresponding true labels in a supervised manner. Similarly, CE(L) will denote cross-entropy only trained on the task labels in the task latent space. This is to substitute and compare with a supervised deep learning model, contrary to the self-super methods described here. For completeness, we also included the common spatial pattern (CSP) using the multi-class generalization from Grosse Grosse-Wentrup and Buss (20] which is a supervised method for extracting discriminative features from EEG data. These features are then used to map unseen data into the same "CSP space". For the EEGMMI and SleepEDFx datasets
and LCUP :)). Cosine similarity will be used as the
method 3 32
respectively compared the task and sleep stage classification performance to 11 67 73 and
14 39 45 [9].
'ERP Core info: httpp: https://eerpint...rr/pppppoororo Data available at: https:://st.. http:///ooff.oot/gggg/o// "See the full ERP Core procedure here: http:://ggthaa...omm///aaaa/aaap htwp:/20haalssss2200oroooo
pdd
'Only the ERN and LRP paradigms response tome-locked

---

The total loss for models with multiple losses is the sum of losses with equal weighting. This is further detailed in the supplementary materials and loss curves are shown in the appendix.
Hyperparameters were chosen based on evaluation during development, and the most critical hyperpa- rameters (the number of blocks and the size of the latent space) were verified on the evaluation set in a grid search. See supplementary material for details on the grid search across both latent dimension and number of blocks. The test set was used only for final evaluation.
Subject and task characterization Non-linear classifiers were trained to classify the subject and task labels of split-latents from the test set to quantify the disentanglement and generalization of the latent spaces. This was performed as two five (5)-fold cross-valle validations (CVs) over the subject and task latents respectively stratified on the subject and task labels on the test data (unseen subjects). Since there is high class-imbalance in the task labels, undersampling was performed for each class to match the number of samples the least represented class. The undersampling was performed on the training split of each fold, and the test split was left untouched. Balanced accuracy [ was used due to the high class imbalacee.. The subject CV was used to evaluate the subject classification accuracy (S.acc%) and ausk-on-subjec classification accuracy (Ti-S.acc while the task CV was used to evaluate the task classification accuracy (Tacce% and subject-on- (SI 3).c%).. The task-on-subjee classification accuracy was evaluated by training a classifier on the subject latents to predict the task of the corresponding input sample, and vice versa for the subject-on-tase classification accuracy.
classification accuracy
For each fold, an end-to-end tree boosted system (XGBoost) was trained on the training split, and subsequently evaluated on the test split. Finally, the results were averaged over the five (5) folds. All classifications were performed on a single-trial level. A K-nearest neighbors (KNN) classifier and an Extra Trees classifier [16] were also trained and evaluated. See supplementary material for these.
ERPs from zero-shot EEG conversions An ERP conversion loss was measured on the test set. First, ground-truth ERP was found for each subject and for each ERP component (task) by averaging over all samples belonging to the same subject and task only in the EEG channel respective to which Kappenman al. [28] found the given ERP component most prominent. Let a:ERP, denote such
(s.r) an ERP for subject and task If disentanglement was successful then the conversion method described in Section 2. and illustrated in Figure shohulld be able to reconstruct the ERP from a given subject and task latent pair. Thereby it should be possible to sample an amount of subject and task latent pairs encoded on the test set, and convert the ERP from these pairs. Let S.s. and D.s. denote sampling task latents from the same or different target subject a respectively, and let S.t. and D.t. denote sampling subject latents from the same or different target task y respectively. For conversion to be valid subject latents must all come from samples with target subject $% a and task latents from samples with & = 7. S.s., D.s., S.t., and D.t. are then used as additional conditions for sampling. We then consider all combinations (S.s., S.t.), (D.s., D.t.), (D.s., S.t.), (S.s., D.t.) for different conversion abstractions. N pairs of subject and task latents corresponding to the targets are drawn using the specific considered combination of conditions. The EEG reconstructed for the k'th sample to obtain ((0n)) E RTT where T is the number of time-samples, The converted ERP (C-ERP) &CEERP ((,,)) + E $ ((n) The ERP conversion loss is calculated as the mean squared error (MSE) between the converted ERP and the per-subject per-task ERP: LC.ERP (a, 7) + [ ::C-ERP 112. An illustration of this procedure and examples of these ERPs are available supplementary material.
is measured by averaging over each of these converted EEG signals:
An ERP was measured for each subject and task combination and was compared with the correspond- ing ERP from converted EEGs. The reported ERP conversion MSE is the average over all target subject and target task combinations. The number of samples (N) was set to 2000 for all methods. See supplementary material for an analysis of how this number affects the ERP conversion MSE.
more detailed summary of the data, pre-proces preprrcesssing hyperparameters and training for all models, architectures and methods can be found in the supplementary material.
7

---

4 Results and Discussion
Table I shows the results of the task and subject classification as well as EEG conversion. The stand- alone CL., CE and CE(t) models do not have a decoder, and therefore do not have a reconstruction loss. The standard error of the mean is reported for all classification accuracies and ERP conversion MSEs over the five (n 5) repeats of each model.
Table 1: Single-trial balanced subject classification accuracy (S.acc%) sask-on-subject classification accuracy (T- S.acc%), task classification accuracy (T.acc%). subject-on-task classification accuracy (S) T.)cc%), and zero- shot same-subject tame-task (S.s., S.t.), different-subjeet different-task (D.s, D.t.), diffcrent-subject same-task (D.s., S.t.), and same-subject lifferent-task (S.s., D.t.) ERP conversion MSE. All ERP conversion MSE values have scales of 10 V2. Confusion matrices are provided in the supplementary material.
Model
Sacc%
Th- S
Tacefi
Si-Tacc%
(8s., S.L.)
(D.s, D..)
(Dss, S.t)
(S.s,
CSLPAE
0.28
45.41
0.37
48.48
79.64 ::
4.21
0.10
580 0.15
6.65 0.23
SLP-AE
74.63 + 0.74
47.23 = 0.31
47.00 4 0.32
74.70 =: 0.73
3.82 + 0.04
19.92 1: 0.10
6.12 1: 0.09
5.02 0.08
C-AE
79.42 0.48
37.34
0.45
46.59 + 0.23
73,27 = 0.25
4.28 + 0.06
20.28 1 0.07
11.33 0.47
10.64 0.30
AE
60.68 + 0.16
31.62 = 0.27
31.43
+ 0.28
61.08 = 0.38
3.54 = 0.12
20,82 = 0.07
11.20 + 0.32
10.74 0.48
CL
78.82
+ 0.46
37.65
0.54
45.36 +: 0.37
71.70 = 0.55
CE
79.25 1 0.37
35.52 1 0.38
45.22 1. 0.23
64.73
1 0.44
CRO)
45.80 4: 0.24
44.27 0.59
CSP
35.22 + 0.11
69.89 = 0.10
From the results on ERP Core (Table the best subject and task classifications were obtained using the proposed CSLP-AE whereas the second best performant models for subject classification and task classification were C-AE and SLP-AE respectively. We further observe substantial performance increase the subject and task classification of the SLP-AE when compared to the conventional AE whereas SLP-AE even provides higher task accuracies than conventional supervised training (i.c., CL, CE, CE((), and CSP). Importantly, AE and C-AE exhibit poor conversion performance and can only reconstruct good ERPs in the standard autoencoder regime considering same subject and same task (S.s., S.t.) We observe that the SLP-AE and CSLP-AE both perform well in conversion when either the task or subject latents come from other samples (D.s., S.t.) and (S.s., D.t.) achieving reconstruction errors similar to (S.s., S.t.). Examples of converted ERPs from (D.s., S.t.) and (S.s., D.t.) are shown in Figure 3b
Experiments on the EEGMMI dataset (Table showed similar results with the CSLP-AE model outperforming both the C-AE and SLP-AE models, and achiev- ing on par performance with the current state-of-the-art model: CSLP-AE task accuracy of 64.28+0. 16% compared to 65.07% achieved by EEGNet (67). How- ever, the SLP-AE model performed con- siderably worse than the CSLP-AE and C-AE models which performed similarly on the SleepEDFx dataset. Split-latent permutation, therefore, does not seem increase task classification accuracy on non-time-locked data.
Table 2: Task classification accuracy PhysioNet B8
EEG Motor Movemen//Imagery Dataset 6. (EEGMMI) and Sleep-EDF Expanded Dataset (29 (SleepEDFx)
Model
BEGMMI
SleepEDFX
CSLP-AE (ours)
64.28 1 0.16
75.16 1 0.95
(ours)
61.89 + 0.41
75.
SLP-AB (ours)
57.93 = 0.56
70.59 # 1.18
BEGNet (Wang
65.07
I-CTTrams (Xie et al.
26
64,22
CNN (Dose
$8.59
XSloopNet2 (Phan ct 46
84.0
Zhu al. 70
82.8
SeySleepNet ( (Phan el al.
45
82.6
SteepTransformer (Phan
ArtnSleep (Eldele et al. [E]
81.3
SleepEEGNot (Mousavi ct
80.0
The CSLP-AE model achieved a sleep stage classification accuracy of 75..160.969 which is notably lower than the sate-of-the-an model XSleepNet2 [46] with 84.0% accuracy. With an accuracy of 75. 75.16+0.96% the model demonstrates a capability beyond mere chance, effectively characterizing sleep stages outside the constraints of the time-locked paradigm. We presently restricted the model to Fourier-compressee representations of the data but we expect performance could be increased using encoders with larger receptive fields such as WaveNet (9
40 [6] on the raw EEG waveform data.
!-distributed stochastic neighbor embedding 1 the SI.P-AE, C-AE and CSL.P-EE models on the ERP Core dataset (further details of the I-SNE and additional plots are provided in the supplementary). From the figure we observe that the SLP-AE
31 65
(l-SNE) plots are provided in Figure 3a for
8

---

Model: SLP-AE S. space colored
Model:
Model: CSLPAE
Target Sudject 26. larget Nerstign EAN Charndl
Surjecs
Suaged
Sages 24
space
Touk unt
Color
2... S-)
(DC. $11
e MEDOowiie
Figure 3: (a) i-SNE plots of split-latents as encoded on the test set (unseen subjects) of the ERP Core dataset, colored by true labels. Rows show subject and task latent spaces, respectively, while columns indicate model type (SLP-AE, C-AE, and CSLP-AE respectively). ). For task latent space colorized by subject, and vice versa, see supplementary material. (b) Converted ERPs from the same three models for a random target subject and target paradigm. All latents used in conversion were from unseen subjects on the test set, i.e. unseen to unseen conversion. The FCz channel was chosen according to which channel Kappenman et al. (2 found to most prominently show the ERP of the given paradigm. For more conversion examples see supplementary material.
trained subject and task spaces are topologically similar. We further observe that the addition of the split-latent permutation loss to the contrastive learning model had little effect on the topology of the task latent space but some effect the subject latent space. As the observed topology of the latent space did not undergo significant changes, the improvement in conversion performance can primarily be attributed to the capabilities of the decoder and the information flow within it implicitly account for the pernuration-invariance This points to the importance of accounting for structural encoding.
The different ERP conversion abstractions described in Section 3 are increasingly more difficult to convert from. (S.s., S.t.) allows structural information through both latent spaces, while (D.s., S.t.) and (S.s., D.t.) allow structural information through one latent space since one of the latents will come from the same input sample. (D.s., D.t.) is a hard problem requiring conversion with no retention of any structural information from the specific sample for the decoder to rely on, i.e. conversion must be done using only abstract subject and task embeddings from which structure must arise.
Models trained with peenntt-emuttiion learns this structural encoding in the latent spaces since the decoder can rely on at least one of the latent spaces in the latent-swap procedure (Figure 2 to provide the structural information of the EEG signal. This structural information, although indistinguishable in different latent spaces, coincidentally is highly correlated with both subject and task content of the signal. This structural encoding is most notable in the SLP-AE latent space as shown in Figure Ba allowing the classifier to perform well on all classification tasks. However, the SLP-AE model obtains worse performance subject classification compared to the other deep learning models. Interestingly, it had identical performance of subject classification on either subject or task latents, and similarly for task classification (S. S...c%%=SS-TT.ac.. and Tacc%==TiS S.acc% for SLP-AE in Table []. which further accentuates their topological similarities seen in Figure 3a
Contrastive learning in the latent space itself has nothing to do with decoding the structure of the data for reconstruction. The encoder is simply trained to learn an embedding of the subject and task content of the signal. The decoder must, therefore, do the heavy work of extracting this structural information itself. This exposes a property which is applicable to the not contrastive learning. When the stand-alone latent- pecett-prmuuttionn method is used (SLP-AE), the decoder is allowed a reliability in the latent spaces.
method but
A lot of the EEG signal is structural information, therefore, there might be more to gain from minimizing the cermutation-invariann to structural information rather than encoding the subject and task content directly. Thereby, the decoder can learn permutation cermutationiinvaiian invariaat structural encoding of the signal in both latent spaces, which allows the decoder to rely on this information irrespective of the permutation or swap since only one latent space (see Figure 2 is swapped at time. Thus, the permutatoon only trained model (SLP-AEF) does not learn explicitly disentangled representations of the subject and task content of the signal, but rather duplicated latent space iettuuttttooiiivviiaan
9

---

structural encoding which is highly correlated with both subject and task content of the signal. This interestingly is also the goal of the standard autoencoder. The satent-permutatie method allows the model to learn an encoding explicitly in the latent space instead of implicitly in the encoder/decoder networks. This might be a powerful tool since it further constricts the bottleneck in the auto-encoder sense, although at the cost of suboptimally encoding identical latent spaces. This can be, and is, remedied by using both contrastive learning and petent-permutation in conjunction.
Having completely disentangled latent spaces is a local minimum in the latent-permutation method and allows for the ideal swap in Figure 2h. The CSLP-AE model is able to keep the latent spaces disentangled while also providing the structural encoding information required for the conversion method work. This evident from Table [ which shows that the stand-alone tatent-permuttin model (SLP-AE) achieves about half (= 51.7%) the MSE error of the C-AE model on the ERP conversion tasks using samples from different tasks or subjects (D.s, S.t. and S.s., D.t.), and the CSLP- AE model retains this performance. We propose the antent-permutation method as replacement for the standard auto-encoder reconstruction loss to be used in conjunction with contrastive learning and the batch construction method to provide disentangled latent spaces which also allows for the structural encoding information to flow through and ease zero-shot conversion.
The latent-permutationn method does not increase performance on the (D.s., D.t.) conversion task. Arguably the difference in performance between the stand-alone latent- secenttprmutttionn and contrastive learning methods (on D.s, S.t. and S.s., D.t.) is due to this structural encoding property. An analysis provided in the appendix providing a generalization of the aatent-permutann method onto different subject/differeet task conversion to circumvent the structural encoding reliability. Further research could explore different avenues for keeping structural information while also disentangling subject and task latents, or providing a source for this structural encoding using generative methods or distributional power to generate structure, e.g. through the use of a VAEs (30]
5 Conclusion
In this paper, a novel split-latent permutation framework was introduced for disentangling subject and task content in EEG signals and enabling single-trial srrill zero-shot conversion. By combining the pro- posed split-latent permutation framework with contrastive learning, we achieved better performance compared to standard deep learning methods on the standardized ERP Core dataset. The experimental results demonstrated a significant 51.7% improvement in ERP conversion loss on unseen to unseen subject conversions. The method also achieved high single-trial subject classification accuracy (80.32+0.28%) and single-trial task classification accuracy (48 48+0.34%) on unseen subjects.
The conversion results in this paper are limited to the single dataset on which they were trained, and may not generalize other datasets with different experimental conditions. The ERP Core dataset is meant to standardize ERP measurements with more data to come in the future from other laboratories which might alleviate this limitation. Furthermore, the tasks used in the experiments are all seen during training. This limits the scope of the results to the seen tasks and no conclusions can be drawn about the generalization of the model unseen tasks.
Limitations
Broader Impact Zero-shot conversion of EEG signals, similar other deep fake methods, can be used for malicious purposes. Methods discussed in this paper intrude on one of the most sacred places yet be exploited and confused by technology: the human mind. Malicious use of this technology includes the ability to decode thoughts and intentions from EEG signals, and the ability to create fake
EEG signals confuse verification systems using EEG signals as a biometric identifier [ 37 5T].
Care must be taken when developing and deploying such technology to ensure that it is not used for malicious purposes. However, can also be used to improve the quality of life for people with and without disabilities. It provides a base for generalizing EEG signal representations across subjects and tasks, which can be used to improve the performance of BEG-based BCI systems, especially on a single-trial level as attestable by the results in Table I Similarly, it could provide base for novel analysis strategies, such as predicting drug or stimulus reactions in a healthy versus diseased brain, or as biomarkers of brain disorders.
10

---

Acknowledgments and Disclosure of Funding
Funding in direct support of this work: Lundbeck Foundation grant R347-2020-2439.
References
[1] A. C. Belkina, C. O. Ciccolella, R. Anno, R. Halpert, J. Spidlen, and J. E. Snyder-Cappione. Automated optimized parameters for T-distributed stochastic neighbor embedding improve visualization and analysis of large datasets. Nature Communications, 10(1):5415, 2019.
[2] Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798-188) 2013.
[3] B. Blankertz, R. Tomioka, Lemm, M. Kawanabe, and K.-R. Muller. Optimizing spatial filters for robust EEG single-trial analysis. IEEE Signal Processing Magazine, 25(1):41-50 2007.
[4] L. Bollens, T. Francart, and H. Van Hamme. Learning Subject-Invariant Representations from Speech-Evoked EEG Using Variational Autoencoders. ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1256 1260. IEEE, 2022.
[5] K. H. Brodersen, C. S. Ong. E. Stephan, and M. Buhmann. The balanced accuracy and its posterior distribution. In 2010 20th International Conference on Pattern Recognition, pages 3121-3124. IEFE, 2010.
[6] T. Chen and C. Guestrin. XGBoost: Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pages 785-794, 2016.
[7] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1597-1607, PMLR, 2020.
[8] S. Chopra,
R. Hadsell, and Y. LeCun. Learning similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'0S) volume 1, pages 539-546. IEEE, 2005.
[9] Chorowski, R. Weiss, S. Bengio, and A. Van Den Oord. Unsupervised Speech Representa- tion Learning Using WaveNet Autoencoders, IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(12):041-2053 2019.
[10] J.-C. Chou, C.-C. Yeh, and H.-Y. Lee. One-shot voice conversion by separating speaker and content representations with instance normalization. lrXiv:9904005742, 2019.
[11] M. Del Pozo- -azos, J. B. Alonso, J. R. Ticay-Rivas, and C. M. Travieso. Electroencephallggrmm subject identification: A review. Expert Systems with Applications, 41(15):6537-6554 2014.
[12] H. Dose, J. S. Møller, H. K. Iversen, and S. Puthusserypady. An end-to-end deep learning approach to MI-EEG signal classification for BCIs. Expert Systems with Applications, 1 114: 532-542. 2018.
[13] N. W. Duncan and G. Northoff. Overview of potential procedural and participant-related confounds for neuroimaging of the resting state. Journal of Psychiatry and Neuroscience, 38 (2):84996, 0044-66.20 2013.
[14] E. Eldele, Z. Chen, C. Liu, M. Wu, C.-K. Kwoh, X. Li, and C. Guan. An attention-based deep learning approach for sleep stage classification with single-channel EEG. JEEE Transactions on Neural Systems and Rehabilitation Engineering, 29:809- 818, 2021.
[15] K. Fukushima. Cognitron: A self-organizing multilayered neural network. Biological Cybernet- ics, 20(3-4):121-136, 1975.
[16] Geurts, D. Ernst, and L.. Wehenkel. Extremely randomized trees. Machine Learning, 63:3-42, 2006.

---

[17] E. Gibson, N. J. Lobaugh, S. Joordens, and A. R. McIntosh. EEG variability: Task-driven or subject-driven signal of interest? Neurolmage,
2022.
[18] A. L. Goldberger, Mietus, G. B. Moody, C.-K. Peng, and H. E. Stanley. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation, 101(23): e215-e220, 2000.
Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J.
[19] M. Golmohammadi, A. H. Harati Nejad Torbati, S. Lopez de Diego, I. Obeid, and J. Picone. Automatic analysis of EEGs using big data and hybrid deep learning architectures. Frontiers in Human Neuroscience, 13:76, 2019.
[20] M. Grosse- Grosse-Wentrup and M. Buss. Multiclass common spatial patterns and information theoretic feature extraction. IEEE Transactions on Biomedical Engineering. 55(8):1991-200 2008.
[21] R. Hadsell, S. Chopra, and LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference Computer Vision and Puttern Recognition (CVPR (C))) volume 2, pages 1735-1742. IEEE, 2006.
[22] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition (CVPR), pages 770-778, 2016.
[23] G. Hinton and R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504-007, 2006.
[24] 7. Hu, 7. Zhang, 7. Liang. I.. Zhang, L. Li, and G. Huang. A new perspective on individual reliability beyond group effect for event-related potentials: multisensory investigation and computational modeling. Neurolmage, 250: 1 18937, 2022.
[25] A. Jaiswal, A. R. Babu, M. Z. Zadeh, D. Banerjee, and F. Makedon. A survey contrastive self-supervised learning. Technologies, 9(1::2 2020.
[26] S. Jirayucharoensak, S. Pan-Ngum, and P. Israsena. EEG-based emotion recognition using deep learning network with principal component based covariate shift adaptation. The Scientific World Journal, 2014, 2014.
[27] H. Kameoka, T. Kaneko, K. Tanaka, and N. Hojo. StarGAN-VC: non-parallel many-to-many Voice Conversion Using Star Generative Adversarial Networks. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 266-273. IEEE, 2018.
[28] E. S. Kappenman, J. L. Farrens, W. Zhang, A. X. Stewart, and J. Luck. ERP CORE: An open resource for human event-related potential research. Neurolmage, 225:117465,, 2021.
[29] B. Kemp, A. H. Zwinderman, B. Tuk, H. A. Kamphuisen, and J. J. Oberye. Analysis of sleep-dependent neuronal feedback loop: the slow-wave microcontinuity of the EEG. IEEE Transactions Biomedical Engineering, 4709):185--114,, 2000.
[30] D. P. Kingma and M. Welling, Auto-Encoding Variational Bayes. arXiv:1312.6114 2013.
[31] D. Kobak and P. Berens. The art of using t-SNE for single-cell transcriptomics. Nature Communications, 10(1):5416, 2019.
[32] Z. J. Koles, M. Lazar, and S. Z. Zhou. Spatial patterns underlying population differences in the background EEG. Bruin Topography, 2(4):275-284, 1990.
[33] A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet Classification with Deep Convolutional Neural Networks. Communications of the ACM, 60(6):84-90, 2017.
[34] M. Längkvist, Karlsson, and Louthi Sleep stage classification using unsupervised feature learning, Advances in Artificial Neural Systems, 2012.
[35] J. Li and A. Cichocki. Deep learning of multifractal attributes from motor imagery induced BEG. In International Conference on Neural Information Processing, pages 503-510. Springer, 2014.
12

---

[36] J. Li. Z. Struzik. L. Zhang, and A. Cichocki. Feature learning from incomplete EEG with denoising autoencoder. Neurocomputingg 165:23-31, 2015.
[37] E. Maiorana. Learning deep features for aaskindependeett HEG-based biometric verification. Pattern Recognition Letters, 143:122-129, 2021.
[38] C. Morabito, M. Campolo, N. Mammone, M. Versaci, S. Franceschetti, F. Tagliavini, Sofia,
D. Fatuzzo, A. Gambardella, A. Labate, et al. Deep learning representation from electroen- cephalography of early-stage Creutzfeldt-Jakob disease and features for differentiation from rapidly progressive dementia. International Journal of Neural Systems, 27(02): 16000)9,,5009, 2017.
[39] Mousavi, Afghah, and U. Acharya. SleepEEGNet: Automated sleep stage scoring with sequence to sequence deep learning approach. PloS one, 14(5):e021 6456, 2019.
[40] A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, Se- nior, and K. Kavukcuoglu. WaveNet: A Generative Model for Raw Audio. arXiv:1609.03499 2016.
[41] A. v. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. ar 18X7:1807.007455 2018.
[42] O. Özdenizei, Y. Wang, T. Koine-Akino, and D. Erdogmuş. Learning invariant representations from EEG via adversarial inference. IEEE Access, 8:27074-27085, 2020.
[43] W. Peng. EEG preprocessing and denoising. EEG Signal Processing and Feature Extraction, pages 71-87, 2019.
[44] J. interface interface using multi-frequency visual stimulation and deep neural networks. In 2018 International Conference on Electronics, Communications and Computers (CONIELECOMP), pages 18 24. IEEE, 2018
J.
and J. Aspina--ermande Development of brain computer
[45] H. Phan, Andreotti, N. Cooray, O. Y. Chén, and M. De Vos. SeqSleepNet: end-to-end hierarchical recurrent neural network for bequence-to-sequeece automatic sleep staging. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2703):400410,, 2019.
[46] H. Phan, O. Y. Chén, M. C. Tran, P. Koch, A. Mertins, and M. De Vos. XSleepNet: Multi-view sequential model for automatic sleep staging. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):50335515,, 2021.
[47] H. Phan, Mikkelsen, O. Y. Chén, P. Koch, Mertins, and M. De Vos. SteepTrasffommer: Au- tomatic Sleep Staging with Interpretability and Uncertainty Quantification. IEEE Transactions on Biomedical Engineering, 69(8):2456 2467, 2022.
[48] K. Qian, Y. Zhang, S. Chang, X. Yang, and M. Hasegawa-Johnson. AutoVC: Zero-shot voice style transfer with only autoencoder loss. In International Conference Machine Learning (ICML), pages 5210-5219. PMLR, 2019.
[49] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pages 3748-8763 PMLR, 2021.
[50] S. Rayatdoost, Y. Yin. D. Rudrauf, and M. Soleymani. Subject-invariant eeg representation learning for emotion recognition. In ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3955-3959, 2021. S.ASSP39728.2021.9002
doi: 10.1109/
[51] K. Revett, Cognitive biometrics: novel approach to person authentication. International Journal of Cognitive Biometrics, 1(1):1-9, 2012.
[52] Y. Roy, H. Banville, I. Albuquerque, A. Gramfort, T. H. Falk, and J. Faubert. Deep learning- based electroencephaloggaahy analysis: a systematic review. Journal of Neural Engineering, 16 (5)0550011, 2019.
13

---

[53] A. B. Said, A. Mohamed. T. Elfouly, K. Harras, and Z. J. Wang. Multimodal deep learning approach for joint EEG-EMG data compression and classification. In 2017 IEEE Wireless Communications and Networking Conference (WCNC), pages 1-6. IEEE, 2017.
[54] G. Schalk, D. J. McFarland, T. Hinterberger, N. Birbaumer, and J. R. Wolpaw. BCI2000: general-purpose brain-computer interface (BCI) system. IEEE Transactions on Biomedical Engineering, 11(6):1034-10) 1043, 2004.
[55] M. L. Seghier and J. Price. Interpreting and utilising intersubject variability in brain function. Trends in Cognitive Sciences, 22(6):517-530, 2018.
[56] Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang. Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan, et al. Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4779-4783. IEEE, 2018.
[57] X. Shen, X. Liu, X. Hu, D. Zhang, and S. Song. Contrastive learning of subject-invariant EEG representations for cross-subject emotion recognition. IEEE Transactions Affective Computing, 2022.
[58] S. Shurrab and R. Duwairi. Self-supervised learning methods and applications in medical imaging analysis: A survey. PeerJ Computer Science, 8:e1045, 2022.
[59] K. Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016.
[60] Stober, A. Sternin, A. M. Owen, and J. A. Grahn. Deep feature learning for EEG recordings. arXiv:1511.04300 2015.
[61] H. Tang. W. Liu, W.-L. Zheng, and B.-L. Lu. Multimodal emotion recognition using deep neural networks, In International Conference on Neural Information Processing, pages 81 1-819. Springer, 2017.
[62] A. Thomas, C. Ré, and R. Poldrack. Self-supervised learning of brain dynamics from broad neuroimaging data. Advances Neural Information Processing Systems, 35:21255-21263 2022.
[63] R. Tripathy and U. R. Acharya. Use of features from RR-time series and EEG signals for automated classification of sleep stages in deep neural network framework. Biocyberneties and Biomedical Engineering, 38(4):890-90) 902, 2018.
[64] D. Ulyanov, A. Vedaldi, and V. Lempitsky. Instance normalization: The missing ingredient for fast stylization. 1rXiv:1607.08002, 2016.
[65] L. Van Der Maaten. Accelerating t-SNE using tree-based algorithms. The Journal of Machine Learning Research, 15(1):3221-3245, 2014.
[66] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t.. Kaiser, and I. Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
[67] X. Wang, M. Hersche, B. Tömekce, B. Kaya, M. Magno, and L. Benini. An Accurate EEGNet- based Motor Imagery Brain Computer Interface for Low-Power Edge Computing. In 2020 IEEE International Symposium on Medical Measurements and Applications (MeMeA), pages
1-6. IEFE, 2020.
[68] S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, N. E. Y. Soplin, J. Heymann, M. Wiesner, N. Chen, et al. ESPnet: End-to-Ei End Speech Processing Toolkit. 1040001804.00015, 2018.
[69] K. Q Weinberger and L. K. Saul. Distance metric learning for large margin nearest neighbor classification. Journal of Machine Learning Research, 10(2), 2009.
[70] Wen and 7. Zhang. Deep convolution neural network and autoencoders based unsupervised feature learning of EEG signals. IEEE Access, 6:25399-2545 224100, 2018.
14

---

[71] D.-Y. Wu and H.-Y. Lee. One-Shot Voice Conversion by Vector Quantization. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7734- 7738, Barcelona, Spain, 2020.
[72] D.-.. Wu, Y.H. Chen, and H.-y. Lee. VQVC+: One-Shot Voice Conversion by Vector Quanti- zation and U-Net Architecture. In Interspeech 2020, pages 4691-4695, Shanghai, China, 2020. ISCA.
[73] J. Xie, J. Zhang, J. Sun, Z. Ma, L. Qin, G. Li, H. Zhou, and Y. Zhan. A Transformer-baseee approach combining deep learning network and spetialltemporal information for raw BEG classification. IEEE Transactions Neural Systems and Rehabilitation Engineering, 30:
2126-2136, 2022.
[74] H. Xu and K. N. Plataniotis. Affective states classification using EEG and eepessupereieed deep learning approaches. In 2016 IEEE 18th International Workshop on Multimedia Signal Processing (MMSP), pages 1-6. IEEE, 2016.
[75] Z. Yin and J. Zhang. Recognition of cognitive task load levels using single channel EEG and stacked denoising autoencoder. 2016 35th Chinese Control Conference (CCC), pages 3907-3912. IEEE, 2016.
[76] Z. Yin and J. Zhang. Cross-session classification of mental workload levels using EEG and an adaptive deep learning model. Biomedical Signal Processing and Control, 33:30-47, 2017.
[77] Y. Yuan, G. Xun, F. Ma, Q. Suo, H. Xue, K. Jia, and A. Zhang. A novel channel-aware attention framework for multi-channel EEG seizure detection via multi-view deep learning. 2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), pages 206-209. IEEE, 2018
[78] X. Zhang, L.. Yao, D. Zhang, X. Wang, Q. 7. Sheng, and Gu. Multi-person brain activity recognition via comprehensive EEG signal analysis. In Proceedings of the 14th EAI Interna- tional Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services, pages 28-37, 2017.
[79] T. Zhu, W. Luo, and F. Yu. Convolution-an and attention sateetion-bbsed neural network for automated sleep stage classification. International Journal of Environmental Research and Public Health, 17 (11):4152, 2020.
15

---

A Appendix
A.1 Structural encoding and generalization of split-latent permutation loss
We observed a tendency in the SLP-AE model trained using only the split-latent permutation loss, in which the model would simply learn identical latent spaces. We discussed how this tendency stems from the information-propagatiin trained using the self-reconstrruction loss where one of the latents given as input to the decoder is swapped with that of another sample where both samples either belong to the same task or the same subject. Given that only one latent is swapped at a time, the model can learn a permutation-invariae encoding of the signal not specific to either the subject or task content of the signal, and it could rely on this information during (S.s., S.t.), (D.s., S.t.). (S.s., D.t.) conversion.
through one of the latents during training. Split-latent permutation
In this material we generalize the split-latent permutation using a quadruplet sampling method instead to instances where the input sample and the reconstruction target (output sample) is not the same, but which in special cases becomes identical to both the split-latent permutation and the self-reconstructiinn loss.
During training we sample a batch of K quadruplets, {(Xg, X X5, ( For each index of the batch l we choose two random subjects, Us. and Vs, and two random tasks, Mj and Ng. The quadruplet samples are sampled such that
Xg has subject and task (Uk, Ms)
(6)
X has subject and task (V&, M&) X has subject and task (Uk, N&)
(7)
(8)
X has subject and task (V&, N₂)
(9)
Similar to the split-latent permutation, the encoding of these quadruplets should match and disentan- gle the subject and task content into their respective latent spaces, such that a latent-swap between two latents (which ideally encode the same information) has minimal impact the reconstruc- tion/conversion. With these quadruplets we can now generalize the split-latent permutation such that both latents are swapped with latents from other samples which should encode the same information. The samples encode the following latents
X encodes (=) 2
(S.a) (T.a))
(10)
x encodes (z (8 (T.b))
, =
(11)
X encodes (2 (8,c) (T.e))
(12)
x encodes (2 2/6
(S,d) (T,d))
(13)
where
#k and z
(5.a)
(S.c)
both ideally encode Up
(14)
(S,b)
(S,d) both ideally encode Vi
and 2k
(15)
Ak
(T
and z
(T.b) both ideally encode Mj
(16)
2k
(T.o)
and (T.d) both ideally encode N z
(17)
All of these pairs of latents which ideally encode the same information are swapped in the generalized salit-latent permutation loss, which we will refer to as the quadruplet permutation loss (QP-loss). With this full swap of latents, there is no direct path between the input sample and the output sample, and the model is forced to encode the subject and task content into their respective latents. The reconstructions are as follows
x = (z(2
(S.c)
(T.b))
should reconstruct X
(18)
X D& (S.d)
(T.a) should reconstruct X
,7k
(19)
X D (S.a)
P=(22(5,)) .2
(T.d)
should reconstruct X
(20)
x = D (8,)) (T,))
(2(2 .
should reconstruct X
(21)
16

---

Quadruplet Permutation Loss
SS.a)
X
Xa
z (Taa)
(S,b)
V
x
X
M
(T)))
M
M
S(S.c)
U
X
X c
2 (T.c)
(S,d)
V
X
Xd
(Td)
N
Figure 4: Illustration of the quadruplet permutation loss. The quadruplet permutation loss is generalization of the split-latem permutation loss, where latents are swapped in pairs such there no direct path between the input sample and the reconstruction. The quadruplet permutation loss illustrated with the quadruplet (X2, x.. Xf. Xg) where latents are swapped such that z (8.a) and z (S.e) are swapped, z (T.a) and
((Th)) are swapped, etc., hefore decoding. This is done all quadruplet samples yiclding the quadruplet #k permutation loss as the MSE loss between the input sample and the reconstruction.
The swap of latents is illustrated in Figure 4 The quadruplet permutation loss is defined
Cop = 4K
- -
X 2112 |XX
2112 + | - x(13
(22)
1=1
The quadruplet permutation loss collapses to the split-latent permutation loss in some special cases. When input samples Xj and X are the same, then it becomes the same-subject permutation loss, and when input samples X and X are the same, then it becomes the same-task permutation loss. In the case where all input samples the same, then the quadruplet permutation loss becomes the self-reconstruction loss.
Quadruplet permutation loss results
We provide here results using the same training and testing setup in the paper. We conduct an ablation study on contrastive learning, peent-permutation and quadruplet permutation loss. The following four models are trained in five repetitions each:
SQP-AE: Quadruplet permutation loss only.
CSQP-AE: Quadruplet permutation loss and contrastive loss in conjunction.
SQLP-AE:
Quadruplet permutation loss and
loss in conjunction.
CSQLP-AE: Quadruplet permutation loss, contrastive loss and latent antent-permutatiin loss in conjunction.
Here we see that the quadruplet permutation loss degrades performance on subject classification accuracy considerably, but with similar performance task classification accuracy. We also see that
17

---

Table 3: Single-trial balanced subject classification accuracy (S.acc%) lask-on-subject classification accuracy (Tl S.acc%), task classification accuracy (Tacc))) subject-on-task classification accuracy (S) T) and zero- shot same-subject same-task ERP conversion MSE (S.s., S.t.), -ifferent-subject different-task ERP conversion MSE (D.s, D.t.), different- differeet-subject same- same-task ERP conversion MSE (D.s., S.t.). same- same-subject different taak ERP conversion MSE (S.s. D.t.) All ERP conversion MSE values have scales of 10 V2. Epoch window was Is.
Model
S.0ce%
ace%
Tace%
S-Tacck
(S.s.
(D.s,
(Ds.,
(S.s. D.t.
CSQLP-AE
76.10
0.76
43.36
0.46
46.17
76.47
+ 0.46
1.91
0.08
6.90 + 0.05
3.43 + 0.05
394 0.16
SQLP-AE
69.26
0.50
46.86
1.35
44,80
+ 0.77
69.60
1 0.25
1,48 0.05
6.44 + 0.03
287 1 0.10
2,97 0,05
CSQP-AE
73.04
0.30
1042 35.89
44.83
1:21
71.56
1: 0.64
6.20 0.12
7.00 +: 0.08
6.60 1 0.11
6.59 1 0.10
SQP-AE
0.33
43.92
0.29
48.88 0.13
0.39
0.07
6.99 +: 0.04
6.07 + 0.04
5.99 0.06
CSLP-AE
80.32 0.23
45A1
0.37
48.48 +
79.64
09.6440..37
4.21
20,06 + 0.10
5.80 0.15
6.6 0.23
SLP-AE
74.63 0.74
47:23 + 0.31
47,00
0.33
74.70
0.73
3.82 0.04
19.92 + 0.10
6.12 + 0.09
5,02 0.08
C-AB
0.44
37.34 0.45
46.59 + 0.23
73.27 + 0.25
4.28 0.06
20.28 + 0.07
11.33 + 0.47
10.64 0.30
the (D.s., D.t.) conversion loss now is on the same level as (D.s., S.t.) and (S.s., D.t.) conversion for the models without latent permutation. Adding the latent- prrent-prmmtttiin,, although it introduces the structural encoding pathway to the model again, considerably decreases both (S.s., S.t.) conversion compared to CSLP-AE and SLP-AE.
provide 1-SNE Figure 5a Furthermore, we provide (D.s., D.t.) conversion examples in Figure 5b for the generalized models and the CSLP-AE model.
plots of the generalized models here and the CSLP-AE model in
CSOP-AE
Model:
CSQLP-AE
by
Color by 5.
943002 22
Su2gect 28
space colored by T
Tart 2: ERreCameet
999 ants
Bock snPerStae dasts
(a)
conversion
Target Subject:: 23, Target Paradigm: ERN, Cha
ERWhncorect
ERN/Corect
Teal
50P45
50
53
30.
30-
30
-20
-2.4
0.0
0.4 -04
0.0
0.
Tem
Time
(b)
Figure 5: (a) f-SNE plots of split-latents as encoded on the (unseen subjects), colored by true labels. Rows show subject and latent spaces, respectively, while columns indicate models CSLP-AE, SQP-AE, CSQP-AE, SQLP-AE, and CSQLP-AE respectively. (b) (D.s., D.t.) converted ERPs from the same five models for a random target subject and target paradigm. All latents used in conversion were from unseen subjects on the test set, i.c. unseen to unseon conversion.
18

---

The SQP-AE model achieves specialized latent spaces with disentangled subject and task content as evident from the t-SNE plots in Figure Notably, the latent space similar to the CSLP-AE model, but using simply the quadruplet permutation loss. In this sense, the quadruplet permutation loss is similar to the contrastive loss that encourages the model learn disentangled latent space while also directly learning all conversion schemes. Further research could focus on this quadruplet permutation loss and its relation to the contrastive loss. We view it as a contrastive loss that relates input samples to the output sample (the reconstruction), i.c. auto-encoder contrastive loss, whereas a standard contrastive loss operates in the latent space itself. When we add the letent-permutation that the structural encoding property occurs again in the SQLP-AE
loss back to the model, model, while the CSQLP-AE model does not have this property due to the contrastive loss used in specializing the latent space. Therefore, the SQP-AE model might be most comparable to the CSLP P-AE model from the paper, as it both optimizes for conversion and latent disentanglement.
19

---

A.2
Loss component scaling
In this section we provide loss progressions on the ERP Core dataset for 200 epochs with standard enror of the mean over the different runs shown as confidence bounds.
trainiooll, Joss
trainisub, Looor 5
tra nwtooet.ccom
contrott nntock_contro_t
200 Con
onal 2a15
3.4
Figure 6: Training loss progressions on ERPCore [iv] dataset. Total loss is sum of loss components. sub_cout wub_contra_s and task_ _ool__cotrr_____ J are contrastive losses in the subject and task latent spaces respectively. recon is the reconstruc- tion loss. latent_ permute_____ and latent__aemmnne permute_ are the latent permutation loss terms respectively. The pink loss progressions from CSLP-AE, the purple progressions from SL. and the orange/brown progressions are from C-AE.
20

---

A.3
Glossary of symbols and abbreviations
Table 4: Comprehensive glossary symbols and abbreviations
Symbol/Abbreviatioon
Meaning
EEG
Electroencephalography
ERP
Event-related potential
BCI
Brain-computer interfacing
CV
Cross-validation
MSE
Mean squared error
AE
Auto-encoder
CNN
Convolutional Neural Network
FHVАE
Factorized Hierarchical Variational Autoencoder
CSP
Common Spatial Pattern
CSLP-AE
Contrastive Split-Latent Permutation Autoencoder
SLP-AE
Split-Latent Permutation Autoencoder
C-AE
Contrastive Autoencoder
CL
Encoder using contrastive loss
CE
Supervised Encoder using cross-entropy
CE(1)
Supervised Encoder using cross-entropy in task space only
Eo()
Encoder parameterized by Decoder parameterized by Ø EEG Sample used as input to encoder Subject latent space
Do
X
S
T
Task latent space
C
A given latent space (one of subject task) Subject latent
2(S)
T(T)
Task latent
X
Reconstructed EEG sample from output of decoder
(X x!)
Pair of EEG samples at i'th batch index with either common subject or task
(2))))) (Tal)
Subject and task latent from encoding X Subject and task latent from encoding X Reconstructed EEG sample of X where task latents are swapped in latent space
(8,b) (b))
X (T.))
(((T,0)
Reconstructed EEG sample of x where task latents are swapped in latent space T
LLP(C;X33 X!)
Split-latent permutation loss which swaps latents latent space L and where X and X have either common subject or task corresponding to C --pair cross-entropy loss latent space for the ''th row Symmetric temperature-scaal
LNN-Xmn (C:,,,
Lcurp(:;,,))
cross-entrope loss in latent space L
(S.s.. S.t.)
Same subject, same task conversion scheme Same subject, different task conversion scheme Different subject, same task conversion scheme Different subejct, different task conversion scheme Target subject Target task
(S.s., D.t.)
(D.s., S.t.)
(D.s., D.t.)
a(9.)
k 'th reconstructed EEG from targets
CERP
Average ERP from conversion with sampled targets
$PRP
Average ERP from ground truth targets
(a))
21


Reviews:
[{"review_id": "Hbr3MbcRce", "reviewer": "Reviewer_VVQF", "summary": "The paper proposes the use of swappable auto-encoder embedding along with contrastive learning in the embedding space to improve the performance of subject and task classification on evoked potentials in EEGs beyond using just an auto-encoder with contrastive learning in the embedding space. The supervised performance is improved and zero-shot conversion of EEG to new subjects or tasks is evaluated on real data from 40 subjects.     ", "questions": "Comment 1: Line 100, the statement of the choice of using a shared encoder, \"presently employ a shared encoder to reduce the number of parameters used\", fails to acknowledge that the style embedding may be more efficiently performed using different processing architecture. \n\nComment 2: A mix of capitals and lowercase for subject index is used in Figure 2, which doesn't match the text. I think there should be hats on the variables on the output boxes on right side.\n\nQuestion 1: In Figure 1, what are $k$ and $p$? \n\nQuestion 2: How many attention heads are in the transformer? \n\nQuestion 3:  Is it necessary to use non-linear classifiers in the split latent space? In much work on contrastive learning a linear classifier is used to probe the information content of the latent space. Is the use of a non-linear classifier necessary because of the complexity of the transformer in the decoder? \n\nComment 3: The proposed work seems to be a general approach (not specific to EEG). Notably, the papers approach is introduced with insights from speaker and speech conversion that used a fixed encoder for speaker. The concept of swapped latent permutations and contrast learning could be used more generally including with a pretrained encoder for either subject or task while training the other encoder and the shared decoder.  This possibility and any key difference should be discussed to round out the initial motivation. \n\nComment 4: In a similar vein to using a pretrained encoder, staggering the training of the encoders by freezing and unfreezing each of them such that one acts as conditional input is perhaps one way to disentangle the latent spaces of SLP-AE without contrastive learning. Any thoughts? \n\nComment 5: Figure 1 could be expanded as it is quite dense and has a relatively small font size. \n", "limitations": "The limitations are mentioned in the weaknesses. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well written and clear.\n\nThe results are on large dataset (for EEG)  show consistent improvements. \n\nThe discussion are extensive and insightful. The paper shows that alone split latent permutation does not promote disentanglement and contrastive learning is needed to encourage disentanglement.  This can be seen by comparing  the task-on-subject and subject-on-task to the task and subject classification, respectively, SLP-AE is actually higher on the former two and the gap for CSLP-AE is much less than the gap for the C-AE. \n", "weaknesses": "My main concern is the lack of discussion on the generality of this method for EEG beyond ERPs. It should clarify that the current approach is likely limited to ERP and other short task responses in comparison to responses that are not time locked or have more variation in phase and duration than ERPs (e.g., sleep stage classification or seizure early warning for epilepsy subjects). It is not clear to me how this approach would be applied to more continuous cases without time-locked responses in relatively short windows. Insights from the prior work reference on speech conversion are welcomed. \n\nWhile a number of ablations and baselines are tested, the papers results would be more significant if they were compared to the related work [4,37,42].\n\nThe methodology is quite expansive in terms of hyperparameter selection, but the fixing of the relative loss terms that may have vastly different units and scales (e.g., cross-entropy and MSE for instance) may not be optimal. I'd like to see more discussion on how these loss terms may interact during learning if they differ in scale. In particular it seems that C-AE could be made to perform better if perhaps the contrastive loss was weighted more heavily than the auto-encoder. The split permutation reconstruction provides some similar information in the same rage as the reconstruction error.  While likely out of scope it would be interesting to see how the proposed approach would work on simple image datasets with synthetic structure like a colored version of MNIST where the color substitutes for a subject.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments and careful assessment finding that the paper is well written and clear, results show consistent improvements on a large dataset (for EEG), and that the discussions are both extensive and insightful.\n### Re. Generality beyond ERPs.\nWe include two additional widely used benchmark datasets for EEG analyses: the  motor imagery (MI) dataset [x, iii], and the Sleep-EDF Expanded (SleepEDFx) database [v, iii]. For the MI dataset, we consider 3 s post-stimuli EEG for MI classification. Although MI is time-locked like ERPCore, the 3 s period reflects response time which can be considered less time-locked to stimuli. The SleepEDFx database was used to further investigate the generality of the proposed approach to non-stimuli-locked data as suggested.\nWe observe similar comparative performances as the ERPCore dataset on the MI, namely that the CSLP-AE procedure provides superior performance to C-AE and SLP-AE. We further contrast our approach to state-of-the-art (SOTA) procedures for the same datasets namely [i], [ii], [vi], [vii], [viii], [ix], [xi], [xii] and [xiii], and find that the performance of the proposed CSLP-AE is on-par with these current SOTA procedures for MI.\n### Re. Comparison to related work.\nWe did not include comparisons to [4,37,42] as the classification setup in [4] is based on binary classification using samples of same task vs. different task as opposed to the presently considered multi-task classification procedure.\n[37] and [42] did not produce explicit subject latent representations but used adversarial training to achieve latent task representations that are subject invariant. This is a slightly different aim than ours, which is to explicitly provide both subject and tasks latent representations which are further used in zero-shot conversion.\n### Re. Loss terms.\nTo keep the ablations simple and the computational overhead of model estimation reasonable we did not carefully tune regularization factors but fixed these at equal weight. Notably, loss magnitudes of the different components were on the same scale and with this setting of identical regularization the effect of the loss contributions had the anticipated impact on the generalization results. We have included in the PDF examples of the loss contributions highlighting their similar scale. We did not investigate careful tuning beyond the ablations systematically setting loss terms to zero.\n### Re. Comment 1: Line 100, the statement of the choice of using a shared encoder.\nWe will include this in the paper noting that other choices of architectures could be relevant also to explore such as a separate as opposed to shared encoder. We used a shared encoder to reduce the number of parameters. However, there is a small part of the bottleneck which is separate for each latent space. The encoder can thereby learn general patterns of the input and then separately learn specific encodings as part of the per-latent space separate ConvBlocks.  We do not reject the possibility of using separate encoders for both latent spaces and view this as an expansion of the framework.\n### Re. Comment 2: A mix of capitals and lowercase for subject index.\nWe apologize for the confusion. We will fix the figure to match the notation of the text.\n### Re. Question 1: In Figure 1, what are $k$ and $p$? \n$k$ is kernel size and $p$ is padding. We used zero-padding of 1 on each side of the signal when doing convolutions with a kernel size of 3 to have equal output. This will be clarified in the figure text.\n### Re. Question 2: How many attention heads are in the transformer? \nThe transformer models used a single fully-connected attention head per layer. We opted for this due to memory requirements of using more heads per layer rather than one single large head.\n### Re. Question 3: Is it necessary to use non-linear classifiers in the split latent space?\nThere is no guarantee that the latent space will be linearly separable due to the the reconstruction and latent permutation losses. We therefore used a standard XGBoost  procedure which is a non-linear classification procedure that is widely used and found to perform well in practice. To keep the analysis unbiased by choices of classification procedures we only considered this classifier for all the considered procedures and model ablations. Results from simpler classification procedures are provided in Table 17 of Supplementary Materials section G.\n### Re. Comment 3: Use of pre-trained encoders\nPretrained encoders have indeed been used successfully in the context of voice-conversion, c.f. [40]. However, a drawback of such pretrained encoders is that the resulting latent space from the non-fixed encoder relies on the fixed conditionings of the pre-trained encoder. The two encoders cannot then co-adapt and adjust their latent representations with respect to each other during training. Whereas we agree that such pretrained encoders could have been employed, our proposed approach of training both encoders simultaneously provides a simple procedure for learning, which we find more elegant than pretraining one of the encoders. However, it would be interesting in future work to explore if such pre-trained encoders can enhance performance potentially also by initializing the respective encoders from pretrained models tuned to characterize subject and task latent representations, respectively.  We did not find in practice challenges minimizing the complete loss, see also the included training curves in the Author Rebuttal PDF.\n### Re. Comment 4: Training the encoders by freezing and unfreezing each of them.\nSee answer to previous question.\n### Re. Comment 5: Figure 1 could be expanded as it is quite dense and has a relatively small font size.\nWe will increase the size of the figure including font sizes in the revised manuscript.\n### References: See Author Rebuttal."}, {"review_id": "AFmSGxiLoU", "reviewer": "Reviewer_qodj", "summary": "The authors hypothesize that converting EEG signals between tasks and subjects requires extracting latent representations that capture both content and style. They introduce a novel framework called CSLP-AE, which utilizes contrastive learning to guide the latent representations and achieve EEG conversion. The proposed approach outperforms conventional training methods, providing reliable characterizations of subject and task, and allowing for zero-shot conversion of unseen subjects. The CSLP-AE framework has the potential for broader applications in signal conversion and the analysis of biological signals.\n", "questions": "Why is the attention mechanism is necessary in this model? Why in Figure 3 does the task latent space t-SNE qualitatively show better separation of the tasks than the subject t-SNE if the CSLP-AE performed better on the subject classification task?\n\nFigure 3A reveals that SLP-AE exhibits a topologically similar embedding, whereas C-AE and CSLP-AE do not demonstrate a similar pattern. It would be beneficial to provide additional explanation for this discrepancy. Additionally, the t-SNE embeddings of T. space from C-AE and CSLP-AE show similarity, despite the author's claim that Split-Latent Permutation is a key contribution. Further clarification on this aspect would be helpful.", "limitations": "The generalizability of the model to different datasets and tasks is unknown. While the author acknowledges the limitation of relying on a single dataset, it would be beneficial to explore the applicability of their findings by conducting tests on other datasets with different experimental conditions. This would provide a more comprehensive understanding of the generalizability of the proposed approach.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper uniquely frames a voice conversion problem as an EEG analysis problem. The paper presents a new framework that leverages state-of-the-art deep learning models to perform this disentanglement task. The paper clearly describes the benefit of disentangling the content and style of an EEG signal. The paper also clearly describes the model architecture and the training objective. \n\nA main strength is the utilization of contrastive learning to guide the latent splits, effectively representing both subject (style) and task (content) aspects. By incorporating contrastive learning into the framework, the model gains the ability to disentangle and explicitly capture the unique characteristics associated with subjects and tasks, enhancing the overall interpretability and performance of the model. ", "weaknesses": "The main weakness of the paper is that it does show results for more datasets. The task- and subject- latent spaces were not quantitively compared to see how cluster separation improved with the CSLP-AE.\n\nLimited generalizability: The findings are constrained to the single dataset utilized in the experiments, despite efforts made to standardize protocols. \n\nLimited task representation: The tasks employed in the experiments were exclusively those seen during training. Consequently, the scope of the results is confined to these known tasks, and the ability to draw conclusions regarding the model's generalization to unseen tasks is limited.\n\nRestriction to time-locked paradigm: Only epochs with stimuli were included in the analysis, which restricts the scope of the conversion and representations to the time-locked paradigm. This limitation overlooks the potential application of the model to continuous cases, hindering a comprehensive analysis.\n\nLack of dataset diversity: The reliance on a single dataset, although standardized, may limit the representation of diverse experimental conditions. The inclusion of data from other laboratories in the future, as anticipated with the ERP Core dataset, could help mitigate this limitation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful review and constructive comments and for acknowledging that the paper uniquely frames a voice conversion problem as an EEG analysis problem presenting a new framework that leverages state-of-the-art deep learning models.\n### Re. Limited generalizability and lack of dataset diversity.\nWe will in the revised paper include two additional datasets considering tasks not covered by ERPCore, namely, the Movement/Imagery dataset [x] involving MI classification and the Sleep EDF Expanded sleep stage classification dataset [v] as detailed in the Author Rebuttal.\n### Re. Restriction to time-locked paradigm.\nTo address this limitation, we consider the sleep-stage classification task as outlined in the Author Rebuttal, which is not time-locked. We here observe that the procedure is able to characterize sleep-stages and thus not restricted to the time-locked paradigm, however, whereas it provides on par performance with SOTA for MI we find it to be inferior to current SOTA for sleep stage classification. We argue that this could be remedied by using encoders with larger receptive fields such as WaveNet [xv] instead of strided convolutions, which was also mentioned in the manuscript. For results and comparison to current state-of-the-art on these two additional datasets please see the included Author Rebuttal PDF.\n### Re. The task- and subject- latent spaces were not quantitively compared to see how cluster separation improved with the CSLP-AE.\nWe presently quantitatively evaluate the subject and task latent spaces by their ability to discriminate tasks and subjects which is commonly used to evaluate representation learning procedures (see for instance [xiv]. We agree that unsupervised clustering approaches could also be used for quantitative evaluation of cluster separation; however, this will involve several additional design choices, which will make comparative analysis difficult.\n### Re. Why is the attention mechanism necessary in this model?\nTransformer architectures have proven to provide state-of-the-art results across domains. In particular, their ability to efficiently learn and leverage contexts when extracting latent representations have been widely explored. We therefore included attention mechanisms through the Transformer architecture to leverage this capability.  Transformers have previously successfully been applied in the context of EEG data in [ii], [ix], and [xii] which we use as comparison in the Author Rebuttal.\n### Re. Why in Figure 3 does the task latent space t-SNE qualitatively show better separation of the tasks than the subject t-SNE if the CSLP-AE performed better on the subject classification task?\nWhereas the task classification accurately discriminates between the different tasks we observe that the task classification is more challenged separating the types of stimuli within a task, i.e. LRP contralateral vs. Ipsilateral and P3 rare vs. P3 frequent are difficult to separate whereas the different paradigms, i.e., LRP, P3 etc.  generally differentiates well from the other stimuli tasks. The classification performance is evaluated on not only correctly predicting the paradigm but also the two tasks within each stimuli paradigm correct, see also confusion matrices in the Supplementary material section F.2. This is the reason why the classification performance for task is in general lower than for subjects.\n### Re. Figure 3A reveals that SLP-AE exhibits a topologically similar embedding, whereas C-AE and CSLP-AE do not demonstrate a similar pattern. It would be beneficial to provide additional explanation for this discrepancy.\nWe attribute the similar topology of SLP-AE to the SLP-AE's ability to handle identical encodings in the subject and task spaces by the decoder learning to read out the subject and task information separately from the two spaces. To alleviate this issue, we propose the use of contrastive learning and note that this issue could also be alleviated with careful bottleneck tuning which in general is much more challenging to adequately tune than imposing contrastive learning to promote subject and task disentangling respectively in the two spaces. This is discussed in lines 144-152 of the paper. In the supplementary, we further consider a quadruplet latent permutation framework in which structural information cannot be preserved in the decoder and observe that this prevents the identical latent representations observed when preserving structural information in the split-latent permutation framework.\n### Re. Additionally, the t-SNE embeddings of T. space from C-AE and CSLP-AE show similarity, despite the author's claim that Split-Latent Permutation is a key contribution. Further clarification on this aspect would be helpful.\nAs the CSLP-AE includes contrastive learning we can expect some similarity between the two. However, from the quantitative evaluations of performance we observe that CSLP-AE has improved subject and task discrimination from the extracted latent spaces. These effects are consistently observed also on the movement/imagery dataset [x] provided in the Author Rebuttal.\n### References: See Author Rebuttal."}, {"review_id": "9PziOuBiCN", "reviewer": "Reviewer_4KfP", "summary": "Electroencephalography (EEG), a non-invasive neuroimaging technique, is often compromised by high noise levels and subject variability, making consistent signal extraction challenging. To mitigate this, the authors introduce a novel framework, the Contrastive Split-Latent Permutation Autoencoder (CSLP-AE). This framework enhances EEG conversion using contrastive learning, guiding latent representations and enabling the clear distinction of subject variability (style) and task activation (content). Beyond providing a better generalizable understanding of subjects and tasks, this technique also facilitates conversion between unseen subjects. This suggests it has a wider potential for signal conversion, content extraction, and style extraction in biological signal analysis.", "questions": "Could you provide a comparison of your method with other established approaches that disentangle content from style in the literature?\n\nWith regards to Equation (4), it appears that if the cosine similarity is chosen as the similarity metric,  $L_{NT-Xent}(L; Z', Z'', k) = L_{NT-Xent}(L; Z'', Z', k)$. Could you clarify why Equation (5) is then necessary?\n\nThe integration of Equation (3) with Equation (5) in the optimization process is not explicitly explained. Could you provide more details on how these equations are combined in the optimization?\n\nThe declared goal of the proposed method is to divide the representations into subject and tasks. How have you confirmed that this objective is met? Specifically, can subject representations predict the subject but not the task, and vice versa?", "limitations": "The authors addressed the limitations and potential negative social impact of their work. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is innovative in being among the first to separate style and content in EEG signals.\n\nThe proposed method outperforms its competitors in experimental tests. ", "weaknesses": "While the concept of disentangling content from style has been extensively discussed in the literature, the authors did not compare their method with these existing approaches in their experiments.\n\nIn Equation (4), it seems that $L_{NT-Xent}(L; Z', Z'', k) = L_{NT-Xent}(L; Z'', Z', k)$ if we choose the cosine similarity as the similarity metric. The necessity for Equation (5) is not fully clear. The authors also do not explain how to integrate Equation (3) with Equation (5) in the optimization process. \n\nThe declared goal of the proposed method is to divide the representations into subject and tasks. However, the authors did not provide concrete evidence demonstrating the achievement of this objective. Is it that the subject representations can predict the subject but not the task, and vice versa?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful assessment of the manuscript acknowledging that the paper is innovative in being among the first to separate style and content in EEG signals providing favourable performance and that it may have potential for wider adoption within biomedical signal processing and analysis problems.\n### Re. Could you provide a comparison of your method with other established approaches that disentangle content from style in the literature?\nAs also remarked to reviewer VVQF, we did not include comparisons to the existing EEG literature [4,37,42] for a few different reasons. In [4], the task classification setup is based on binary classification based on samples of same vs. different task as opposed to the multi-class task classification problem presently considered in the manuscript.\n[37] and [42] did not produce explicit subject latent representations but used adversarial training to achieve latent representations that are subject invariant. This is a slightly different aim than ours, which is to explicitly provide both subject and tasks latent representations. We agree that further comparison to state-of-the-art would strengthen the paper, and we have therefore included results on two additional datasets on movement/imagery [x, iii] and sleep stage classification [v, iii] using the same framework and model architecture as presented in the original manuscript. In the supplied Author Rebuttal PDF, we contrast our performance to current state-of-the-art on these two benchmark datasets. We observe on-par performance of our proposed approach to current SOTA baselines for the MI whereas the model can also operate for sleep-stage classification, i.e., on data that is not time-locked although at performance inferior to SOTA sleep stage classification procedures.\n### Re. Clarification of necessity of Equation (5)\nWhereas Equation (4) considers a given sample $k$ to all other samples, Equation (5) considers all combinations of pairs by summing over $k$. By doing so with both ($Z^A$, $Z^B$) and ($Z^B$, $Z^A$), the contrast is ensured to be based on all pair-wise combinations, as the normalization in Equation (4) is summing across the second latent embedding matrix where $i≠k$. The necessity of equation (5) is simply to provide a symmetrical contrastive loss across both rows and columns of the corresponding similarity matrix. This is also the loss used in [41].\n### Re. Integration of Equation (3) with Equation (5) in the optimization process.\nWe acknowledge that this should have been made clearer in the manuscript, and the revised version will reflect this. The two loss components Equation (3) and Equation (5) are combined into a joint loss function by adding the two loss components equally, i.e., Equation (3) and Equation (5) are both weighted by one in the optimization process, such that CSLP-AE minimises $L =L_{\\mathrm{LP}}(\\mathcal{S}; ·, ·)+ L_{\\mathrm{LP}} (\\mathcal{T}; ·, ·)+ L_{\\mathrm{CLIP}}(\\mathcal{S}; ·, ·) + L_{\\mathrm{CLIP}}(\\mathcal{T} ; ·, ·)$ (see also the description around line 194 of the manuscript). This means that Equation (5) is calculated for both the subject and task latents and added to the loss. Since all loss components have similar scale (as shown in the Author Rebuttal PDF) weighing each term the same (by one) was found to be justified.\n### Re. The declared goal of the proposed method is to divide the representations into subject and tasks. How have you confirmed that this objective is met? Specifically, can subject representations predict the subject but not the task, and vice versa?\nWe use classification performance to quantitatively evaluate the representations in characterizing subject and tasks as shown in Table 1. Specifically, we show the classification performances for task-on-subject (T⊢S) and subject-on-task (S⊢T) problems. Here, T⊢S refers to classifying tasks based on the subject latents, and vice-versa for the S⊢T problem. This is also detailed in lines 222-227 in the manuscript with supporting information concerning classifier training details in Section D.2 in the supplementary material. We would also like to refer to the discussion concerning structural encoding as an explanation for the (seemingly) similar latent space topologies (see lines 274-335 in the manuscript).\n### References: See Author Rebuttal."}, {"review_id": "XXuwgNwTSG", "reviewer": "Reviewer_MXTP", "summary": "This research paper introduces a novel framework called the contrastive split-latent permutation autoencoder (CSLP-AE) for EEG signal conversion, with the primary aim of disentangling task-related content and subject-specific style. This technique is based on disentangled representation learning which has been widely used in CV domains such as gait analysis and face recognition.\n\nThe authors hypothesize that the conversion of EEG signals between tasks and subjects necessitates the learning of latent representations that account for both content and style. They employ contrastive learning to guide these latent splits to represent the subject (style) and task (content) separately.\n\nThe efficacy of the CSLP-AE framework was tested on a standardized ERP dataset, ERP Core, which contains data from the same subjects across various standardized paradigms. The authors contrasted the performance of the CSLP-AE against conventional supervised, unsupervised, and self-supervised training.\n\nThe results of the study showed that the proposed framework significantly improved performance compared to standard deep learning methods on unseen-to-unseen subject conversions by 51.7%. Furthermore, it achieved high accuracy in single-trial subject classification (80.32±0.28%) and single-trial task classification (48.48±0.34%) on unseen subjects.\n\nIn essence, this study provides a novel and effective method for EEG signal conversion, significantly enhancing the accuracy and interpretability of EEG analyses. It further paves the way for potential applications in a broader range of biological signal-processing tasks.", "questions": "1. Could the authors specify whether any part of the network was frozen during different training tasks? Providing this information would help in understanding the training dynamics of the proposed model.\n\n2. In the baseline comparisons, the authors referred to 'CE' models trained with supervised learning cross-entropy loss. Could the authors detail the structure of the model used in this supervised training process? It would be beneficial if the authors could use well-established EEG decoding models as classification baseline comparisons. This would facilitate comparison with previously published results and provide more context for the performance of the proposed model.\n", "limitations": "Please refer to weaknesses", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The introduction of the contrastive split-latent permutation autoencoder (CSLP-AE) for EEG ERP classification is an innovative contribution to the field. By adopting disentangled representation learning from other domains to the EEG domain, the work paves the way for new avenues of research and applications in the bio-signal domain.\n\n2. The figures representing the framework architecture and different training tasks are very clear. This visual clarity significantly enhances the reader's understanding of the proposed method and its applications.\n\n3. The methods section contains detailed and coherent explanations, which enables an effective understanding of the process and intricacies involved in the proposed model.\n\n4. The manuscript provides a comprehensive comparative analysis of the proposed method with/without some loss components which verified the contribution of each proposed task. The results demonstrated the necessity and impact of each component, thereby fortifying the empirical robustness of the work.\n\n", "weaknesses": "1. While the diagrams and colours used in the Methodology section are commendable, the notations used in the methods section could be improved for enhanced comprehension. Providing a clear and comprehensive key for symbols and abbreviations would be beneficial for readers.\n\n2. The manuscript could further elaborate on how the model accommodates noise caused by other factors within the EEG data. Clarification on whether this noise is modelled within subject-specific or task-related features would be beneficial. Additional analysis and discussion on this topic would bolster the paper's validity.\n\n3.  Table 1 indicates effective feature disentanglement as evidenced by the comparison between T-on-S and S-on-T. However, the difference between S and S-T within the proposed model is not significantly different, suggesting subject-specific information is still greatly present within the T latent features. It would be beneficial to discuss potential reasons for this observation and how it aligns with the proposed model's disentanglement claim.\n\n4. The textual content in Figure 3 is slightly difficult to read due to its small size. Additionally, presenting t-SNE separately for S and T spaces may not intuitively display feature disentanglement. It might be more effective to reduce the total number of points for each colour and integrate both spaces into a single image for each model, using colour to represent S and shape variations to represent T. This modification could enhance the visual clarity and intuitiveness of the model's results.\n\n5. The paper currently offers limited evaluation of the reconstructed signal quality, with a single plot in the time domain (Figure 3. B). It would be valuable to include evaluation in the frequency domain as well. Implementing additional quantitative metrics such as the Inception Score and Frechet Inception Distance could supplement the existing Euclidean distance measure and thus, provide a more comprehensive evaluation of the reconstruction quality.\n\n6. While the paper presents compelling results, it could be further strengthened by providing more extensive comparisons with current state-of-the-art methods. For the classification tasks, a comparison with sophisticated EEG decoding methods based on CNNs would be beneficial. Similarly, for the reconstruction tasks, contrasting the proposed model's performance with other well-established techniques based on AE, VAEs and GANs would be insightful. These comparisons would help situate the proposed method within the larger context of cutting-edge techniques and provide a more robust validation of its performance.\n\n7. For improved reproducibility, it would be beneficial to provide a detailed outline of the model architecture and hyperparameters. For instance, the weight allocation between each loss component would be incredibly important for researchers seeking to reproduce or extend this work.\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful and constructive critique acknowledging that the proposed approach is an innovative contribution to the field with visual clarity, detailed and coherent explanations, and a comprehensive comparative analysis of the proposed method.\n### Re. Clear and comprehensive key for symbols and abbreviations.\nWe appreciate the comment and will include such comprehensive information in the revised manuscript.\n### Re. Accommodation of noise caused by other factors within the EEG data.\nThis noise is discussed in the manuscript as part of the structural encoding discussion, see line 305. Specifically, we consider structural information also constituted by noise. The AE objective will focus on also recovering noise-structures in the reconstruction admitted by the encoding of structural information. In the supplementary material, we discuss how the structural information can be blocked from the decoder by considering a quadruplet latent permutation (QLP, see Supplementary Figure 1, section A). Notably, QLP is also found to explicitly enhance conversion between different tasks and different subjects (D.s, D.t.) (see Supplementary Table 1).\n### Re. Subject-specific information is still greatly present within the T latent features.\nWe agree that subject information is still present in the task latent space and vice versa for the subject latent space, and we attribute this to the ability of the decoder to extract subject and task information from the respective spaces in which the two latent representations can be identically encoded (see also the discussion of this and the motivation for the use of contrastive learning in combination with the split latent permutation framework in lines 144-152). We further attribute this to inclusion of structural information and demonstrate how this can be remedied considering the QLP explored in section A of the supplementary material. Whereas contrastive learning promotes latent spaces to characterize task and subjects respectively as shown in Table 1, subject and task information is still preserved in the opposing latent spaces. We attribute this to the contrastive loss successfully promoting subject and task disentanglements in respective spaces; however, the high capacity of the spaces still accommodates a large degree of subject and task information. Careful bottleneck tuning can remedy this, for a discussion in the context of voice conversion see [40].\n### Re. Textual content in Figure 3 is slightly difficult to read due to its small size.\nWe apologize for the lack of clarity on the textual content of Figure 3, we will improve this in the revised manuscript. Less points could be a possibility, but the current t-SNE shows the topological similarities between different model latent spaces as intended. t-SNE plots showing the task latent space colored by subject and the subject latent space colored by task are provided in the supplementary material Section F.3.\n### Re. The paper currently offers limited evaluation of the reconstructed signal quality.\nDue to space limitations the main paper only included a single conversion plot in Figure 3.B. However, in the supplementary material section H we included further conversion plots on both test and training data for all paradigms and conversion schemes.\nWe quantitatively evaluate conversion reconstruction performance using the mean-squared-error metric as this is what the AE objective function is minimizing, and the mean square error loss is also optimal regarding minimizing mean discrepancies. Finally, the evoked response is calculated as the average across trials which is also least squares optimal. We therefore considered the MSE metric as opposed to  other procedures such as the inception score, Frechet inception distance, the scale-invariant signal to distortion, or correlation metric typically used in image and audio-processing.\n### Re. More extensive comparisons with current state-of-the-art methods.\nWe agree that additional analyses and comparison to current SOTA task classification will improve the manuscript and have therefore included two additional datasets on Motor Imagery (MI) [x] and sleep stage classification [v] as part of the Author Rebuttal and included a comparison to SOTA deep learning procedures for task classification on these datasets. We find find that our approach provides on par performance for MI and can even operate in the situation where data is not time-locked (I.e., sleep stage classification) although not at the level of current SOTA (see also the included Author Rebuttal associated PDF).\n### Re. For improved reproducibility, it would be beneficial to provide a detailed outline of the model architecture and hyperparameters.\nThis is included in both the manuscript and the supplementary materials. See line 210-214 in the manuscript as well as Supplementary section C and D.\n### Re. Could the authors specify whether any part of the network was frozen during different training tasks?\nNo parts of the network were frozen during different training tasks. Since losses were found to on similar scales (see Author Rebuttal PDF), we chose to use a complete loss function rather than stepwise minimizing the latent space losses individually.\n### Re. Could the authors detail the structure of the CE model used in this supervised training process?\nThe CE models with supervised learning use the same encoder as the other models, i.e., a strided CNN and a transformer bottleneck. This is very much akin to the t-CTrans model from [xii] although with strided convolutions instead of average pooling.\n### Re. Well-established EEG decoding models as classification baseline comparisons\nWe have for the two new included datasets also included comparison to current SOTA  for these datasets (see results in Author Rebuttal and PDF) in which we find that our procedure provides on par performance to current prominent SOTA methods for MI.\n### References: See Author Rebuttal."}]

Author rebuttal:
We thank the reviewers for their efforts in reviewing this manuscript and providing constructive and insightful comments. Individual responses to each reviewer’s questions and comments have been prepared and are provided below where appropriate. 

Multiple reviewers address the fact that results have only been provided for a single dataset (ERPCore). While the dataset does contain several different experimental paradigms, we agree that focusing on only ERP-related data might be too narrow in scope. We have therefore trained and tested our framework on two additional benchmark EEG datasets: the EEG Motor Movement/Imagery Dataset (EEGMMI) [x, iii] containing EEG recordings from 109 subjects performing several different motor imagery (MI) tasks; and the Sleep-EDF Expanded (SleepEDFx) database [v, iii] containing 153 single-EEG polysomnography studies from 78 unique subjects. The SleepEDFx is included to explicitly probe the model integrity on data that is not time-locked to an external stimulus, as opposed to the ERPCore dataset. 

As shown in the Author Rebuttal associated PDF, our framework can accurately classify MIs at a level that is on par with current state-of-the-art (64.28 ± 0.16 % for our model vs. 65.07% for an EEGNet-based model [xi], see Table 1 in the Author Rebuttal associated PDF). We also find that our framework generalizes well to EEG data that is not time-locked to an external stimulus. As shown in Table 2 in the Author Rebuttal associated PDF, our framework can predict sleep stages (Wake, REM, Non-REM 1, 2, 3) at a level of 75.16±0.96 %% accuracy with minimal changes to the framework. 

We applied our framework “as is” with the following minor modifications: since the SleepEDFx dataset contains a limited number of EEG channels, we only considered a single EEG channel and applied a short-time Fourier transform to fit the data to our setup. Furthermore, to match the latent dimension of the models from the present study, we opted to only use three blocks instead of four and resized the latent dimension’s temporal resolution from 64 to 40 to end up with latents of size 25x40 compared to the present study which had a 16x64 latents. Finally, these models were only run for 50 epochs due to time constraints and therefore not to convergence. Detailed information regarding experimental setups for the two additional datasets will be provided in the updated supplementary material. 

For the EEGMMI dataset the samples had 3 seconds duration at a sampling rate of 160 Hz, and we therefore needed to use five blocks instead of four to match the latent dimension of the present study. The models were run to convergence for 800 epochs. No other tweaks were made to conform to this dataset. 

While these results on additional databases are on par but do not exceed state-of-the-art, they are trained "as is”, employing the architecture, most hyperparameters and framework as used in the present study. Additionally, they are based on learning self-supervised latent representations as opposed to the fully supervised, current state-of-the-art classification models. Furthermore, no significant additional model optimization via excessive hyper-parameter tuning was attempted here, which may or may not yield additional improvements in task accuracy on all datasets considered. 

To keep the ablations simple and the computational overhead of model estimation reasonable, we did not carefully tune regularization factors but fixed these at one (i.e., equal weighting for all losses). For such equal weighting to work properly, the magnitudes of the different loss components need to be on the same, or similar, scale. We have provided in the Author Rebuttal associated PDF examples of the different loss contributions highlighting their similar scale ensuring the validity of using equal weighting for the losses. Future work should prioritize further hyperparameter optimization. 

# References 

[i] Dose et al., Expert Syst. Appl., 2018, doi: 10.1016/j.eswa.2018.08.031. 

[ii] Eldele et al., IEEE Trans. Neural Syst. Rehabil. Eng., 2021, doi: 10.1109/TNSRE.2021.3076234. 

[iii] Goldberger et al., Circulation, 2000, doi: 10.1161/01.CIR.101.23.e215. 

[iv] Kappenman, et al., NeuroImage, 2021, doi: 10.1016/j.neuroimage.2020.117465. 

[v] Kemp, et al., IEEE Trans. Biomed. Eng., 2000, doi: 10.1109/10.867928. 

[vi] Mousavi, et al., PLOS ONE, 2019, doi: 10.1371/journal.pone.0216456. 

[vii] Phan, et al., IEEE Trans. Neural Syst. Rehabil. Eng., 2019, doi: 10.1109/TNSRE.2019.2896659. 

[viii] Phan, et al., IEEE Trans. Pattern Anal. Mach. Intell., 2022, doi: 10.1109/TPAMI.2021.3070057. 

[ix] Phan, et al., IEEE Trans. Biomed. Eng., 2022, doi: 10.1109/TBME.2022.3147187. 

[x] Schalk, et al., IEEE Trans. Biomed. Eng., 2004, doi: 10.1109/TBME.2004.827072. 

[xi] Wang, et al., IEEE MeMeA, 2020. doi: 10.1109/MeMeA49120.2020.9137134. 

[xii] Xie et al., IEEE Trans. Neural Syst. Rehabil. Eng., 2022, doi: 10.1109/TNSRE.2022.3194600. 

[xiii] Zhu, et al. Int. J. Environ. Res. Public Health, 2020, doi: 10.3390/ijerph17114152. 

[4] Bollens et al., ICASSP, 2022. doi: 10.1109/ICASSP43922.2022.9747297.  

[37] Özdenizci et al., IEEE Access, 2020. doi: 10.1109/ACCESS.2020.2971600. 

[40] Qian et al., ICML, 2019. arXiv:1905.05879 [eess.AS]. 

[42] Rayatdoost et al., ICASSP, 2021. doi: 10.1109/ICASSP39728.2021.9414496.  

Meta-review: All reviewers have found merit in this work and judged it clear, meaningful and even insightful. Constructive interactions happened between reviewers and authors during discussion period. This work should be endorsed for publication at NeurIPS 2023.
Decision: Accept (poster)