SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Anders Vestergaard Nørskov Alexander Neergaard Zahid Morten Mørup  
Department of Applied Mathematics and Computer Science Technical University of Denmark andersxa@gmail.com {aneol, mmor}@dtu.dk https://github.com/andersxa/CSLP-AE
Electroencephalography (EEG) is a prominent non-invasive neuroimaging technique providing insights into brain function. Unfortunately, EEG data exhibit a high degree of noise and variability across subjects hampering generalizable signal extraction. Therefore, a key aim in EEG analysis is to extract the underlying neural activation (content) as well as to account for the individual subject variability (style). We hypothesize that the ability to convert EEG signals between tasks and subjects requires the extraction of latent representations accounting for content and style. Inspired by recent advancements in voice conversion technologies, we propose a novel contrastive split-latent permutation autoencoder (CSLP-AE) framework that directly optimizes for EEG conversion. Importantly, the latent representations are guided using contrastive learning to promote the latent splits to explicitly represent subject (style) and task (content). We contrast CSLP-AE to conventional supervised, unsupervised (AE), and self-supervised (contrastive learning) training and find that the proposed approach provides favorable generalizable characterizations of subject and task. Importantly, the procedure also enables zero-shot conversion between unseen subjects. While the present work only considers conversion of EEG, the proposed CSLP-AE provides a general framework for signal conversion and extraction of content (task activation) and style (subject variability) components of general interest for the modeling and analysis of biological signals.
Electroencephalography (EEG) is a non-invasive method for recording brain activity, commonly used in neuroscience research to analyze event-related potentials (ERPs) and gain insights into cognitive processes and brain function [28]. However, EEG signals are often noisy, contain artifacts, and exhibit high sensitivity to subject variability [13, 43], making it challenging to analyze and interpret the data. In particular, the inherent subject variability is well known to confound recovery of the task content of the signal, and a study by Gibson et al. [17] demonstrated that across-subject variation in EEG variability and signal strength was more significant than across-task variation. A major challenge modeling EEG data is thus to remove the intrinsic subject variability in order to recover generalizable patterns of the underlying neural patterns of activation.
Supervised methods explicitly classifying tasks can potentially filter subject variability and recover generalizable patterns of neural activity. However, explicitly disentangling subject and task content in EEG signals is valuable not only for classifying task content but also for characterizing the subject variability which ultimately can provide biomarkers of individual variability. As such, rather than
focusing only on the experimental effects and considering inter-subject variability as noise it should be treated as an important signal enabling the understanding of individual differences [24, 55].
Deep learning-based feature learning has become prevalent in EEG data analysis. As such, autoencoders have shown promise in learning transferable and feature-rich representations [36, 60, 70] and have been applied to various downstream tasks, including brain-computer interfacing (BCI) [35, 44, 78], clinical epilepsy and dementia detection [19, 38, 77], sleep stage classification [34, 63], emotion recognition [26, 61], affective state detection [53, 74] and monitoring mental workload/fatigue [75, 76] with promising results. For a systematic review on deep learning-based EEG analysis, see e.g. Roy et al. [52].
The task of disentangling content (signal) from style (individual variability) is a well-known aim in voice conversion technologies. A study on speech representation learning achieved promising results in disentangling speaker and speech content using speaker-conditioned auto-encoders [9], while Chou et al. [10] proposed using instance normalization to enforce speaker and content separation in the latent space. In Qian et al. [48] zero-shot voice conversion was proposed using a simple autoencoder conditioned on a pretrained speaker embedding model and exploring bottleneck constriction to obtain content and style disentanglement with good results. Wu and Lee [71] and Wu et al. [72] disregarded the pretrained speaker embedding model by generating content latents based on a combination of instance normalization and vector quantization of an encoded signal, while the speaker latents were generated based on the difference between content codes and the encoded input signal.
Disentangling subject and task content has been shown to enhance model generalization in emotional recognition and speech processing tasks [4, 50, 57]. Bollens et al. [4] used two explicit latent spaces in a factorized hierarchical variational autoencoder (FHVAE) to model high-level and low-level features of EEG data and found that the model was able to disentangle subject and task content. They also found that high-level features were more subject-specific and low-level features were more task-specific. Rayatdoost et al. [50] and Ozdenizci et al. [42] explored adversarial training to promote latent representations that were subject invariant. Recently, self-supervised learning has gained substantial attention due to its strong performance in representation learning enabling deep learning models to efficiently extract compact representations useful for downstream tasks. This includes the use of (pre-)training on auxiliary tasks [62] as well as contrastive learning methodologies guiding the latent representations [25]. Shen et al. [57] used contrastive learning between EEG signal representations of the same task and different subjects to learn subject-invariant representations and found that the learned representations were more robust to subject variability and improved generalizability. For a survey of self-supervised learning in the context of medical imaging, see also [58].
Inspired by recent advances in voice conversion, we propose a novel contrastive split-latent permutation autoencoder (CSLP-AE) framework that directly optimizes for EEG conversion. In particular, we hypothesize that the auxiliary task of optimizing EEG signal conversion between tasks and subjects requires the learning of latent representations explicitly accounting for content and style. We further use contrastive learning to guide the latent splits to respectively represent subject (style) and task (content). The evaluation of the proposed method is conducted on a recent, standardized ERP dataset, ERP Core [28], which includes data from the same subjects across a wide range of standardized paradigms making it especially suitable for signal conversion across multiple tasks and subjects. We contrast CSLP-AE to conventional supervised, unsupervised (AE [23]), and self-supervised training (contrastive learning [7, 41, 49, 59, 69]).
The aim of this paper is to develop a modeling framework for performing generalizable (i.e., zero-shot) conversion of EEG data considering unseen subjects. The procedure should enable conversion of EEG from one subject to another as well as one task to another task. In this context, "tasks" refer to the specific ERP components present in the EEG data, such as face or car perception, word judgement of related or unrelated word pairs, perception of standard or deviant auditory stimuli, etc. [28].
The standard autoencoder (AE) model consists of an encoder, denoted as  $E_{\theta}(X): \mathcal{X} \to \mathcal{Z}$ , and a decoder, denoted as  $D_{\phi}(Z): \mathcal{Z} \to \mathcal{X}$ . The encoder maps the input data to a latent space, while the decoder reconstructs the input from the latent space. The encoder and decoder are parameterized by  $\theta$  and  $\phi$  respectively.
To enable the model to perform conversion, it needs to be conditioned on the target subject and/or task. The latent space appears to be the suitable place for conditioning, as it is a compact representation often referred to as the bottleneck of the model. However, since the latent space is shared across subject and task representations, partitioning it into specific subject and task streams is non-trivial.
To address this challenge, a split-latent space is explored, which explicitly divides the latent space into subject and task disentangled representations. This is achieved by introducing a split in the model design within the encoder. The split-latents can be obtained by encoding the input data as follows:  $E_{\theta}(\mathbf{X}) = (z^{(S)},z^{(\mathcal{T})})$ , where  $z^{(S)}$  represents the subject latent and  $z^{(\mathcal{T})}$  represents the task latent, such that  $E_{\theta}(\mathbf{X}): \mathcal{X} \to (\mathcal{S},\mathcal{T})$ . Note that the  $(S)$  and  $(\mathcal{T})$  denominations are not inherent from the model architecture but are necessary distinctions for use in the model loss functions. These split-latents can then be joined and decoded using a similar split within the decoder. By feeding the split-latents into the decoder, the input can be reconstructed as  $\hat{\mathbf{X}} = D_{\phi}(z^{(S)},z^{(\mathcal{T})})$ , such that  $D_{\phi}(z^{(S)},z^{(\mathcal{T})}): (\mathcal{S},\mathcal{T}) \to \mathcal{X}$ . The concept of split-latent space has been explored by researchers within speech [48] and in the EEG domain [4] using separate encoders for each latent space. We presently employ a shared encoder to reduce the number of parameters used. However, separate encoders - or even pre-trained encoders, specifically, on subjects - could help kick-start the training process or work as conditioning with frozen weights. We leave this to further studies.
While voice conversion [48, 68] and other voice synthesis problems [40, 56] based on autoencoders generally use models with expansive receptive fields, e.g. as in WaveNet [40], other studies have found similar performance in voice conversion using simpler architectures, such as CNNs [10, 27, 33]. Voice conversion is usually done over a large number of samples with exceptionally high sampling rate compared to EEG data. ERP Core uses a sampling frequency of  $256\mathrm{Hz}$  (downsampled from  $1024\mathrm{Hz}$ ) over an epoch window of 1 second, which only yields a time resolution of 256 samples. Therefore, a large receptive field is unnecessary, and stripped convolution to reduce time-resolution is used instead. The proposed EEG auto-encoder model with split latent space is illustrated in Figure 1.
Figure 1: The proposed EEG auto-encoder model with split latent space. The encoder and decoder are mirrored deep convolutional neural networks using one-dimensional convolutions. Each part of the auto-encoder consists of ConvBlocks which are made up of three convolutions with residual connections (as in He et al. [22]), rectified linear unit (ReLU) activation [15] and instance normalization [64] (similar to Chou et al. [10]). The encoder applies a ConvBlock together with a strided convolution (stride=2), to reduce the time-resolution by half, four times. The decoder is mirrored and uses transposed convolutions (fractionally stripped, stride=½) to upscale the time-resolution by a factor of two with each block. Both the encoder and decoder models use a transformer [66] with four layers on each side of the bottleneck to confer attention. Finally, on the encoder side, a split is made into subject and task latent spaces. The decoder takes these split-latents as inputs and joins them again in the bottleneck in order to reconstruct the input.  $k$  is the kernel size and  $p$  is the padding on both sides.
2.1 Split-latent permutation
To guide the autoencoder in Figure 1 in disentangling task and subject we propose the use of latent-permutation, which is a self-supervised approach that guides the latent space by ensuring consistency in the subject and/or task encodings between permutations. This can be seen as a direct loss relating to the conversion method described in this section.
To achieve zero-shot conversion the respective subject and task information need to be extracted from the input data, and with an explicit split of the latent space, the conversion becomes straightforward and practical. Depending on the desired conversion task, such as converting from subject  $U$  to subject  $V$  or from task  $M$  to task  $N$ , the corresponding split-latents are simply swapped with that of the target subject or task. This conversion method is illustrated in Figure 2a.
Given a pair of input samples  $(\pmb{X}_i^a,\pmb{X}_i^b)$ , where  $i$  is the batch index, the two pairs of split-latents are defined from  $E_{\theta}$  splitting the latent space in two parts yielding  $(z_i^{(\mathcal{S},a)},z_i^{(\mathcal{T},a)})$  and  $(z_i^{(\mathcal{S},b)},z_i^{(\mathcal{T},b)})$ . A latent permutation is performed which swaps two of the latents in a given latent space  $\mathcal{L}\in \{\mathcal{S},\mathcal{T}\}$  before reconstruction. A comprehensive glossary of symbols and abbreviations is provided in the appendix.
Consider the latent space  $\mathcal{L} = \mathcal{T}$ . The pair of input samples  $(X_{i}^{a}, X_{i}^{b})$  are sampled to both belong to the same task  $t_{i}$ . The task latents are swapped between the pairs such that the reconstructed EEG data decoded by  $D_{\phi}$  becomes
$$
E _ {\theta} \left(\boldsymbol {X} _ {i} ^ {a}\right) = \left(\boldsymbol {z} _ {i} ^ {(S, a)}, \boldsymbol {z} _ {i} ^ {(T, a)}\right), \quad \hat {\boldsymbol {X}} _ {i} ^ {(T, a)} = D _ {\phi} \left(\boldsymbol {z} _ {i} ^ {(S, a)}, \boldsymbol {z} _ {i} ^ {(T, b)}\right) \tag {1}
$$
$$
E _ {\theta} \left(\boldsymbol {X} _ {i} ^ {b}\right) = \left(\boldsymbol {z} _ {i} ^ {(\mathcal {S}, b)}, \boldsymbol {z} _ {i} ^ {(\mathcal {T}, b)}\right), \quad \hat {\boldsymbol {X}} _ {i} ^ {(\mathcal {T}, b)} = D _ {\phi} \left(\boldsymbol {z} _ {i} ^ {(\mathcal {S}, b)}, \boldsymbol {z} _ {i} ^ {(\mathcal {T}, a)}\right) \tag {2}
$$
here colorized according to their corresponding input sample  $X_{i}^{a}$  or  $X_{i}^{b}$ . This is the same-task latent-permutation since pairs belong to the same task class. A corresponding swap can be done for the subject latent space  $S$  with pairs belonging to the same subject,  $s_i$ . The task and subject latent space permutations are illustrated in Figure 2b and Figure 2c respectively.
The latent-permutation loss is defined as the sum over the pair of reconstruction losses between the two samples and their corresponding reconstructions with split-latents from latent space  $\mathcal{L}$  swapped:
$$
L _ {L P} \left(\mathcal {L}; \boldsymbol {X} _ {i} ^ {a}, \boldsymbol {X} _ {i} ^ {b}\right) = \frac {1}{N} \sum_ {i = 1} ^ {N} \left(\left\| \boldsymbol {X} _ {i} ^ {a} - \hat {\boldsymbol {X}} _ {i} ^ {(\mathcal {L}, a)} \right\| _ {2} ^ {2} + \left\| \boldsymbol {X} _ {i} ^ {b} - \hat {\boldsymbol {X}} _ {i} ^ {(\mathcal {L}, b)} \right\| _ {2} ^ {2}\right) \tag {3}
$$
Consider the scenario where  $\mathcal{L} = \mathcal{T}$  again using the reconstructions from Eq. (1). In Eq. (3) the  $L_{2}$ -norm is calculated between the input data  $\pmb{X}_i^a$  and the reconstruction of the latent-permutation  $(\pmb{z}_i^{(\mathcal{S},a)},\pmb{z}_i^{(\mathcal{T},b)})$ . According to the smoothness meta-prior proposed by Bengio et al. [2] a pair of task latents should be invariant to local perturbations in the input. Seen in the context of latent-permutation, such a local perturbation could be equivalent to two task representations of the same class  $t_i$  from different input samples. When the latent space is locally smooth (i.e. encoder is consistent) then this term approximates the autoencoder reconstruction loss, i.e. if  $\pmb{z}_i^{(\mathcal{T},a)}\approx \pmb{z}_i^{(\mathcal{T},b)}$  then  $D_{\phi}(z_i^{(\mathcal{S},a)},z_i^{(\mathcal{T},a)})\approx D_{\phi}(z_i^{(\mathcal{S},a)},z_i^{(\mathcal{T},b)})$  where  $D_{\phi}(z_i^{(\mathcal{S},a)},z_i^{(\mathcal{T},a)})$  is the standard reconstruction when both  $E_{\theta}$  and  $D_{\phi}$  are consistent. This is expanded upon in the supplementary material.
Notably, such autoencoding enables flow of structural information specific for the given input sample and its reconstruction beyond subject and task content. In the supplementary material, we explore a setup where the output sample is from a different subject and task than the input sample.
In contrast to AutoVC [48], which achieves disentanglement by adding an explicit speaker latent and reducing the capacity of the content encoder through a smaller latent dimension, the proposed split-latent permutation directly optimizes for conversion. It is not guaranteed that it will explicitly separate task and subject information in their respective spaces. When the capacities of the latents are sufficiently large, both subject and task content can potentially be encoded together, and the decoder can learn to extract the corresponding subject and task content from these identically encoded splits. To address this issue and avoid relying solely on disentanglement achieved through careful bottleneck tuning, we explore contrastive learning to suffice the smoothness criteria, disentangle and specialize each latent space. Limiting the capacity of the latent space is explored in the supplementary material.
Figure 2: (a): Split-latent conversion example. Two samples  $X^a$  and  $X^b$ , where  $X^a$  corresponds to subject  $U$  and task  $M$  and  $X^b$  corresponds to subject  $V$  and task  $N$  respectively, are encoded yielding a pair of split-latents for each sample. In this example, to convert from task  $M$  to task  $N$ , the subject  $(U)$  latent is kept while the task  $(M)$  latent is simply swapped with the corresponding latent from the other sample which encodes task  $N$ . The split-latents are then decoded to ideally obtain a sample from the data distribution given subject  $U$  and task  $N$ . This is the ideal case where the split-latents are perfectly disentangled and independent. (b) and (c): Illustration of the object class-dependant swap. The input samples are encoded yielding their corresponding split-latents  $(z^{(S,a)}, z^{(S,b)}, z^{(T,a)}, z^{(T,b)})$ . Depending on the object class (tasks in (b), subjects in (c)), the split-latents are swapped between the two pairs of latents. These are then decoded yielding the reconstructed EEG data. The same-task pairs both have blue (M) task-latents since these are intentionally sampled from the same class. The intention of the latent-permutation method is expressed when the model is consistent in its latent representation of the class, and when the split latent spaces are sufficiently disentangled in the ideal case. In such cases the swap will have negligible impact on the conversion, i.e. if they encode the same (subject or task) information.
2.2 Contrastive Learning
Contrastive learning is a self-supervised learning method which aims to learn a representation of the data by maximizing the similarity between positive pairs and minimizing it between negative pairs [7, 8, 21] achieving local smoothness around classes. We apply contrastive learning to each split-latent space. This is done by utilizing a batch construction technique to sample pairs for both subjects and tasks separately, and then applying the contrastive loss to each split-latent space according to denomination, thus specializing the given latent space for either subject or task embeddings.
Specifically, we consider the multi-class  $N$ -pair loss [59] which is a deep metric learning method [69] utilizing a special batch construction technique. The batch construction technique involves sampling two samples from each class, and then constructing  $K$  pairs of samples from the batch. This batch construction is similarly required for the latent-permutation in Section 2.1. We use the batch construction method combined with the InfoNCE generalization from Oord et al. [41] with the  $\tau$  temperature parameter as in Chen et al. [7]. The full generalization is equivalent to the CLIP-loss from Radford et al. [49] in which we minimize the symmetric cross-entropy loss of the temperature-scaled similarity matrix based on the NT-Xent loss [7]. Let  $Z^{A} \in \mathbb{R}^{C \times K}$  and  $Z^{B} \in \mathbb{R}^{C \times K}$  be latent representation matrices of the  $K$  pairs of samples, then  $L_{\mathrm{NT - Xent}}$  and  $L_{\mathrm{CLIP}}$  are defined as
$$
L _ {\mathrm {N T - X e n t}} \left(\mathcal {L}; \boldsymbol {Z} ^ {\prime}, \boldsymbol {Z} ^ {\prime \prime}, k\right) = - \log \frac {\exp \left(\sin \left(\boldsymbol {z} _ {k} ^ {\prime} , \boldsymbol {z} _ {k} ^ {\prime \prime}\right) / \tau\right)}{\sum_ {i = 1} ^ {K} \mathbb {1} _ {[ i \neq k ]} \exp \left(\sin \left(\boldsymbol {z} _ {k} ^ {\prime}, \boldsymbol {z} _ {i} ^ {\prime \prime}\right) / \tau\right)} \tag {4}
$$
$$
L _ {\mathrm {C L I P}} \left(\mathcal {L}; \boldsymbol {Z} ^ {A}, \boldsymbol {Z} ^ {B}\right) = \frac {1}{K} \sum_ {k = 1} ^ {K} \left(L _ {\mathrm {N T - X e n t}} \left(\mathcal {L}; \boldsymbol {Z} ^ {A}, \boldsymbol {Z} ^ {B}, k\right) + L _ {\mathrm {N T - X e n t}} \left(\mathcal {L}; \boldsymbol {Z} ^ {B}, \boldsymbol {Z} ^ {A}, k\right)\right) \tag {5}
$$
where  $\mathbb{1}_{[c]}\in \{0,1\}$  is the indicator function yielding 1 iff the condition  $c$  holds true.  $\mathrm{sim}(z_i',z_k'')$  is a similarity metric.  $\mathcal{L}$  denotes from which latent space the pairs have corresponding labels, e.g. for the task latent space  $\mathcal{T}$ , the  $k$ 'th pair has the same task  $t_k$  which is different from the other tasks in the same batch, i.e.  $t_1\neq t_2\neq \ldots \neq t_K$ .  $L_{\mathrm{CLIP}}(\mathcal{T};\cdot ,\cdot)$  therefore is a contrastive loss across tasks, while  $L_{\mathrm{CLIP}}(\mathcal{S};\cdot ,\cdot)$  is across subjects.
Data We consider the ERP Core dataset<sup>1</sup> from Kappenman et al. [28] providing a standardized ERP dataset containing data from 40 subjects across six different tasks based on seven widely used ERP components: N170 (Face Perception Paradigm), MMN (Mismatch Negativity, Passive Auditory Oddball Paradigm), N2pc (Simple Visual Search Paradigm), N400 (Word Pair Judgement Paradigm), P3 (Active Visual Oddball Paradigm), and LRP and ERN (Lateralized Readiness Potential and Error-related Negativity, Flankers Paradigm). We only consider data processed by Script #1 up until ICA preparation.<sup>2</sup> For more information on the data and paradigms, see Kappenman et al. [28].
Only the epoch windows around the time-locking event as described in Kappenman et al. [28] are used as epochs, therefore, for each paradigm there are two available "tasks" each with a different resulting ERP. The two classes assigned for each event per paradigm are: N170; faces/cars, MMN; deviants/standards, N2pc; contralateral/ipsilateral, N400; unrelated/related, P3; rare/frequent, and ERN and LRP; incorrect/correct. The dataset was split across subjects into a training set of  $70\%$ , an evaluation set of  $10\%$ , and a test set of  $20\%$  of the subjects respectively. The exact splits are available in the supplementary material.