SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[71] D.-Y. Wu and H.-Y. Lee. One-Shot Voice Conversion by Vector Quantization. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7734-7738, Barcelona, Spain, 2020.
[72] D.-Y. Wu, Y.-H. Chen, and H.-y. Lee. VQVC+: One-Shot Voice Conversion by Vector Quantization and U-Net Architecture. In Interspeech 2020, pages 4691–4695, Shanghai, China, 2020. ISCA.
[73] J. Xie, J. Zhang, J. Sun, Z. Ma, L. Qin, G. Li, H. Zhou, and Y. Zhan. A Transformer-based approach combining deep learning network and spatial-temporal information for raw EEG classification. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 30: 2126-2136, 2022.
[74] H. Xu and K. N. Plataniotis. Affective states classification using EEG and semi-supervised deep learning approaches. In 2016 IEEE 18th International Workshop on Multimedia Signal Processing (MMSP), pages 1-6. IEEE, 2016.
[75] Z. Yin and J. Zhang. Recognition of cognitive task load levels using single channel EEG and stacked denoising autoencoder. In 2016 35th Chinese Control Conference (CCC), pages 3907-3912. IEEE, 2016.
[76] Z. Yin and J. Zhang. Cross-session classification of mental workload levels using EEG and an adaptive deep learning model. Biomedical Signal Processing and Control, 33:30-47, 2017.
[77] Y. Yuan, G. Xun, F. Ma, Q. Suo, H. Xue, K. Jia, and A. Zhang. A novel channel-aware attention framework for multi-channel EEG seizure detection via multi-view deep learning. In 2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), pages 206-209. IEEE, 2018.
[78] X. Zhang, L. Yao, D. Zhang, X. Wang, Q. Z. Sheng, and T. Gu. Multi-person brain activity recognition via comprehensive EEG signal analysis. In Proceedings of the 14th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services, pages 28-37, 2017.
[79] T. Zhu, W. Luo, and F. Yu. Convolution-and attention-based neural network for automated sleep stage classification. International Journal of Environmental Research and Public Health, 17 (11):4152, 2020.
A.1 Structural encoding and generalization of split-latent permutation loss
We observed a tendency in the SLP-AE model trained using only the split-latent permutation loss, in which the model would simply learn identical latent spaces. We discussed how this tendency stems from the information-propagation through one of the latents during training. Split-latent permutation is trained using the self-reconstruction loss where one of the latents given as input to the decoder is swapped with that of another sample where both samples either belong to the same task or the same subject. Given that only one latent is swapped at a time, the model can learn a permutation-invariant encoding of the signal not specific to either the subject or task content of the signal, and it could rely on this information during (S.s., S.t.), (D.s., S.t.), (S.s., D.t.) conversion.
In this material we generalize the split-latent permutation using a quadruplet sampling method instead to instances where the input sample and the reconstruction target (output sample) is not the same, but which in special cases becomes identical to both the split-latent permutation and the self-reconstruction loss.
During training we sample a batch of  $K$  quadruplets,  $\{(X_k^a,X_k^b,X_k^c,X_k^d)\}_{k = 1}^K$ . For each index of the batch  $k$  we choose two random subjects,  $U_{k}$  and  $V_{k}$ , and two random tasks,  $M_{k}$  and  $N_{k}$ . The quadruplet samples are sampled such that
$$
X _ {k} ^ {a} \text {h a s s u b j e c t a n d t a k} \left(U _ {k}, M _ {k}\right) \tag {6}
$$
$$
X _ {k} ^ {b} \text {h a s s u b j e c t a n d t a k} \left(V _ {k}, M _ {k}\right) \tag {7}
$$
$$
X _ {k} ^ {c} \text {h a s s u b j e c t a n d t a k} \left(U _ {k}, N _ {k}\right) \tag {8}
$$
$$
X _ {k} ^ {d} \text {h a s s u b j e c t a n d t a k} \left(V _ {k}, N _ {k}\right) \tag {9}
$$
Similar to the split-latent permutation, the encoding of these quadruplets should match and disentangle the subject and task content into their respective latent spaces, such that a latent-swap between two latents (which ideally encode the same information) has minimal impact on the reconstruction/conversion. With these quadruplets we can now generalize the split-latent permutation such that both latents are swapped with latents from other samples which should encode the same information. The samples encode the following latents
$$
X _ {k} ^ {a} \text {e n c o d e s} \left(\boldsymbol {z} _ {k} ^ {(S, a)}, \boldsymbol {z} _ {k} ^ {(\mathcal {T}, a)}\right) \tag {10}
$$
$$
X _ {k} ^ {b} \text {e n c o d e s} \left(\boldsymbol {z} _ {k} ^ {(S, b)}, \boldsymbol {z} _ {k} ^ {(T, b)}\right) \tag {11}
$$
$$
X _ {k} ^ {c} \text {e n c o d e s} \left(\boldsymbol {z} _ {k} ^ {\left(\mathcal {S}, c\right)}, \boldsymbol {z} _ {k} ^ {\left(\mathcal {T}, c\right)}\right) \tag {12}
$$
$$
X _ {k} ^ {d} \text {e n c o d e s} \left(\boldsymbol {z} _ {k} ^ {\left(\mathcal {S}, d\right)}, \boldsymbol {z} _ {k} ^ {\left(\mathcal {T}, d\right)}\right) \tag {13}
$$
$$
\boldsymbol {z} _ {k} ^ {(S, a)} \text {a n d} \boldsymbol {z} _ {k} ^ {(S, c)} \text {b o t h i d e a l l y e n c o d e} U _ {k} \tag {14}
$$
$$
\boldsymbol {z} _ {k} ^ {(S, b)} \text {a n d} \boldsymbol {z} _ {k} ^ {(S, d)} \text {b o t h i d e a l l y e n c o d e} V _ {k} \tag {15}
$$
$$
\boldsymbol {z} _ {k} ^ {(\mathcal {T}, a)} \text {a n d} \boldsymbol {z} _ {k} ^ {(\mathcal {T}, b)} \text {b o t h i d e a l l y e n c o d e} M _ {k} \tag {16}
$$
$$
\boldsymbol {z} _ {k} ^ {(\mathcal {T}, c)} \text {a n d} \boldsymbol {z} _ {k} ^ {(\mathcal {T}, d)} \text {b o t h i d e a l l y e n c o d e} N _ {k} \tag {17}
$$
All of these pairs of latents which ideally encode the same information are swapped in the generalized split-latent permutation loss, which we will refer to as the quadruplet permutation loss (QP-loss). With this full swap of latents, there is no direct path between the input sample and the output sample, and the model is forced to encode the subject and task content into their respective latents. The reconstructions are as follows
$$
\hat {X} _ {k} ^ {a} = D _ {\phi} \left(\boldsymbol {z} _ {k} ^ {(S, c)}, \boldsymbol {z} _ {k} ^ {(\mathcal {T}, b)}\right) \text {s h o u l d r e c o n s t r u c t} X _ {k} ^ {a} \tag {18}
$$
$$
\hat {X} _ {k} ^ {b} = D _ {\phi} \left(\boldsymbol {z} _ {k} ^ {(\mathcal {S}, d)}, \boldsymbol {z} _ {k} ^ {(\mathcal {T}, a)}\right) \text {s h o u l d r e c o n s t r u c t} X _ {k} ^ {b} \tag {19}
$$
$$
\hat {X} _ {k} ^ {c} = D _ {\phi} \left(\boldsymbol {z} _ {k} ^ {(\mathcal {S}, a)}, \boldsymbol {z} _ {k} ^ {(\mathcal {T}, d)}\right) \text {s h o u l d r e c o n s t r u c t} X _ {k} ^ {c} \tag {20}
$$
$$
\hat {X} _ {k} ^ {d} = D _ {\phi} \left(\boldsymbol {z} _ {k} ^ {(\mathcal {S}, b)}, \boldsymbol {z} _ {k} ^ {(\mathcal {T}, c)}\right) \text {s h o u l d r e c o n s t r u c t} X _ {k} ^ {d} \tag {21}
$$
Figure 4: Illustration of the quadruplet permutation loss. The quadruplet permutation loss is a generalization of the split-latent permutation loss, where the latents are swapped in pairs such that there is no direct path between the input sample and the reconstruction. The quadruplet permutation loss is illustrated with the quadruplet  $(X_{k}^{a}, X_{k}^{b}, X_{k}^{c}, X_{k}^{d})$  where the latents are swapped such that  $z_{k}^{(\mathcal{S},a)}$  and  $z_{k}^{(\mathcal{S},c)}$  are swapped, and  $z_{k}^{(\mathcal{T},a)}$  and  $z_{k}^{(\mathcal{T},b)}$  are swapped, etc., before decoding. This is done for all quadruplet samples yielding the quadruplet permutation loss as the MSE loss between the input sample and the reconstruction.
The swap of latents is illustrated in Figure 4.
The quadruplet permutation loss is defined as
$$
\mathcal {L} _ {\mathrm {Q P}} = \frac {1}{4 K} \sum_ {k = 1} ^ {K} \left(\| X _ {k} ^ {a} - \hat {X} _ {k} ^ {a} \| _ {2} ^ {2} + \| X _ {k} ^ {b} - \hat {X} _ {k} ^ {b} \| _ {2} ^ {2} + \| X _ {k} ^ {c} - \hat {X} _ {k} ^ {c} \| _ {2} ^ {2} + \| X _ {k} ^ {d} - \hat {X} _ {k} ^ {d} \| _ {2} ^ {2}\right) \tag {22}
$$
The quadruplet permutation loss collapses to the split-latent permutation loss in some special cases. When input samples  $X_{k}^{a}$  and  $X_{k}^{c}$  are the same, then it becomes the same-subject permutation loss, and when input samples  $X_{k}^{a}$  and  $X_{k}^{b}$  are the same, then it becomes the same-task permutation loss. In the case where all input samples are the same, then the quadruplet permutation loss becomes the self-reconstruction loss.
Quadruplet permutation loss results
We provide here results using the same training and testing setup as in the paper. We conduct an ablation study on contrastive learning, latent-permutation and quadruplet permutation loss. The following four models are trained in five repetitions each:
- SQP-AE: Quadruplet permutation loss only.
- CSQP-AE: Quadruplet permutation loss and contrastive loss in conjunction.
- SQLP-AE: Quadruplet permutation loss and latent-permutation loss in conjunction.
- CSQLP-AE: Quadruplet permutation loss, contrastive loss and latent-permutation loss in conjunction.
Here we see that the quadruplet permutation loss degrades performance on subject classification accuracy considerably, but with similar performance on task classification accuracy. We also see that
Table 3: Single-trial balanced subject classification accuracy (S.acc%), task-on-subject classification accuracy (T-S.acc%), task classification accuracy (T.acc%), subject-on-task classification accuracy (S-T.acc%), and zero-shot same-subject same-task ERP conversion MSE (S.s., S.t.), different-subject different-task ERP conversion MSE (D.s, D.t.), different-subject same-task ERP conversion MSE (D.s., S.t.), same-subject different-task ERP conversion MSE (S.s., D.t.). All ERP conversion MSE values have scales of  $10^{-11}\mathrm{V}^2$ . Epoch window was 1s.
Model	S.acc%	T-I-S.acc%	T.acc%	S-I-T.acc%	(S.s., S.t.)	(D.s, D.t.)	(D.s., S.t.)	(S.s., D.t.)
CSQLP-AE	76.10 ± 0.76	43.36 ± 0.46	46.17 ± 0.25	76.47 ± 0.46	1.91 ± 0.08	6.90 ± 0.05	3.43 ± 0.05	3.94 ± 0.16
SQLP-AE	69.26 ± 0.50	46.86 ± 1.35	44.80 ± 0.77	69.60 ± 0.25	1.48 ± 0.05	6.44 ± 0.03	2.87 ± 0.10	2.97 ± 0.05
CSQP-AE	73.04 ± 0.50	35.89 ± 0.42	44.83 ± 0.21	71.56 ± 0.64	6.20 ± 0.12	7.00 ± 0.08	6.60 ± 0.11	6.59 ± 0.10
SQP-AE	73.44 ± 0.33	43.92 ± 0.29	48.88 ± 0.13	70.42 ± 0.39	5.58 ± 0.07	6.49 ± 0.04	6.07 ± 0.04	5.99 ± 0.06
CSLP-AE	80.32 ± 0.28	45.41 ± 0.37	48.48 ± 0.34	79.64 ± 0.37	4.21 ± 0.12	20.06 ± 0.10	5.80 ± 0.15	6.65 ± 0.23
SLP-AE	74.63 ± 0.74	47.23 ± 0.31	47.00 ± 0.32	74.70 ± 0.73	3.82 ± 0.04	19.92 ± 0.10	6.12 ± 0.09	5.02 ± 0.08
C-AE	79.42 ± 0.48	37.34 ± 0.45	46.59 ± 0.23	73.27 ± 0.25	4.28 ± 0.06	20.28 ± 0.07	11.33 ± 0.47	10.64 ± 0.30
the (D.s., D.t.) conversion loss now is on the same level as (D.s., S.t.) and (S.s., D.t.) conversion for the models without latent permutation. Adding the latent-permutation, although it introduces the structural encoding pathway to the model again, considerably decreases both (S.s., S.t.) conversion compared to CSLP-AE and SLP-AE.
We provide  $t$ -SNE [1, 31, 65] plots of the generalized models here and the CSLP-AE model in Figure 5a. Furthermore, we provide (D.s., D.t.) conversion examples in Figure 5b for the generalized models and the CSLP-AE model.
Model: CSLP-AE ↓ S. space colored by S. →
Model: SQP-AE
Model: CSQP-AE
Model:SQLP-AE
Model: CSQLP-AE ↓
Color by S.
T. space colored by T.
Color by T.
(D.s., D.t.) conversion Target Subject: 23, Target Paradigm: ERN, Channel: FCz
Figure 5: (a)  $t$ -SNE plots of split-latents as encoded on the test set (unseen subjects), colored by true labels. Rows show subject and task latent spaces, respectively, while columns indicate models CSLP-AE, SQP-AE, CSQP-AE, SQLP-AE, and CSQLP-AE respectively. (b) (D.s., D.t.) converted ERPs from the same five models for a random target subject and target paradigm. All latents used in conversion were from unseen subjects on the test set, i.e. unseen to unseen conversion.
The SQP-AE model achieves specialized latent spaces with disentangled subject and task content as evident from the  $t$ -SNE plots in Figure 5a. Notably, the latent space is similar to the CSLP-AE model, but using simply the quadruplet permutation loss. In this sense, the quadruplet permutation loss is similar to the contrastive loss in that it encourages the model to learn a disentangled latent space while also directly learning all conversion schemes. Further research could focus on this quadruplet permutation loss and its relation to the contrastive loss. We view it as a contrastive loss that relates input samples to the output sample (the reconstruction), i.e. an auto-encoder contrastive loss, whereas a standard contrastive loss operates in the latent space itself. When we add the latent-permutation loss back to the model, we see that the structural encoding property occurs again in the SQLP-AE model, while the CSQLP-AE model does not have this property due to the contrastive loss used in specializing the latent space. Therefore, the SQP-AE model might be most comparable to the CSLP-AE model from the paper, as it both optimizes for conversion and latent disentanglement.
A.2 Loss component scaling
In this section we provide loss progressions on the ERP Core dataset for 200 epochs with standard error of the mean over the different runs shown as confidence bounds.
Figure 6: Training loss progressions on ERPCore [iv] dataset. Total loss is sum of loss components. sub_contra_s and task_contra_t are contrastive losses in the subject and task latent spaces respectively. recon is the reconstruction loss. latent.permute_s and latent.permute_t are the latent permutation loss terms respectively. The pink loss progressions are from CSLP-AE, the purple progressions are from SLP and the orange/brown progressions are from C-AE.
A.3 Glossary of symbols and abbreviations
Table 4: Comprehensive glossary over symbols and abbreviations
Symbol/Abbreviation	Meaning
EEG	Electroencephalography
ERP	Event-related potential
BCI	Brain-computer interfacing
CV	Cross-validation
MSE	Mean squared error
AE	Auto-encoder
CNN	Convolutional Neural Network
FHVAE	Factorized Hierarchical Variational Autoencoder
CSP	Common Spatial Pattern
CSLP-AE	Contrastive Split-Latent Permutation Autoencoder
SLP-AE	Split-Latent Permutation Autoencoder
C-AE	Contrastive Autoencoder
CL	Encoder using contrastive loss
CE	Supervised Encoder using cross-entropy
CE(t)	Supervised Encoder using cross-entropy in task space only
Eθ(·)	Encoder parameterized by θ
Dφ(·)	Decoder parameterized by φ
X	EEG Sample used as input to encoder
S	Subject latent space
T	Task latent space
L	A given latent space (one of subject or task)
z(S)	Subject latent
z(T)	Task latent
X̂	Reconstructed EEG sample from output of decoder
(Xi,a, Xi,b)	Pair of EEG samples at i'th batch index with either a common subject or task
(z(i(S,a), zi(T,a))	Subject and task latent from encoding Xi a
(z(i(S,b), zi(T,b))	Subject and task latent from encoding X i b
X̂i(T,a)	Reconstructed EEG sample of Xi a where task latents are swapped in latent space T
X̂i(T,b)	Reconstructed EEG sample of Xi b where task latents are swapped in latent space T
LLP(L; Xi a, Xi b)	Split-latent permutation loss which swaps latents in latent space L and where Xi a and Xi b have either common subject or task corresponding to L
LNT-Xent(L; ·, ·, k)	N-pair cross-entropy loss in latent space L for the k'th row
LCLIP(L; ·, ·)	Symmetric temperature-scaled cross-entropy loss in latent space L
(S.s., S.t.)	Same subject, same task conversion scheme
(S.s., D.t.)	Same subject, different task conversion scheme
(D.s., S.t.)	Different subject, same task conversion scheme
(D.s., D.t.)	Different subject, different task conversion scheme
σ	Target subject
γ	Target task
xk(σ,γ)	k'th reconstructed EEG from targets
xCR-ERP	Average ERP from conversion with sampled targets
xERP(σ,γ)	Average ERP from ground truth targets