Symbolic Autoencoding for Self-Supervised Sequence Learning

Mohammad Hossein Amani 1 Nicolas Mario Baldwin* 1 Amin Mansouri* 1 2 Martin Josifoski 1
Maxime Peyrard 3 Robert West 1

Abstract

Parallel
Data

Traditional language models(LMs) excel at nexttoken prediction in text sequences but often struggle with transduction tasks involving distinct symbolic systems, particularly when parallel data is
scarce or nonexistent. This issue is even more pronounced in domains dealing with complex, nonnatural language sequences, such as audio signals, protein structures, or biological sequences,where the strengths of LMs in natural language do
not directly translate. To address this challenge,
we introduce symbolic autoencoding (ΣAE), a
self-supervised framework designed to exploit thewealth of non-parallel data alongside limited parallel data. ΣAE integrates two generative models via a discrete bottleneck layer, optimizing theentire system end-to-end by minimizing unsupervised reconstruction loss for all data such thatthe sequence generated at the discrete bottleneckcan be read out as the transduced input sequence,and separately optimizing the two models withsupervised loss on the subset of labeled parallel data. To allow optimization of the models in
the presence of discrete symbols, we use a family of straight-through gradient estimators. We
demonstrate the effectiveness of ΣAE on four
sequence-to-sequence transduction tasks, showing that it significantly outperforms strong baselines in weakly supervised settings.

Dxz

Dz Unparallel Z Data

Dx Unparallel X Data

Mzx

Mxz

Mzx

Mxz

Mzx

Mxz

DBx

DBz

DBx

DBz

DBx

DBz

^

^

^

^

Figure 1. Illustration of the abstract flow of data in the symbolic autoencoding (ΣAE) framework, exemplified with the Rosetta Stone
problem. Two sequence-to-sequence models (Mxz and Mzx ) aretrained with both parallel data (the Rosetta Stone) through nexttoken prediction and unparallel data through connecting the models
with a discrete bottleneck layer (DBx and DBz ) to autoencode
each language using the other as its hidden representation.

language models display striking emergent abilities to perform many such mappings after exposure to massive anddiverse textual data. However, they fail when one or bothlanguage systems are scarce or nonexistent in the training
data, or when the mapping function deviates significantly
from patterns present during training (Magueresse et al.,2020; Lample & Conneau, 2019; Joshi et al., 2020).
An epitome of such language systems is the ancient Egyptian hieroglyphs, a system of writing used in ancient Egypt.
Although Egyptian hieroglyphs were abundant in ancient
Egyptian papyri, they remained a mystery until the discovery of the Rosetta Stone in 1799, which provided the key
to unlocking the secrets of the hieroglyphs (Budge, 1913).
The stone slab bears the same text inscribed in three distinct
scripts: Egyptian hieroglyphs, Demotic script, and ancient
Greek. This limited parallel dataset, a mere 27 lines of text,
together with abundant unparallel text in the three scripts
was sufficient to guide researchers in understanding the fulltranslation M between ancient Egyptian hieroglyphs (X)and ancient Greek (Z). The elegant solution to this historicchallenge leads us to pose the question: how can we trainmodels to automatically leverage the information in theunparallel data in symbolic systems X and Z to help
with learning the mapping M between them?

1. Introduction and Preliminaries
The field of artificial intelligence has undergone a remarkable transformation in recent years, propelled by the riseof powerful language models. At the heart of this successare sequence-to-sequence (seq2seq) transducers, a class ofmodels trained to infer the mapping M between two symbolic systems X and Z such that Z = M (X). Recent large*
Equal contribution 1 Department of Computer Science, EPFL,
Lausanne, Switzerland 2 Mila, Quebec AI Institute and Universityof Montreal, Montreal, Canada 3 Univ. Grenoble Alpes, CNRS,
Grenoble INP, LIG. Correspondence to: Mohammad Hossein
Amani <mh.amani1998@gmail.com>.

1

Symbolic Autoencoding for Self-Supervised Sequence Learning

In this work, we address this question by introducing symbolic autoencoding (ΣAE), a novel method that simultaneously learns the mapping Mxz from X to Z and themapping Mzx from Z to X, symmetrically, using bothparallel and unparallel data. This is achieved by orchestrating several losses that reuse Mxz and Mzx with varying
inputs and outputs. First, two supervised losses are used
to tune Mxz and Mzx on the scarce parallel data. Given
the scarcity of parallel data, these alone are insufficient tolearn the mappings. Therefore, we introduce two additional
autoencoding losses. One autoencoder reconstructs z ∈ Zafter encoding it discretely into an unknown x ∈ X via
Mzx
Mxz
the path Z −−−→ X −−−→ Z, where Mzx is the encoder
and Mxz is the decoder. The connection between the twomodels is implemented by a discrete bottleneck (DB), a differentiable mechanism that binds two sequence-to-sequencemodels into an end-to-end, fully differentiable sequenceto-sequence-to-sequence model. The discrete bottleneckserves as the differentiable glue between Mxz and Mzx byallowing gradients to pass through the discrete latent space
using straight-through estimator (Bengio et al., 2013) and ageneral family of sampling/quantization methods. Symmetrically, the second autoencoder reconstructs x after encoding
it into z. These two autoencoding losses utilize non-parallel
data, treating X as the hidden representation for the autoencoding of Z, and vice versa. The supervised losses further
ensure that the hidden symbolic systems are grounded in theparallel data and not arbitrary languages.Fig. 1 illustrates
the abstract flow of data in the ΣAE framework.

Baziotis et al. (2019) connected two encoder-decoder models via a hidden sequence layer, employing a reconstruction
loss and a language model prior loss for unsupervised text
compression. Kaiser & Bengio (2018) explored semantic
hashing (Salakhutdinov & Hinton, 2009) and the GumbelSoftmax trick (Jang et al., 2017) for generating interpretable,
discrete encodings. Similarly, Fortuin et al. (2019) investigated training with discrete bottlenecks and examined theuse of continuous paths alongside discrete ones.
Zhu et al. (2017) and He et al. (2016) enforced consistency
across translation tasks, with Zhu et al. (2017) using adversarial networks and He et al. (2016) employing reinforcement learning to update the models. Our work introducesa third approach, leveraging straight-through gradient estimators to train models end-to-end. Furthermore, numerous
studies have focused on the discretization of elements andrepresentations in neural networks (Liu et al., 2022; 2021;
Tamkin et al., 2023; Peng et al., 2018; Maddison et al., 2016).
Our proposed solution also parallels the technique of backtranslation (Sennrich et al., 2015; Çaglar Gülçehre et al.,2015; 2017), which typically involves training an intermediate system on parallel data to translate target monolingual
data into the source language, thereby generating synthetic
parallel corpora for further training (Edunov et al., 2018).
The ΣAE framework is akin to an online version of backtranslation, where the intermediate system is continuously
improving without storing synthetic sequences.

We evaluate ΣAE on four sequence-to-sequence transduction tasks, demonstrating that it significantly outperforms
the three conventional strategies in weakly supervised settings: fine-tuning a pre-trained language model using thelimited parallel data available, using in-context learning toadapt the model to the new task, or the more direct approach
which is training a model from scratch only on supervised
data. Our main contributions are the following:

2.1. Discrete Bottleneck

2. ΣAE Framework
In our setup a DB provides two essential outputs:• Probability vector s represents a discrete distribution over tokens, facilitating training with negative loglikelihood loss when labels are available. (supervised
training)• Quantized vector vq serves as input for subsequent models or layers, such as the decoder in reconstruction tasks
when labels are not available. (unsupervised training)• We introduce ΣAE as a framework for connecting two
seq2seq models via DB and training them using gradientbased optimization.
• We unify celebrated methods such as VQ-VAE and theGumbel-Softmax trick as different design choices for thequantization function in DB and compare their performances in our experiments.
• We empirically benchmark the performance of ΣAE overtraditional supervised baselines using both synthetic andreal-world data.

Thus, DB can be described as a function s, vq = DB(v),P∥V ∥where s ∈ [0, 1]∥V ∥ and i=1 si = 1 with |V | as the sizeof the vocabulary. The discrete nature of the DB implies
that the quantized vector belongs to a finite discrete domain|V |vq ∈ D, like a dictionary of embeddings D = {D[i]}i=1 .
This discrete computation within the DB introduces a pointof non-differentiability, necessitating the use of surrogate
gradients to enable gradient-based optimization.

To facilitate further research, we open-source code and data.
2.1.1. D ISCRETE BOTTLENECK IMPLEMENTATIONS
Related work. Our work intersects with several key areas
in unsupervised and weakly supervised learning throughdiscrete representations.

The DB allows us to see the celebrated methods such as
VQ-VAE and the Gumbel-Softmax reparameterization trick2

Symbolic Autoencoding for Self-Supervised Sequence Learning

2.2. Training Models with DB Head

as different implementations of the same concept. We canclassify DBs into two categories: probability-based andembedding-based.

We incorporate a DB layer into each seq2seq model: DBx
to Mzx and DBz to Mxz , enabling both separate and joint
training modes.

Embedding based DB. Here the probability vector s is afunction of the dictionary embeddings, s = S(·; D). For
instance, this scoring function S can be the softmax functionof any distance metric between the input vector v and thedictionary embeddings D[i], with the quantized vector vq
being the closest dictionary embedding to the input vectorv according to that metric:

For parallel training data (x, z) ∈ Dxz we do a supervised
training step similar to common seq2seq training. Given
input sequence x and target sequence until step t, z <t , themodel Mxz predicts a probability vector stz for the t-th
token z t and recieves a loss (similarly for predicting the x
sequence):Xstz , vzt = DBz (Mxz (x, z <t )), Lxz = −log stz [z t ]hil[i] = ∥v−D[i]∥, vq = D arg min l[i] , s[i] = S(−l[i])itwith l representing the vector of distances between v anddictionary vectors D[i]. In backpropogation, gradientsare directly passed from vq to v using the assignmentvq ← vq + v − sg(v) in the computation graph, wheresg denotes the stop-gradient operation (van den Oord et al.,2017; Bengio et al., 2013). In our experiments we focus
on the vector-quantized DB (VQ DB) similar to VQ-VAE,
with the distance metric being the Euclidean distance.

Given unlabeled data (x ∈ Dx or z ∈ Dz ), the models
generate a latent sequence of quantized vectors (vx<Tx =x{vxt }Tt=0):
stx , vxt = DBx (Mzx (z, vx<t ))These vectors are then used to reconstruct the original input:
stz , vzt = DBz (Mxz (vx<Tx , z <t ))Pusing as the reconstruction loss Lzxz = − t log stz [z t ].
Similar steps are followed for the x sequence. We call these
X Reconstruction and Z Reconstruction modes, where weuse unparallel data, Dx or Dz , to minimize reconstruction
losses Lxzx or Lzxz .
To navigate this multi-objective optimization problem, we
propose three scheduling strategies: Joint Training involves
randomly selecting a batch from Dxz , Dx , or Dz at each
iteration and training in the corresponding mode. Unsupervised Pretraining with Supervised Finetuning starts withtraining on Dx and Dz until convergence, followed by finetuning on Dxz . Conversely, Supervised Pretraining with
Unsupervised Finetuning trains on Dxz until convergence,then shifts to fine-tuning on Dx and Dz .

Probability-based DB. Here the score function does not depend on the dictionary embeddings s = S(x), and the quantized vector vq is computed by decoding/sampling from thescore vector vq = D[decode(s)]. Therefore, to concretely
implement a probability-based DB, we need to define a scorefunction S(·) and a sampling method. In this work we take
S to be a softmax function, and use maximum likelihood
decoding and categorical sampling for decoding:• Softmax DB uses maximum likelihood decoding, i.e., thequantized vector vq corresponds to the most likely token
in the dictionary:exp(v[i]),s[i] = P|V |j=1 exp(v[j])hivq = D arg max s[i]i2.3. Hidden Sequence Collapse in Seq2Seq Models• Gumbel DB uses categorical sampling for decoding:exp(v[i] + gi )s[i] = P|V |j=1 exp(v[j] + gj ),hvq = D arg max s[i]iIn symbolic autoencoding, the encoder seq2seq model autoregressively generates hidden tokens until an End-ofSequence (EOS) token or a maximum length is reached.
This process involves a discrete decision about when to halt
generation, for which the model never receives gradient
feedback. Specifically, in unsupervised training, the loss
gradient doesn’t directly inform the model that mistakenly
assigning a high likelihood to EOS has a penalty beyond
the negative log likelihood loss: it can prematurely stop theentire sequence generation. In our early autoencoder trainings we empirically observed that the models tended to rely
excessively on the first token of the hidden representation,
leading to underutilization of subsequent tokens. This led
us to develop the following solution.

iHere gi is a sample from the Gumbel distribution, i.e.
gi = − log(− log(ui )) where ui ∼ Uniform(0, 1), using
the Gumbel reparameterization trick to translate the sampling to taking the argmax of noisy probabilities (Janget al., 2017).
Crucially, during the backward pass, gradients are passed
to s as if vq was the soft average of dictionary embeddings.
P|V |This is expressed by assigning vq ← vq + i=1 s[i]D[i] −P|V |sg( i=1 s[i]D[i]) in an automatic differentiation library.

3Symbolic Autoencoding for Self-Supervised Sequence Learning
Supervised Baseline

Supervised Pretraining

Sentence Accuracy

SCAN

Joint Training
PCFG

Unsupervised Pretraining

1.0

1.0

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

T5 Finetuning

COGS

1.0

GPT3.5 ICL
CFQ

0.8
0.6

0.0

0.4

0.0
0.1

0.2

0.3

0.1

0.2

0.0

0.3

0.2
0.0
0.05

0.10

0.15

0.02

0.04

0.06

0.08

η

Figure 2. Results for Softmax Discrete Bottleneck – Z Autoregressive Sentence Accuracy per Supervision Ratio (η). At least one training
method with the ΣAE framework consistently outperforms the pretrained and in-context learning baselines except on the CFQ dataset at
8% supervision ratio.

EOS Soft-Masking – Gradient Approximation for Halting the Generation. In unsupervised training mode, hidden sequences in a batch can have varying halting points
as the generation typically continues until the maximum
length is reached or all samples have produce an EOS token. Tokens generated after the EOS are masked out using a binary mask m of size T (the number of tokens),
where each m[i] is 1 if the EOS token has not been generated and 0 otherwise. Applying the mask to the quantized vectors vq<T during the forward pass enforces a halting mechanism by setting vectors post-EOS to a padding
embedding, vq ← vq ⊙ m + D[<PAD>] ⊙ (1 − m),thereby terminating the sequence generation. The challenge arises during the backward pass, as this mask is a
non-differentiable output of the forward computation. To
address this, we propose a gradient approximation for mthat allows the model to learn the EOS effect through afeedback mechanism. To mitigate autoregressive collapse,
we pass the gradients through m to P(Ok = <EOS>) as
Qi−1
if E[m[i]] = k=1 (1 − P(Ok = <EOS>)) had been themasking matrix in the forward computation. This approximation provides direct feedback on the EOS effect by simply
assigning m ← m + E[m] − sg(E[m]). The derivation ofthis approximation is detailed in Appendix A.2.

(T5 large), where a pretrained T5 model is fine-tuned on theavailable parallel data; (2) In-context Learning (ICL) witha Large Language Model (GPT-3.5), which utilizes GPT3.5 to perform tasks based on given context without explicit
fine-tuning; and (3) Supervised Training from Scratch,where a model is trained from scratch on the available parallel data. Further details on the tasks, model architecture,and hyperparameters are provided in Section A.4
Experimental Results. To show the feasibility of symbolic autoencoding with straight-through gradients updates
we performed an unsupervised autoencoding reconstruction experiment for each dataset and DB and observed thatthe models successfully learned a compression of the input
sequences, as shown in Table 2. The results are further
detailed in Sec. A.7.1.
In the weakly supervised task, we simulated a Rosetta Stonelike scenario with a mix of parallel and unparallel data, varying the ratio of parallel data (η) to assess the framework’s
ability to balance and integrate supervised and unsupervised
losses. Notably, the unsupervised task is a special case of
the weakly supervised task where η = 0. Results for theSoftmax DB are detailed in Figure 2.
Our experiments demonstrated that the ΣAE framework canefficiently utilize small amounts of parallel data to improve
performance on larger unparallel datasets. At each supervision ratio η, one of our scheduling methods from Section
2.1 consistently outperformed the supervised baselines. As
expected, model accuracy improved with increased supervised data, narrowing the performance gap as accuracies
converged to their maxima. An exception was observedin the CFQ dataset at an 8% supervision ratio, where finetuning the T5-large model outperformed our methods. Thisis likely due to the CFQ dataset’s closer resemblance tonatural language question answering tasks, benefiting theT5 model, which is pretrained on similar tasks. Additional
remarks on training dynamics and learning behavior areprovided in Section A.6. A detailed analysis of the results
and the full set of performance metrics, including other DBs,
are presented in Section A.7.2.

3. Experiments
Datasets. For our experiments, we utilized four seq2seq
datasets: SCAN (Lake & Baroni, 2017), PCFG SET (Hupkes et al., 2019), CFQ (Keysers et al., 2019), and COGS
(Kim & Linzen, 2020), chosen for their compositional complexity, controlled environments, and precise accuracy measures. We evaluated the framework on the aforementioned
datasets, focusing on sentence accuracy (SA) and token accuracy (TA). Additional performance metrics are discussed
in the appendix in Section A.5. More details on the datasets
are provided in Section A.3.
Baselines. In our experiments, we compare the performance of the ΣAE framework against the following baselines: (1) Supervised Fine-tuning of a Pretrained Model
4Symbolic Autoencoding for Self-Supervised Sequence Learning

References

Kim, N. and Linzen, T. Cogs: A compositional generalization challenge based on semantic interpretation. ArXiv,
abs/2010.05465, 2020.

Bastings, J., Baroni, M., Weston, J., Cho, K., and Kiela,
D. Jump to better conclusions: Scan both left and right.
ArXiv, abs/1809.04640, 2018.

Lake, B. M. and Baroni, M. Generalization without systematicity: On the compositional skills of sequence-tosequence recurrent networks. In International Conference
on Machine Learning, 2017.

Baziotis, C., Androutsopoulos, I., Konstas, I., and Potamianos, A. Seqˆ3: Differentiable sequence-to-sequence-tosequence autoencoder for unsupervised abstractive sentence compression. In North American Chapter of the
Association for Computational Linguistics, 2019.

Lample, G. and Conneau, A. Cross-lingual language model
pretraining. ArXiv, abs/1901.07291, 2019.

Bengio, Y., Léonard, N., and Courville, A. C. Estimatingor propagating gradients through stochastic neurons forconditional computation. ArXiv, abs/1308.3432, 2013.

Liu, D., Lamb, A., Kawaguchi, K., Goyal, A., Sun, C.,
Mozer, M. C., and Bengio, Y. Discrete-valued neuralcommunication. In Neural Information Processing Systems, 2021.

Budge, S. E. A. W. The Rosetta stone. London: British
Museum, 1913.
Edunov, S., Ott, M., Auli, M., and Grangier, D. Understanding back-translation at scale, 2018.

Liu, D., Lamb, A., Ji, X., Notsawo, P. J. T., Mozer, M. C.,
Bengio, Y., and Kawaguchi, K. Adaptive discrete communication bottlenecks with dynamic vector quantization.
ArXiv, abs/2202.01334, 2022.

Fortuin, V., Hüser, M., Locatello, F., Strathmann, H., andRätsch, G. Som-vae: Interpretable discrete representation
learning on time series. In International Conference on
Learning Representations, 2019.

Maddison, C. J., Mnih, A., and Teh, Y. W. The concrete
distribution: A continuous relaxation of discrete randomvariables. ArXiv, abs/1611.00712, 2016.

He, D., Xia, Y., Qin, T., Wang, L., Yu, N., Liu, T.-Y., and
Ma, W.-Y. Dual learning for machine translation. In
Neural Information Processing Systems, 2016.

Magueresse, A., Carles, V., and Heetderks, E. Low-resource
languages: A review of past work and future challenges.
ArXiv, abs/2006.07264, 2020.

Hupkes, D., Dankers, V., Mul, M., and Bruni, E. Compositionality decomposed: How do neural networks generalise? J. Artif. Intell. Res., 67:757–795, 2019.

Peng, H., Thomson, S., and Smith, N. A. Backpropagating through structured argmax using a spigot. ArXiv,
abs/1805.04658, 2018.

Jang, E., Gu, S., and Poole, B. Categorical reparameterization with gumbel-softmax. In 5th International Conference on Learning Representations, ICLR 2017, Toulon,
France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.

Salakhutdinov, R. and Hinton, G. E. Semantic hashing. Int.
J. Approx. Reason., 50:969–978, 2009.
Sennrich, R., Haddow, B., and Birch, A. Improving neuralmachine translation models with monolingual data. ArXiv,
abs/1511.06709, 2015.

Joshi, P., Santy, S., Budhiraja, A., Bali, K., and Choudhury, M. The state and fate of linguistic diversity andinclusion in the NLP world. In Jurafsky, D., Chai, J.,
Schluter, N., and Tetreault, J. (eds.), Proceedings of
the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6282–6293, Online, July
2020. Association for Computational Linguistics. doi:
10.18653/v1/2020.acl-main.560.

Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I.,
and Salakhutdinov, R. Dropout: a simple way to preventneural networks from overfitting. J. Mach. Learn. Res.,
15:1929–1958, 2014.
Tamkin, A., Taufeeque, M., and Goodman, N. D. Codebook
features: Sparse and discrete interpretability for neuralnetworks. ArXiv, abs/2310.17230, 2023.

Kaiser, L. and Bengio, S. Discrete autoencoders for sequence models. ArXiv, abs/1801.09797, 2018.

van den Oord, A., Vinyals, O., and Kavukcuoglu, K. Neural discrete representation learning. In Guyon, I., vonLuxburg, U., Bengio, S., Wallach, H. M., Fergus, R.,
Vishwanathan, S. V. N., and Garnett, R. (eds.), Advancesin Neural Information Processing Systems 30: Annual
Conference on Neural Information Processing Systems
2017, December 4-9, 2017, Long Beach, CA, USA, pp.
6306–6315, 2017.

Keysers, D., Schärli, N., Scales, N., Buisman, H., Furrer, D., Kashubin, S., Momchev, N., Sinopalnikov, D.,
Stafiniak, L., Tihon, T., Tsarkov, D., Wang, X., van Zee,
M., and Bousquet, O. Measuring compositional generalization: A comprehensive method on realistic data. ArXiv,
abs/1912.09713, 2019.
5

Symbolic Autoencoding for Self-Supervised Sequence Learning

Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. Unpaired
image-to-image translation using cycle-consistent adversarial networks. 2017 IEEE International Conference on
Computer Vision (ICCV), pp. 2242–2251, 2017.
Çaglar Gülçehre, Firat, O., Xu, K., Cho, K., Barrault, L.,
Lin, H.-C., Bougares, F., Schwenk, H., and Bengio, Y. On
using monolingual corpora in neural machine translation.
ArXiv, abs/1503.03535, 2015.
Çaglar Gülçehre, Firat, O., Xu, K., Cho, K., and Bengio,
Y. On integrating a language model into neural machinetranslation. Comput. Speech Lang., 45:137–148, 2017.

6

Symbolic Autoencoding for Self-Supervised Sequence Learning

A. Appendix
A.1. Remarks on ΣAE framework
For all our straight-through gradient estimations, as training progresses, models become more confident in their predictions,
resulting in more polarized score distributions. This polarization helps the models identify the most likely token withincreasing certainty, making the scores sparser and improving the accuracy of gradient approximations.
While we only use symbolic autoencoding in reconstruction setups, the framework is adaptable to additional models anddata sources. For instance, one could imagine models Mzy , Myx , etc., each with their own supervised and reconstruction
losses (e.g., Lzy , Lyxz , Lxyx , etc.) to be optimized. Unlike some multi-task scenarios where individual tasks may appear
independent or unrelated, in the ΣAE framework, improvement in one task can directly benefit others, creating a synergythat enhances overall performance.
A.2. EOS Gradient Approximation
The EOS collapse phenomena can be explained by the model’s lack of understanding of the EOS token’s impact. Without
explicit feedback, the model does not learn the importance of distributing information across the entire sequence. Instead, it
packs all information into the first token to ensure it reaches the decoder robustly. This behavior is akin to how models learn
more robust representations under dropout conditions (Srivastava et al., 2014), where information is concentrated into fewer
units.
In the ΣAE framework, we inform the model of the halting effect of the EOS token by approximating a gradient for themask m, which masks the tokens appearing after the first EOS token. This approximation is crucial for the model to learn
the halting effect of the EOS token, essential for generating accurate sequences.
The m is 1 if the EOS token has not been generated and 0 otherwise:(
1 if m[i − 1] = 1 and Oi−1 ̸= <EOS>
m[i] =
0 otherwise(1)

Where O represents the output sequence generated by the model, whether in X or Z.
Hence, the binary random vector m is defined as:
P (m[i]

=

1)

=


1 − P(O(i−1) = <EOS>) P(m[i − 1]

=

1)

=

i−1
Y

(1 − P(Ok = <EOS>)) . (2)k=1Therefore the expected value of m is:
E[m[i]] =i−1
Y(1 − P(Ok = <EOS>))

(3)

k=1

Our ablation studies showed that unsupervised training often failed due to hidden state collapse when this approximation
was not used. Without this gradient approximation, the model struggled to learn effectively, highlighting the importance of
this technique for successful training.
As training progresses, models become more confident in correctly predicting the EOS token, leading to more polarizedprobabilities. This makes the expected mask E[m] a better approximate the true mask, thereby improving the accuracy of
our approximation.
A.3. Dataset Description and Examples
We evaluated the ΣAE framework on four diverse datasets: SCAN, PCFG SET, CFQ, and COGS.
• SCAN (Lake & Baroni, 2017) is a simple language-driven navigation instruction task designed to evaluate the abilityof neural models to learn compositional commands.
7Symbolic Autoencoding for Self-Supervised Sequence Learning

Table 1. Example of Samples from Different Datasets
DatasetSCAN

PCFG SET

CFQ

COGS

Sample
X: look right thrice after run left
Z: I TURN LEFT I RUN I TURN RIGHT I LOOKI TURN RIGHT I LOOK I TURN RIGHT I LOOK
X: echo append append E18 C13 ,L18 M17 , R1 L1 Y1 T18 J18
Z: E18 C13 L18 M17 R1 L1 Y1 T18 J18 J18
X: Who influenced M1 ’s cinematographer , writer , and editor
Z: SELECT DISTINCT ?x0 WHERE?x0 a ns:people.person.
?x0 ns:influence.influence node.influenced ?x1.
?x1 ns:film.cinematographer.film M1.
?x1 ns:film.editor.film M1.
?x1 ns:film.writer.film M1.
X: Olivia rolled Liam.
Z: roll . agent ( x 1 , Olivia ) AND roll . theme ( x 1 , Liam )Train set size

Parallel portion

13382

1% to 8%65734

4% to 32%76594

2% to 16%24155

1% to 8%

• PCFG SET (Hupkes et al., 2019) is a synthetic dataset generated using probabilistic context-free grammars, aimed at
testing the systematic generalization of models.
• CFQ (Keysers et al., 2019) is a large-scale dataset of complex natural language questions and their corresponding
SPARQL query against the Freebase knowledge base designed to measure the compositional generalization capabilities
of semantic parsing models, with questions constructed to reflect the compositional structure of Freebase.
• COGS (Kim & Linzen, 2020): COGS is a dataset for evaluating the generalization of semantic parsing models to novel
linguistic structures, emphasizing the model’s ability to generalize from given sentences to new sentences that have
similar syntactic structures but different lexical items or phrasal constructions.
These datasets were chosen for their controlled environments and precise accuracy measures, making them ideal forevaluating the framework’s performance. Examples of samples from each dataset are provided in Table 1.
The selection of these datasets ensures a comprehensive and nuanced evaluation of the ΣAE framework. They facilitate
direct evaluation of our approach, avoiding reliance on proxy metrics often used with larger datasets. Here, the mappingfrom X to Z is unique and non-reversible, with Z typically being the longer sequence, serving as a reliable ground truth
for X. Our study diverges from the typical use of these datasets for compositional generalization. Instead of focusing on
out-of-distribution testing, we emphasize in-distribution performance assessment. We also conduct a bidirectional evaluationof both Mxz and Mzx models, reflecting realistic seq2seq model applications where translation in both directions holds
equal significance, in line with the suggestions of (Bastings et al., 2018).
A.4. Details on Tasks, Model Architecture, and Hyperparameters
We conducted two sets of experiments on each dataset:• Unsupervised Training: In this scenario, we only have access to unparallel data. The primary goal is to reconstruct Z
from a hidden discrete sequence. The framework matches the dictionary size and the maximum sequence length of thehidden representation to those of X. This setup evaluates the ΣAE framework’s ability to compress the input sequence
into a shorter sequence and accurately reconstruct it.
• Weakly-supervised Training: This scenario simulates the Rosetta Stone problem, where a small portion of the datais parallel, and the rest is unparallel. The objective is to leverage both parallel and unparallel data by minimizing
unsupervised losses (Lzxz and Lxzx ) and supervised losses (Lzx and Lxz ). We conduct experiments for each dataset|Dxz |and DB implementation, varying the supervision ratio η = |Dxz |+|D. This allows us to assess how effectively
x |+|Dz |the framework uses limited parallel data to improve performance on larger unparallel datasets.
In our experiments with the ΣAE framework, we adopted a standardized model architecture and hyperparameter setting
across all tasks to maintain consistency and focus on the framework’s effectiveness. We utilized a six-layer transformer
encoder–decoder model for Mxz and Mzx , with 8 attention heads and a hidden size of 512. The model was trained using theAdam optimizer with learning rate reduction on loss plateau. We used greedy decoding consistently for all tasks, simplifying
the decoding process and ensuring uniformity across experiments.
8Symbolic Autoencoding for Self-Supervised Sequence Learning

Model learning rates were manually chosen on the order of 10−3 or 10−4 , to ensure a decrease in loss during the earlystages of training. Hyperparameters were not extensively tuned. For each task, the same hyperparameters were used acrossdifferent supervision ratios which are available in our configuration files in the code. This uniform approach underscores
the framework’s robustness, although we acknowledge that more nuanced tuning and regularization might yield higher
performance.
In both unsupervised and supervised finetuning after pretraining approaches, a gradual curriculum shift is employed rather
than an abrupt change. This involves slowly altering the probability distribution of the ‘three-sided coin’ used for batch
selection in joint training, to transition smoothly from the initial training phase to the subsequent finetuning phase.
A.5. Evaluation Metrics
In assessing the performance of the ΣAE framework, we measured two distinct metrics: sentence accuracy (SA) andtoken accuracy (TA). These metrics are designed to provide both a holistic and a detailed view of the model’s capabilities.
Sentence accuracy (SA) for a sample is counted as 1 if the entire sentence is correctly generated. Token accuracy (TA) is amore granular measure, where correctness of each predicted token in all sentences are measured separately. This metricallows for partial credit within a sentence, providing a finer understanding of the model’s performance at the token level.
The token accuracy can be measured with two methods: We can teacher-force the correct previous tokens (as per the groundtruth) to the model and measure its accuracy in predicting the next token. Alternatively, the model’s previous outputs (which
may or may not be correct) can be used as inputs for generating subsequent tokens. This autoregressive approach is generally
more challenging than teacher-forcing.
Each X has a unique corresponding Z, simplifying the assessment of accuracy in this direction, therefore, evaluating Mxz
performance is simply done by examining the Autoregressive Z TA/SA, directly measuring the model’s capability to generate
accurate Z sequences. For a given Z, however, there could be multiple valid X sequences. Therefore, to evaluate Mzx , weutilize the Teacher-forced X TA, which restricts the range of correct X sequences for end tokens. Another approach is theReconstruction Z TA/SA, where a model Mxz maps a generated sequence x̂ back to Z, and the accuracy of this reconstructedsequence serves as a proxy for the correctness of x̂.
A.6. Remarks on Experimental Results
We note that the VQ DB faced a peculiar issue of numerical instability on the SCAN dataset after extended training periods(+500 epochs). This instability was addressed through weight clipping, suggesting that while ΣAE offers substantial benefits,
optimizing stability and accuracy across different data representations and tasks may require tailored adjustments. These
insights into performance variations across X and Z spaces not only highlight the framework’s broad applicability but also
pinpoint areas for future refinement to maximize the ΣAE framework’s effectiveness.
A.7. Experiment Results
A.7.1. U NSUPERVISED T RAINING R ESULTS
In the unsupervised task, we trained the discrete autoencoder to compress and reconstruct Z sequences without any
supervised signal, evaluating the learnability of the discrete bottleneck using straight-through gradients. The results,
summarized in Table 2, show that the Softmax DB achieved over 98% token accuracy on the SCAN, CFQ, and COGS
datasets. Both the Gumbel and VQ DBs demonstrated similar effectiveness, indicating robustness in discrete autoencodingwith straight-through gradients for sequence learning tasks. An exception to the high performance was the PCFG SET
reconstruction task, where model performances were notably lower. This variation may be attributed to the unique symbolic
nature of variables within the PCFG SET task, where basic tokenization assigns distinct representations to symbolicallyequivalent variables, leading to observed performance discrepancies.
A.7.2. W EAKLY S UPERVISED T RAINING R ESULTS
In the Z space, the Softmax DB consistently surpassed supervised baselines, significantly enhancing token and sentence
accuracy across all datasets. For instance, with only 8% supervision on the PCFG SET dataset, token accuracy improved
from below 15% to above 80%. While the Gumbel DB generally showed noisier training and slightly weaker performance,
it still outperformed supervised baselines in most scenarios, except for a minor shortfall in the COGS dataset at a 16%

9Symbolic Autoencoding for Self-Supervised Sequence Learning

Table 2. Table of Test Autoregressive token accuracy (Z) (top) and Sentence Accuracy (Z) (bottom) on the unsupervised autoencoding
task (Z reconstruction). A high accuracy is achieved across all datasets, showing the feasibility of learning discrete representations withgradient-based methods.
SCAN

PCFG

COGS

CFQ

Softmax DB

1.00
0.96

0.74
0.31

0.98
0.55

0.99
0.69

Gumbel DB

0.98
0.74

0.75
0.36

0.98
0.51

0.99
0.43

VQ DB

1.00
0.93

0.44
0.00

0.94
0.03

0.90
0.00

supervision ratio. The VQ DB, despite showing a slight weaker performance in supervised baselines, improved the training
similar to the Softmax and Gumbel DBs, achieving over 20% token accuracy on CFQ dataset at 2% supervision ratio.
While no single Discrete Bottleneck or scheduling method universally outperforms others across all datasets and supervision
ratios, for every dataset and η value, at least one of our scheduling methods consistently surpasses the baseline performance.
In other words, training within the ΣAE paradigm always enhances performance, though the optimal choice of the schedulingstrategy depends on the task.
The ΣAE framework’s impact extends into the X space, where the Softmax, Gumbel, and VQ DBs exhibit performance
boosts. Notably, the exception to this trend occurs with teacher-forced token accuracy in the SCAN dataset for the Softmax
DB, indicating a unique challenge in this specific setting.
For all our experiments, we computed 95% confidence intervals via bootstrapped resampling of the test set, however they
are too small to be visible on the plots. This performance analysis underscores the ΣAE framework’s versatility and its
capacity to leverage both unsupervised and weakly supervised data to enhance model training and performance acrossdiverse seq2seq tasks.
We only measure the ICL and supervised finetuning of T5 baselines for Autoregressive Z TA and SA, as the teacher-forced
X TA is not applicable to these baselines. The ICL baseline is a flat line with a fixed number of in-context samples (20) andthe supervised finetuning of T5 is a single point at 100% supervision ratio.
We present the results of our experiments in the following tables. For Softmax discrete bottleneck, we present the results inthe following tables:• Table 3 Shows the performance of the Softmax DB on Autoregressive Z token accuracy, from test inputs• Table 4 Shows the performance of the Softmax DB on Autoregressive Z sentence accuracy, from test inputs• Table 5 shows the performance of the Softmax DB on the Autoregressive Z reconstruction token accuracy, after
mapping to a hidden X• Table 6 shows the performance of the Softmax DB on the Autoregressive Z reconstruction sentence accuracy, after
mapping to a hidden X• Table 7 shows the performance of the Softmax DB on the X token accuracy when teacher-forcing the previous inputs
For Gumbel discrete bottleneck, we present the results in the following tables:• Table 8 Shows the performance of the Gumbel DB on Autoregressive Z token accuracy, from test inputs• Table 9 Shows the performance of the Gumbel DB on Autoregressive Z sentence accuracy, from test inputs• Table 10 shows the performance of the Gumbel DB on the Autoregressive Z reconstruction token accuracy, after
mapping to a hidden X• Table 11 shows the performance of the Gumbel DB on the Autoregressive Z reconstruction sentence accuracy, after
mapping to a hidden X• Table 12 shows the performance of the Gumbel DB on the X token accuracy when teacher-forcing the previous inputs
For VQ discrete bottleneck, we present the results in the following tables:• Table 13 Shows the performance of the VQ DB on Autoregressive Z token accuracy, from test inputs• Table 14 Shows the performance of the VQ DB on Autoregressive Z sentence accuracy, from test inputs
10Symbolic Autoencoding for Self-Supervised Sequence Learning

Table 3. Softmax DB – Autoregressive Z Token Accuracy. ∗ These baselines are not concerned with the discretizer type and are nottrained with our proposed discrete bottleneck. They will appear in all tables for comparison.

SCANη = 0.01

η = 0.02

η = 0.04

η = 0.08

η = 0.99

ICL (GPT3.5)∗
T5 Finetuning∗
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.0
0.78
0.76
0.84
0.79

–
0.47
0.92
0.89
0.96
0.91

–
0.69
0.98
0.98
0.99
0.97

–
0.91
1.00
0.99
1.00
0.99

0.54
–
1.00
—
—
—

PCFG

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.50
0.17
0.56
0.47
0.58

–
0.74
0.30
0.77
0.73
0.82

–
0.85
0.78
0.94
0.91
0.87

–
0.93
0.93
0.91
0.95
0.91

0.17
–
0.97
—
—
—

COGS

η = 0.02

η = 0.04

η = 0.08

η = 0.16

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.35
0.87
0.94
0.94
0.95

–
0.72
0.94
0.97
0.93
0.98

–
0.95
0.98
0.99
0.98
0.99

–
0.99
0.99
1.00
1.00
1.00

0.25
–
1.00
—
—
—

CFQ

η = 0.01

η = 0.02

η = 0.04

η = 0.08

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.45
0.62
0.69
0.73
0.71

–
0.63
0.70
0.75
0.80
0.79

–
0.86
0.78
0.84
0.84
0.82

–
0.96
0.82
0.88
0.88
0.85

0.26
–
0.86
—
—
—

• Table 15 shows the performance of the VQ DB on the Autoregressive Z reconstruction token accuracy, after mapping
to a hidden X• Table 16 shows the performance of the VQ DB on the Autoregressive Z reconstruction sentence accuracy, after mapping
to a hidden X• Table 17 shows the performance of the VQ DB on the X token accuracy when teacher-forcing the previous inputs

Acknowledgments
Robert West’s lab is partly supported by grants from Swiss National Science Foundation (200021 185043, TMSGI2 211379),
H2020 (952215), Microsoft Swiss Joint Research Center, and by generous gifts from Facebook, Google, and Microsoft.
Amin Mansouri was supported in part by the support from Canada CIFAR AI Chair Program, from the Canada Excellence
Research Chairs Program, and Microsoft. This research was enabled in part by compute resources provided by Mila, Quebec
AI Institute (mila.quebec).

11Symbolic Autoencoding for Self-Supervised Sequence Learning

Table 4. Softmax DB – Autoregressive Z Sentence AccuracySCANη = 0.01

η = 0.02

η = 0.04

η = 0.08

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.0
0.18
0.24
0.29
0.25

–
0.02
0.52
0.50
0.71
0.61

–
0.08
0.87
0.87
0.91
0.85

–
0.53
0.97
0.95
0.98
0.95

0.11
–
1.00
—
—
—

PCFG

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.39
0.01
0.45
0.32
0.47

–
0.61
0.11
0.73
0.70
0.81

–
0.82
0.75
0.94
0.92
0.87

–
0.91
0.94
0.92
0.96
0.91

0.01
–
0.97
—
—
—

COGS

η = 0.02

η = 0.04

η = 0.08

η = 0.16

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.03
0.48
0.63
0.60
0.66

–
0.45
0.71
0.82
0.83
0.84

–
0.70
0.89
0.95
0.94
0.95

–
0.87
0.95
0.97
0.97
0.97

0.09
–
1.00
—
—
—

CFQ

η = 0.01

η = 0.02

η = 0.04

η = 0.08

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.01
0.25
0.43
0.49
0.47

–
0.13
0.40
0.50
0.59
0.60

–
0.55
0.53
0.65
0.66
0.64

–
0.83
0.61
0.73
0.73
0.69

0.01
–
0.69
—
—
—

12

Symbolic Autoencoding for Self-Supervised Sequence Learning

Table 5. Softmax DB – Reconstruction Z TA

SCAN

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.74
0.99
0.99
0.98

0.81
0.98
0.99
0.83

0.89
0.98
0.99
0.93

0.92
0.97
0.97
0.97

0.96
—
—
—

PCFG

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.37
0.71
0.68
0.76

0.50
0.80
0.75
0.79

0.74
0.86
0.86
0.88

0.78
0.91
0.89
0.87

0.83
—
—
—

COGS

η = 0.02

η = 0.04

η = 0.08

η = 0.16

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.92
0.98
0.98
0.99

0.95
0.99
0.97
0.99

0.97
1.00
0.99
1.00

0.98
1.00
1.00
1.00

0.99
—
—
—

CFQ

η = 0.01

η = 0.02

η = 0.04

η = 0.08

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.94
0.97
0.98
0.98

0.96
0.97
0.98
0.99

0.97
0.98
0.99
0.99

0.98
0.99
0.99
0.99

0.99
—
—
—

Table 6. Softmax DB – Reconstruction Z SA

SCAN

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.05
0.82
0.90
0.84

0.11
0.76
0.81
0.18

0.24
0.72
0.91
0.41

0.28
0.65
0.66
0.66

0.46
—
—
—

PCFG

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.01
0.21
0.15
0.19

0.09
0.29
0.22
0.26

0.20
0.44
0.33
0.44

0.28
0.63
0.47
0.41

0.35
—
—
—

COGS

η = 0.02

η = 0.04

η = 0.08

η = 0.16

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.02
0.51
0.55
0.75

0.04
0.76
0.48
0.81

0.35
0.93
0.68
0.90

0.48
0.97
0.96
0.95

0.57
—
—
—

CFQ

η = 0.01

η = 0.02

η = 0.04

η = 0.08

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.11
0.29
0.36
0.40

0.23
0.36
0.44
0.51

0.36
0.50
0.54
0.52

0.48
0.61
0.62
0.60

0.53
—
—
—

13

Symbolic Autoencoding for Self-Supervised Sequence Learning

Table 7. Softmax DB – Teacher-forced X TA

SCAN

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.66
0.57
0.39
0.50

0.77
0.66
0.58
0.45

0.84
0.78
0.70
0.69

0.88
0.84
0.82
0.81

0.88
—
—
—

PCFG

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.41
0.50
0.47
0.48

0.50
0.54
0.50
0.50

0.53
0.57
0.54
0.61

0.57
0.61
0.57
0.63

0.65
—
—
—

COGS

η = 0.02

η = 0.04

η = 0.08

η = 0.16

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.87
0.90
0.00
0.88

0.95
0.96
0.88
0.96

0.98
0.99
0.95
0.98

0.99
1.00
0.99
0.99

1.00
—
—
—

CFQ

η = 0.01

η = 0.02

η = 0.04

η = 0.08

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.74
0.77
0.73
0.71

0.79
0.80
0.80
0.78

0.82
0.83
0.83
0.82

0.85
0.85
0.85
0.84

0.88
—
—
—

14

Symbolic Autoencoding for Self-Supervised Sequence Learning

Table 8. Gumbel DB – Autoregressive Z Token AccuracySCANη = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.0
0.75
0.76
0.80
0.79

–
0.47
0.89
0.88
0.93
0.90

–
0.69
0.95
0.95
0.96
0.96

–
0.91
0.97
0.96
0.97
0.96

0.54
–
0.97
—
—
—

PCFG

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.0
0.16
0.25
0.44
0.56

–
0.47
0.25
0.66
0.62
0.64

–
0.69
0.70
0.81
0.85
0.81

–
0.91
0.82
0.86
0.88
0.87

0.54
–
0.89
—
—
—

COGS

η = 0.02

η = 0.04

η = 0.08

η = 0.16

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.0
0.74
0.76
0.78
0.75

–
0.47
0.80
0.79
0.81
0.81

–
0.69
0.84
0.82
0.84
0.83

–
0.91
0.85
0.84
0.84
0.84

0.54
–
0.86
—
—
—

CFQ

η = 0.01

η = 0.02

η = 0.04

η = 0.08

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.0
0.52
0.56
0.55
0.60

–
0.47
0.57
0.62
0.61
0.66

–
0.69
0.63
0.65
0.65
0.68

–
0.91
0.65
0.69
0.68
0.71

0.54
–
0.65
—
—
—

15

Symbolic Autoencoding for Self-Supervised Sequence Learning

Table 9. Gumbel DB – Autoregressive Z Sentence AccuracySCANη = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.0
0.13
0.22
0.23
0.21

–
0.02
0.46
0.48
0.56
0.55

–
0.08
0.79
0.78
0.80
0.81

–
0.53
0.86
0.85
0.86
0.86

0.11
–
0.89
—
—
—

PCFG

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.0
0.01
0.06
0.26
0.45

–
0.02
0.08
0.57
0.51
0.55

–
0.08
0.62
0.74
0.78
0.75

–
0.53
0.76
0.80
0.81
0.81

0.11
–
0.84
—
—
—

COGS

η = 0.02

η = 0.04

η = 0.08

η = 0.16

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.0
0.30
0.41
0.38
0.32

–
0.02
0.48
0.56
0.53
0.53

–
0.08
0.62
0.64
0.63
0.63

–
0.53
0.70
0.68
0.66
0.66

0.11
–
0.73
—
—
—

CFQ

η = 0.01

η = 0.02

η = 0.04

η = 0.08

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.0
0.14
0.21
0.21
0.29

–
0.02
0.23
0.29
0.29
0.37

–
0.08
0.30
0.34
0.35
0.41

–
0.53
0.33
0.40
0.39
0.45

0.11
–
0.34
—
—
—

16

Symbolic Autoencoding for Self-Supervised Sequence Learning

Table 10. Gumbel DB – Reconstruction Z TA

SCAN

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.74
0.96
0.97
0.81

0.78
0.94
0.98
0.88

0.86
0.96
0.97
0.90

0.90
0.95
0.97
0.94

0.94
—
—
—

PCFG

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.33
0.32
0.56
0.57

0.46
0.58
0.63
0.75

0.70
0.73
0.72
0.82

0.75
0.83
0.81
0.85

0.79
—
—
—

COGS

η = 0.02

η = 0.04

η = 0.08

η = 0.16

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.90
0.96
0.96
0.97

0.93
0.98
0.97
0.98

0.96
0.98
0.98
0.99

0.97
0.99
0.99
0.99

0.98
—
—
—

CFQ

η = 0.01

η = 0.02

η = 0.04

η = 0.08

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.92
0.95
0.94
0.98

0.94
0.96
0.95
0.98

0.95
0.96
0.96
0.98

0.96
0.97
0.97
0.98

0.97
—
—
—

Table 11. Gumbel DB – Reconstruction Z SA

SCAN

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.05
0.60
0.65
0.11

0.08
0.43
0.71
0.18

0.17
0.54
0.68
0.39

0.27
0.54
0.62
0.55

0.30
—
—
—

PCFG

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.01
0.01
0.07
0.05

0.06
0.11
0.11
0.21

0.20
0.19
0.15
0.32

0.25
0.38
0.25
0.42

0.21
—
—
—

COGS

η = 0.02

η = 0.04

η = 0.08

η = 0.16

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.02
0.29
0.24
0.30

0.02
0.48
0.42
0.46

0.20
0.59
0.56
0.59

0.27
0.61
0.60
0.63

0.34
—
—
—

CFQ

η = 0.01

η = 0.02

η = 0.04

η = 0.08

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.05
0.10
0.07
0.26

0.10
0.14
0.13
0.27

0.17
0.21
0.19
0.31

0.21
0.25
0.27
0.34

0.21
—
—
—

17

Symbolic Autoencoding for Self-Supervised Sequence Learning

Table 12. Gumbel DB – Teacher-forced X TA

SCAN

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.66
0.60
0.40
0.36

0.76
0.70
0.63
0.64

0.84
0.78
0.76
0.62

0.87
0.85
0.84
0.76

0.88
—
—
—

PCFG

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.39
0.38
0.45
0.43

0.48
0.49
0.50
0.53

0.51
0.55
0.52
0.55

0.55
0.58
0.56
0.57

0.57
—
—
—

COGS

η = 0.02

η = 0.04

η = 0.08

η = 0.16

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.84
0.88
0.86
0.85

0.93
0.95
0.93
0.94

0.97
0.98
0.97
0.98

0.98
0.99
0.98
0.99

0.99
—
—
—

CFQ

η = 0.01

η = 0.02

η = 0.04

η = 0.08

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.71
0.75
0.72
0.69

0.77
0.79
0.78
0.77

0.80
0.81
0.81
0.81

0.83
0.84
0.84
0.84

0.85
—
—
—

18

Symbolic Autoencoding for Self-Supervised Sequence Learning

Table 13. VQ DB – Autoregressive Z Token AccuracySCANη = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.0
0.73
0.67
0.84
0.75

–
0.47
0.89
0.88
0.95
0.91

–
0.69
0.98
0.95
0.99
0.97

–
0.91
1.00
0.97
1.00
0.99

0.54
–
1.00
—
—
—

PCFG

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.0
0.23
0.31
0.41
0.12

–
0.47
0.86
0.66
0.86
0.13

–
0.69
0.87
0.90
0.90
0.13

–
0.91
0.93
0.89
0.93
0.13

0.54
–
0.93
—
—
—

COGS

η = 0.02

η = 0.04

η = 0.08

η = 0.16

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.0
0.86
0.92
0.91
0.94

–
0.47
0.95
0.97
0.97
0.97

–
0.69
0.99
0.99
0.99
0.98

–
0.91
1.00
0.99
1.00
0.98

0.54
–
1.00
—
—
—

CFQ

η = 0.01

η = 0.02

η = 0.04

η = 0.08

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.0
0.49
0.71
0.69
0.44

–
0.47
0.77
0.81
0.83
0.54

–
0.69
0.91
0.91
0.94
0.52

–
0.91
0.95
0.96
0.96
0.61

0.54
–
0.84
—
—
—

19

Symbolic Autoencoding for Self-Supervised Sequence Learning

Table 14. VQ DB – Autoregressive Z Sentence AccuracySCANη = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.0
0.10
0.12
0.29
0.15

–
0.02
0.41
0.44
0.64
0.54

–
0.08
0.85
0.70
0.91
0.84

–
0.53
0.98
0.87
0.98
0.93

0.11
–
1.00
—
—
—

PCFG

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.0
0.04
0.14
0.23
0.00

–
0.02
0.86
0.59
0.86
0.00

–
0.08
0.87
0.91
0.90
0.00

–
0.53
0.94
0.89
0.94
0.00

0.11
–
0.91
—
—
—

COGS

η = 0.02

η = 0.04

η = 0.08

η = 0.16

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.0
0.47
0.51
0.59
0.51

–
0.02
0.74
0.81
0.82
0.63

–
0.08
0.92
0.93
0.94
0.62

–
0.53
0.97
0.97
0.97
0.59

0.11
–
0.84
—
—
—

CFQ

η = 0.01

η = 0.02

η = 0.04

η = 0.08

η = 0.99

ICL (GPT3.5)
T5 Finetuning
Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

–
0.0
0.01
0.69
0.41
0.01

–
0.02
0.53
0.78
0.66
0.02

–
0.08
0.79
0.92
0.85
0.07

–
0.53
0.87
0.96
0.91
0.05

0.11
–
0.25
—
—
—

20

Symbolic Autoencoding for Self-Supervised Sequence Learning

Table 15. VQ DB – Reconstruction Z TA

SCAN

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.73
0.95
0.99
0.86

0.77
0.95
0.98
0.97

0.83
0.95
0.98
0.99

0.91
0.96
0.97
0.99

0.99
—
—
—

PCFG

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.35
0.59
0.68
0.32

0.67
0.77
0.81
0.38

0.77
0.91
0.83
0.32

0.81
0.93
0.89
0.32

0.53
—
—
—

COGS

η = 0.02

η = 0.04

η = 0.08

η = 0.16

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.93
0.97
0.97
0.98

0.98
0.99
0.98
0.99

0.99
0.99
0.99
0.98

1.00
1.00
1.00
0.98

0.98
—
—
—

CFQ

η = 0.01

η = 0.02

η = 0.04

η = 0.08

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.89
0.97
0.97
0.94

0.95
0.98
0.98
0.94

0.98
0.99
0.99
0.94

0.98
0.99
0.99
0.94

0.98
—
—
—

Table 16. VQ DB – Reconstruction Z SA

SCAN

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.05
0.40
0.89
0.17

0.12
0.38
0.76
0.67

0.29
0.39
0.71
0.90

0.61
0.52
0.66
0.85

0.91
—
—
—

PCFG

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.02
0.10
0.13
0.00

0.18
0.30
0.33
0.00

0.19
0.64
0.39
0.00

0.25
0.75
0.56
0.00

0.12
—
—
—

COGS

η = 0.02

η = 0.04

η = 0.08

η = 0.16

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.32
0.41
0.35
0.44

0.68
0.68
0.69
0.57

0.78
0.86
0.88
0.52

0.95
0.93
0.96
0.53

0.39
—
—
—

CFQ

η = 0.01

η = 0.02

η = 0.04

η = 0.08

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.01
0.27
0.19
0.00

0.19
0.40
0.41
0.01

0.51
0.57
0.56
0.00

0.52
0.67
0.71
0.00

0.22
—
—
—

21

Symbolic Autoencoding for Self-Supervised Sequence Learning

Table 17. VQ DB – Teacher-forced X TA

SCAN

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.65
0.56
0.41
0.27

0.73
0.67
0.62
0.61

0.81
0.74
0.71
0.57

0.86
0.84
0.78
0.79

0.88
—
—
—

PCFG

η = 0.04

η = 0.08

η = 0.16

η = 0.32

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.32
0.43
0.48
0.33

0.52
0.54
0.53
0.34

0.55
0.59
0.55
0.33

0.58
0.65
0.58
0.33

0.62
—
—
—

COGS

η = 0.02

η = 0.04

η = 0.08

η = 0.16

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.85
0.88
0.85
0.88

0.96
0.95
0.93
0.93

0.99
0.98
0.98
0.93

0.99
0.99
0.99
0.93

0.97
—
—
—

CFQ

η = 0.01

η = 0.02

η = 0.04

η = 0.08

η = 0.99

Supervised Baseline
Joint training
Supervised Pretraining
Unsupervised Pretraining

0.60
0.74
0.68
0.49

0.77
0.78
0.78
0.55

0.83
0.83
0.84
0.53

0.86
0.86
0.86
0.53

0.88
—
—
—

22