Title: The Free Transformer
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: transformer, VAE

Abstract: We propose a variational training procedure for decoder transformers to condition the token generation on random latent variables. Experiments with 1.5B and 8B parameter transformers show that, even without token-space reasoning, making the model free to learn and sample latent quantities translates into substantial performance gains on downstream tasks.

TL;DR: We make a VAE-decoder transformer