Title: Latent Compactness: A Unified Perspective on Generative Autoencoders from VAE to VQ-VAE
Primary area: generative models
Keywords: Variational Autoencoder, VQ-VAE, latent space compactness, generative models, representation learning, latent regularization

Abstract: For a long time, the generative capability of VAE has been explained through the lens of variational inference. Conventional theories treat the KL divergence constraint and reparameterization as a unified mechanism that jointly shape the latent space. In this work, we disentangle these two components through experiments, demonstrating that the KL divergence is the key factor in fostering the formation of semantic manifolds. Reparameterization plays two roles: first, it ensures that latent representations are not deterministic points but rather anisotropic Gaussian ellipsoids, promoting a more uniform distribution in the latent space; second, it enriches the set of semantically defined points during training. This latter role is crucial for enabling meaningful sampling-based generation. Finally, we propose a unified framework where both VAE and VQ-VAE emerge as special cases. The compactness enforced by the KL divergence regularizes the latent structure, and this principle also explains why VQ-VAE, despite lacking stochasticity or a continuous prior can still achieve effective generation.

TL;DR: This paper disentangles the roles of KL divergence and reparameterization in VAE for image generation, demonstrating that the KL divergence is key to establishing semantic manifolds, while reparameterization serves as a regularization mechanism.