Isolating Sources of Disentanglement in VAEs

We decompose the evidence lower bound to show the existence of a term measuring the total correlation between latent variables. We use this to motivate the β-TCVAE (Total Correlation Variational Autoencoder) algorithm, a refinement and plug-in replacement of the β-VAE for learning disentangled representations, requiring no additional hyperparameters during training. We further propose a principled classifier-free measure of disentanglement called the mutual information gap (MIG). We perform extensive quantitative and qualitative experiments, in both restricted and non-restricted settings, and show a strong relation between total correlation and disentanglement, when the model is trained using our framework. Learning disentangled representations without supervision is a difficult open problem. Disentangled variables are generally considered to contain interpretable semantic information and reflect separate factors of variation in the data. While the definition of disentanglement is open to debate, many believe a factorial representation, one with statistically independent variables, is a good starting point [1, 2, 3]. Such representations distill information into a compact form which is oftentimes semantically meaningful and useful for a variety of tasks [2, 4]. For instance, it is found that such representations are more generalizable and robust against adversarial attacks [5]. Many state-of-the-art methods for learning disentangled representations are based on re-weighting parts of an existing objective. For instance, it is claimed that mutual information between latent variables and the observed data can encourage the latents into becoming more interpretable [6]. It is also argued that encouraging independence between latent variables induces disentanglement [7]. However, there is no strong evidence linking factorial representations to disentanglement. In part, this can be attributed to weak qualitative evaluation procedures. While traversals in the latent representation can qualitatively illustrate disentanglement, quantitative measures of disentanglement are in their infancy. In this paper, we: • show a decomposition of the variational lower bound that can be used to explain the success of the β-VAE [7] in learning disentangled representations. • propose a simple method based on weighted minibatches to stochastically train with arbitrary weights on the terms of our decomposition without any additional hyperparameters. • introduce the β-TCVAE, which can be used as a plug-in replacement for the β-VAE with no extra hyperparameters. Empirical evaluations suggest that the β-TCVAE discovers more interpretable representations than existing methods, while also being fairly robust to random initialization. • propose a new information-theoretic disentanglement metric, which is classifier-free and generalizable to arbitrarily-distributed and non-scalar latent variables. While Kim & Mnih [8] have independently proposed augmenting VAEs with an equivalent total correlation penalty to the β-TCVAE, their proposed training method differs from ours and requires an auxiliary discriminator network. 32nd Conference on Neural Information Processing Systems (NIPS 2018), Montréal, Canada.