Revisiting Auxiliary Latent Variables in Generative Models
Extending models with auxiliary latent variables is a well-known technique to in-crease model expressivity.Bachman & Precup; Naesseth et al.; Cremer et al.; Domke & Sheldon show that Importance Weighted Autoencoders can be viewed as extending the variational family with auxiliary latent variables.Similarly, we show that this view encompasses many of the recent developments in variational bounds.The success of enriching the variational family with auxiliary latent variables motivates applying the same techniques to the generative model.We develop a generative model analogous to the IWAE bound and empirically show that it outperforms the recently proposed Learned Accept/Reject Sampling algorithm, while being substantially easier to implement.Furthermore, we show that this generative process provides new insights on ranking Noise Contrastive Estimation and Contrastive Predictive Coding.
Monte Carlo Objectives are analyzed using auxiliary variable variational inference, yielding a new analysis of CPC and NCE as well as a new generative model.Proposes a different view on improving variational bounds with auxiliary latent variable models and explores the use of those models in the generative model.