A Bayesian Nonparametric Topic Model with Variational Auto-Encoders
Topic modeling of text documents is one of the most important tasks in representation learning.In this work, we propose iTM-VAE, which is a Bayesian nonparametric topic model with variational auto-encoders.On one hand, as a BNP topic model, iTM-VAE potentially has infinite topics and can adapt the topic number to data automatically.On the other hand, different with the other BNP topic models, the inference of iTM-VAE is modeled by neural networks, which has rich representation capacity and can be computed in a simple feed-forward manner.Two variants of iTM-VAE are also proposed in this paper, where iTM-VAE-Prod models the generative process in products-of-experts fashion for better performance and iTM-VAE-G places a prior over the concentration parameter such that the model can adapt a suitable concentration parameter to data automatically.Experimental results on 20News and Reuters RCV1-V2 datasets show that the proposed models outperform the state-of-the-arts in terms of perplexity, topic coherence and document retrieval tasks.Moreover, the ability of adjusting the concentration parameter to data is also confirmed by experiments.
A Bayesian Nonparametric Topic Model with Variational Auto-Encoders which achieves the state-of-the-arts on public benchmarks in terms of perplexity, topic coherence and retrieval tasks.This paper constructs an infinite Topic Model with Variational Auto-Encoders by combining Nalisnick & Smith's stick-breaking variational auto-encoder with latent Dirichlet allocation and several inference techniques used in Miao.