Title: A BAYESIAN NONPARAMETRIC TOPIC MODEL WITH VARIATIONAL AUTO-ENCODERS

Abstract: Topic modeling of text documents is one of the most important tasks in representation learning. In this work, we propose iTM-VAE, which is a Bayesian nonparametric (BNP) topic model with variational auto-encoders. On one hand, as a BNP topic model, iTM-VAE potentially has infinite topics and can adapt the topic number to data automatically. On the other hand, different with the other BNP topic models, the inference of iTM-VAE is modeled by neural networks, which has rich representation capacity and can be computed in a simple feed-forward manner. Two variants of iTM-VAE are also proposed in this paper, where iTM-VAE-Prod models the generative process in products-of-experts fashion for better performance and iTM-VAE-G places a prior over the concentration parameter such that the model can adapt a suitable concentration parameter to data automatically. Experimental results on 20News and Reuters RCV1-V2 datasets show that the proposed models outperform the state-of-the-arts in terms of perplexity, topic coherence and document retrieval tasks. Moreover, the ability of adjusting the concentration parameter to data is also confirmed by experiments.

1. INTRODUCTION:

Probabilistic topic models focus on discovering the abstract "topics" that occur in a collection of documents and representing a document as a weighted mixture of the discovered topics. Classical topic models, the most popular being LDA (Blei et al., 2003) , have achieved success in a range of applications, such as information retrieval (Wei & Croft, 2006) , document understanding (Blei et al., 2003) , computer vision (Rasiwasia & Vasconcelos, 2013) and bioinformatics (Rogers et al., 2005) . A major challenge of topic models is that the inference of the distribution over topics does not have a closed-form solution and must be approximated, using either MCMC sampling or variational inference. Hence, any small change to the model requires re-designing a new inference method tailored for it. Moreover, as the model grows more expressive, the inference becomes increasingly complex, which becomes the bottleneck to discover the latent semantic structures of complicated data. Hence, black-box inference methods (Ranganath et al., 2014; Mnih & Gregor, 2014; Kingma & Welling, 2014b; Rezende et al., 2014) , which require only limited knowledge from the models and can be flexibly applied to new models, is desirable for topic models. Among all the black-box inference methods, Auto-Encoding Variational Bayes (AEVB) (Kingma & Welling, 2014b; Rezende et al., 2014 ) is a promising one for topic models. AEVB contains an inference network that can map a document directly to a variational posterior without the need for further local variational updates on test data, and the Stochastic Gradient Variational Bayes (SGVB) estimator allows efficient approximate inference for a broad class of posteriors, which makes topic models more flexible. Hence, an increasing number of work has been proposed recently to combine topic models with AEVB, such as (Miao et al., 2016; Srivastava & Sutton, 2017; Card et al., 2017; Miao et al., 2017) .

Deciding the number of topics is another challenge for topic models. One option is to use model selection, which trains models with different topic numbers and selects the best on the validation set. Bayesian nonparametric (BNP) topic models, however, side-step this issue by making the number of topics adaptive to data. For example, Teh et al. (2006) proposed Hierarchical Dirichlet Process (HDP), which models each document with a Dirichlet Process (DP) and all DPs for documents in a corpus share a base distribution that is itself also from a DP. HDP extends LDA in that it can adapt the number of topics to data. Hence, HDP has potentially an infinite number of topics and allows the number to grow as more documents are observed. Unlike the black-box inference based models, traditionally, one needs to redesign the inference methods when there are some changes in the generative process of HDP (Teh et al., 2006; Wang et al., 2011; Hughes et al., 2015) .

In this work, we make progress on this problem by proposing an infinite Topic Model with Variational Auto-Encoders (iTM-VAE), which is a Bayesian nonparametric topic model with AEVB. Coupling Bayesian nonparametric techniques with deep neural networks, iTM-VAE is able to capture the uncertainty regarding to the number of topics, and the inference can be conducted in a simple feed-forward manner. More specifically, iTM-VAE uses a stick-breaking process (Sethuraman, 1994) to generate the mixture weights for a countably infinite set of topics, and use neural networks to approximate the variational posteriors. The main contributions of the paper are:

• We propose iTM-VAE, which, to our best knowledge, is the first Bayesian nonparametric topic model equipped with AEVB.

• We propose iTM-VAE-Prod whose distribution over words is a product of experts rather than a mixture of multinomials.

• We propose iTM-VAE-G, which helps the model to adjust the concentration parameter to data automatically.

• The experimental results show that iTM-VAE and its two variants outperform the state-ofthe-art models on two challenging benchmarks significantly.

2. RELATED WORK:

Topic models have been studied extensively in a variety of applications such as document modeling, information retrieval, computer vision and bioinformatics (Blei et al., 2003; Wei & Croft, 2006; Putthividhy et al., 2010; Rasiwasia & Vasconcelos, 2013; Rogers et al., 2005) . Please see (Blei, 2012) for an overview. Recently, with the impressive success of deep learning, neural topic models have been proposed and achieved encouraging performance in document modeling tasks, such as Replicated Softmax (Hinton & Salakhutdinov, 2009) , DocNADE (Larochelle & Lauly, 2012) , fDARN (Mnih & Gregor, 2014) and NVDM (Miao et al., 2016) . These models achieved competitive performance on modeling documents. However, they do not explicitly model the generative story of documents, hence are less explainable.

Several recent work has been proposed to model the generative procedure explicitly and the inference of the topic distributions is computed by deep neural networks. This makes these models interpretable, powerful and easily extendable. For example, Srivastava & Sutton (2017) proposed AVITM model, which embeds the original LDA (Blei et al., 2003) (Miao et al., 2017) , which proposed GSM, GSB, RSB and RSB-TF to model documents. The RSB-TF from (Miao et al., 2017) , which uses a heuristic indicator to guide the growth of the topic numbers, also has the ability to adapt the topic number. However, the performance of RSB-TF does not match its complexity. Instead, iTM-VAE exploits Bayesian nonparametric techniques to decide the number of topics, which is much more elegant. And the performance of iTM-VAE also outperforms RSB-TF.

3. PRELIMINARY:

In this section, we briefly describe the stick-breaking process (Sethuraman, 1994; Murphy, 2012) , Variational Auto-Encoders (Kingma & Welling, 2014b; Rezende et al., 2014) and the Kumaraswamy distribution (Kumaraswamy, 1980) , which are all essential for iTM-VAE.

3.1. THE STICK-BREAKING PROCESS:

We first describe the stick-breaking process, which is used to model the mixture weights over the countably infinite topics for the generative procedure of iTM-VAE. More specifically, the stickbreaking process generates an infinite sequence of mixture weights π = {π k } ∞ k=1 as follows:

EQUATION

This is often denoted as π ∼ GEM(α), where GEM stands for Griffiths, Engen and Mc-Closkey (Ewens, 1990) and α is the concentration parameter. One can see that

π k satisfies 0 ≤ π k ≤ 1 and ∞ k=1 π k = 1.

3.2. VARIATIONAL AUTO-ENCODERS:

Variational Auto-Encoder (VAE) is among the most successful generative models recently. Specifically, it assumes that a sample x is generated mathematically as follows:

EQUATION

where p(z) is the prior of the latent representation z and p φ (x|z) is a conditional distribution with parameters φ to generate x. The approximation to the posterior of the generative process, q ψ (z|x), is parametrized by an inference neural network with parameters ψ. According to (Kingma & Welling, 2014b; Rezende et al., 2014) , the Evidence Lower Bound (ELBO) of VAE can be written as:

EQUATION

)

where KL is the Kullback-Leibler divergence.

The parameters φ and ψ can be optimized jointly by applying the Stochastic Gradient Variational Bayes (SGVB) (Kingma & Welling, 2014b) estimator and Reparameterization Trick (RT). An essential requirement of SGVB and RT is that the latent variable z can be represented in a differentiable, non-centered parameterization (DNCP) (Kingma & Welling, 2014a) , which allows the gradients to go through the MC expectation. However, this is not always satisfied by the Beta prior of the stick-breaking process for iTM-VAE. Hence, we resort to the little known Kumaraswamy distribution (Kumaraswamy, 1980) to solve this problem.

3.3. KUMARASWAMY DISTRIBUTION:

Kumaraswamy distribution (Kumaraswamy, 1980 ) is a continuous distribution, which is mathematically defined as:

Kumaraswamy(x; a, b) = abx a-1 (1 -x a ) b-1 (4)

where x ∈ [0, 1] and a, b > 0. The inverse cumulative distribution function (CDF) can be expressed in a simple and closed-form formulation, and samples from Kumaraswamy distribution can be drawn by:

EQUATION

Kumaraswamy is similar to Beta distribution, yet more suitable to SGVB since it satisfies the DNCP requirement. Moreover, the KL divergence between Kumaraswamy and Beta can be closely approximated in closed-form. Hence, we use it to model the inference procedure of iTM-VAE.

4. THE MODEL:

In this section, we describe iTM-VAE, a Bayesian nonparametric topic model with VAE. Specifically, we first describe the generative process of iTM-VAE in Section 4.1, and then the inference process is introduced in Section 4.2. After that, two variants of the model, iTM-VAE-Prod and iTM-VAE-G, are described in Section 4.3 and Section 4.4, respectively.

4.1. THE GENERATIVE PROCEDURE:

The generative process of iTM-VAE is similar to the original VAE. The key difference is that the topic distribution (latent variable) π = {π k } ∞ k=1 is drawn from the GEM distribution to make sure of ∞ k=1 π k = 1 and 0 ≤ π k ≤ 1. Specifically, we suppose each topic θ k = σ(φ k ) is a probability distribution over vocabulary, where φ k ∈ R V is the parameter of the topic-specific word distribution, σ(•) is the softmax function and V is the vocabulary size. In iTM-VAE, there are unlimited number of topics and we denote Θ = {θ k } ∞ k=1 and Φ = {φ k } ∞ k=1 as the collections of these countably infinite topics and the corresponding parameters. The generation of a document by iTM-VAE can then be mathematically described as:

• Draw a topic distribution π ∼ GEM(α) • Then we get a distribution G(θ; π, Θ) = ∞ k=1 π k δ θ k (θ) • For each word w i in the document: 1) draw a topic θi ∼ G(θ; π, Θ); 2) w i ∼ Cat( θi )

where α is a hyper-parameter, Cat( θi ) is a categorical distribution parameterized by θi , and δ θ k (θ) is a discrete dirac function, which equals to 1 when θ = θ k and 0 otherwise. Thus, the joint probability of a document with N words w 1:N = {w i } N i=1 , the topic distribution π and the sampled topics θ1:N = { θi } N i=1 can be written as:

EQUATION

where p(π|α) = GEM(α), p(θ|π, Θ) = G(θ; π, Θ) and p(w|θ) = Cat(θ).

Similar to (Srivastava & Sutton, 2017) , we collapse the variable θ1:N and rewrite Equation 6 as:

EQUATION

where p(w i |π, Θ) = CAT( θ) and θ = ∞ k=1 π k θ k . In practice, following (Miao et al., 2017) , we factorize the parameter φ k of topic θ k as φ k = t k W where t k ∈ R H is the k-th topic factor vector, W ∈ R H×V is the word factor matrix and H ∈ R + is the factor dimension. For simplicity, we still use Φ to denote all the parameters regarding to the generative procedure of iTM-VAE. Note that, although we parameterize the generative process with Φ, iTM-VAE is still a nonparametric model, since it has potentially infinite model capacity, and can grow the number of parameters with the amount of training data.

Notably, different with traditional nonparametric Bayesian topic models, the topics in iTM-VAE are not drawn from a base distribution, but are treated as part of the parameters of the model and are optimized directly. This key difference indicates that there is no need to use an additional base distribution to generate the countably infinite candidate topics such that these topics are shared by different documents. Instead, the topic parameters of iTM-VAE are shared across all documents naturally.

4.2. THE INFERENCE PROCEDURE:

In this section, we describe the inference process of iTM-VAE, i.e. how to draw π given a document

w 1:N . To elaborate, suppose ν = [ν 1 , ν 2 , . . . , ν K-1 ] is a K -1 dimensional vector where ν k is a random variable sampled from a Kumaraswamy distribution κ(ν; a k , b k ) parameterized by a k and b k , iTM-VAE models the joint distribution q ψ (ν|w 1:N ) as: 1 [a 1 , . . . , a K-1 ; b 1 , . . . , b K-1 ] = g(w 1:N ; ψ) (8) q ψ (ν|w 1:N ) = K-1 k=1 κ(ν k ; a k , b k ) (9)

where g(w 1:N ; ψ) is a neural network with parameters ψ. Then, π = {π k } K k=1 can be drawn by: ν ∼ q ψ (ν|w 1:N ) (10)

π = (π 1 , π 2 , . . . , π K-1 , π K ) = ν 1 , ν 2 (1 -ν 1 ), . . . , ν k-1 K-2 l=1 (1 -ν l ), K-1 l=1 (1 -ν l ) (11)

In the above procedure, we truncate the infinite sequence of mixture weights π = {π k } ∞ k=1 by K elements, and ν K is always set to 1 to ensure K k=1 π k = 1. Notably, as discussed in (Blei et al., 2006) , the truncation of variational posterior does not indicate that we are using a finite dimensional prior, since we never truncate the GEM prior. Moreover, the truncation level K is not part of the generative procedure specification. Hence, iTM-VAE still has the ability to model the uncertainty of the number of topics and adapt it to data. People can manage to use truncation-free posteriors in the model, however, as observed by Nalisnick & Smyth (2017) , it does not work well. On the opposite, the truncated-fashion posterior of iTM-VAE is simple and works well in practice.

iTM-VAE can be optimized by maximizing the Evidence Lower Bound (ELBO):

EQUATION

where we replace Θ with Φ for p(w 1:N |π, Φ) to emphasize that Φ is the parameter to be optimized, and p(ν|α) is products of K -1 Beta(1, α) distributions according to Section 3.1. The details of the optimization can be found in Appendix 7.2.

4.3. ITM-VAE WITH PRODUCTS OF EXPERTS:

In Equation 7, θ is a mixture of multinomials. One drawback of this formulation is that it cannot make any predictions that are sharper than the distributions being mixed, pointed out by (Hinton & Salakhutdinov, 2009) , which may result in some topics that are of poor quality and do not match well with human judgment. One solution to this issue is to replace the mixture of multinomials with a weighted product of experts which is able to make sharper predictions than any of the constituent experts (Hinton, 2006) . We develop a products-of-experts version of iTM-VAE, which is referred as iTM-VAE-Prod. Specifically, we compute a mixed topic distribution θ = σ( ∞ k=1 π k φ k ) for each document, where π k is sampled from GEM(α), and then each word of the document is sampled from Cat( θ). The benefit of the products-of-experts is demonstrated in Section 5.

4.4. PLACING A PRIOR ON THE CONCENTRATION PARAMETER:

In the generative process, the concentration parameter α of GEM(α) can have significant impact on the growth of number of topics. The larger the α is, the more "breaks" it will create, and consequently, more topics will be used. Hence, it is generally reasonable to consider placing a prior on α so that the model can adjust the concentration parameter to data automatically.

Concretely, since the Gamma distribution is conjugate to Beta(1, α), we place a Gamma(s 1 , s 2 ) prior on α. Then the ELBO of iTM-VAE can be written as:

EQUATION

where p(α|s 1 , s 2 ) = Gamma(s 1 , s 2 ), p(v k |α) = Beta(1, α), q(α|γ 1 , γ 2 ) is the variational posterior for α with γ 1 , γ 2 as parameters across the whole dataset. The derivation for Equation 13 can be found in Appendix 7.3. In experiments, we find iTM-VAE-Prod always performs better than iTM-VAE, therefore we only place the prior for iTM-VAE-Prod, and refer this variant as iTM-VAE-G.

5. EXPERIMENTS:

In this section, we evaluate the performance of iTM-VAE and its variants on two public benchmarks: 20News and RCV1-V2. To make a fair comparison, we use exactly the same data and vocabulary as (Srivastava & Sutton, 2017) . 2 We compare iTM-VAE and its variants with several state-of-the-arts, such as DocNADE, NVDM, NVLDA and ProdLDA (Srivastava & Sutton, 2017) , GSM, GSB, RSB and RSB-TF, as well as some classical topic models such as LDA and HDP.

The configuration of the experiments is as follows. We use a two-layer fully-connected neural network for g(w 1:N ; ψ) of Equation 8, and the number of hidden units is set to 256 and 512 for 20News and RCV1-V2, respectively. The factor dimension is set to 200 and 1000 for 20News and RCV1-V2, respectively. The truncation level K in Equation 11 is set to 200 so that the maximum topic numbers will never exceed the ones used by baselines.3 Batch-Normalization (Ioffe & Szegedy, 2015) is used to stabilize the training procedure. The hyper-parameter α for GEM distribution is cross-validated on validation set from [10, 20, 30, 50, 100] . We use Adam (Kingma & Ba, 2015) to optimize the model and the learning rate is set to 0.01 for all experiments. The code of iTM-VAE and its variants is available at http://anonymous.

5.1. PERPLEXITY EVALUATION:

Perplexity is widely used by topic models to measure the goodness-to-fit capability, which is defined as: exp(-

1 D D d=1 1 |w d | log p(w d ))

, where D is the number of documents , and |w d | is the number of words in the d-th document w d . Following previous work, the variational lower bound is used to estimate the perplexity.

Table 1 shows the perplexity of different topic models on 20News and RCV1-V2 datasets. Among these baselines, RSB and GSM (Miao et al., 2017) achieved the lowest perplexities on 20News and RCV1-V2, which are 785 and 521, respectively. While the perplexities achieved by iTM-VAE-Prod on these two benchmarks are 769 and 508, respectively, which performs better than the state-of-theart. Moreover, Figure 1-(a ) demonstrates perplexities of finite topic models with different number of topics on 20News. We can see that, a suitable topic number of these models should be around 10 and 20. Interestingly but as expected, the number of effective topics4 discovered by iTM-VAE is about 19, which indicates that the Bayesian nonparametric topic model, iTM-VAE, has the ability to determine a suitable number of topics automatically.

5.2. TOPIC COHERENCE EVALUATION:

As the quality of the learned topics is not directly reflected by perplexity (Newman et al., 2010) , topic coherence is designed to match the human judgment. As Lau et al. (2014) showed that Normalized Pointwise Mutual Information (NPMI) matches the human judgment most closely, we adopt it as the measurement of topic coherence , same as (Miao et al., 2017; Srivastava & Sutton, 2017) . 5 Since the number of topics discovered by iTM-VAE is dynamic, we define the Effective Topic as the topic which becomes the top-1 significant topic of a training sample among the training set more than τ × D times, where D is the number of training samples and τ is a ratio, which is set to 0.5% in our experiments. 6 2017), where LDA is based on (Hoffman et al., 2010) and HDP is based on (Wang et al., 2011) .

Methods:

Table 1 : Comparison of perplexity (lower is better) and topic coherence comparison (higher is better) of different topic models on 20News and RCV1-V2 datasets. The symbol "-" indicates either the model fails to converge within 24 hours, or the original paper does not provide the corresponding values. Following (Miao et al., 2017) , we use an average over topic coherence computed by top-5 and top-10 words for all topics across five random runs, which is more stable and robust (Lau & Baldwin, 2016).

Table 1 shows the topic coherence of different topic models on 20News and RCV1-V2 datasets.

We can clearly see that iTM-VAE-Prod outperforms all the other topic models significantly, which indicates that the topics discovered by iTM-VAE-Prod match more closely to human judgment. Some topics learned by iTM-VAE-Prod are illustrated in Appendix 7.1.

We also illustrate the topic coherence of finite topic models with different numbers of topics, and compare them with iTM-VAE-Prod and iTM-VAE on 20News dataset, shown in Figure 1 -(b). The topic coherence of iTM-VAE-Prod outperforms all baselines over all topic numbers. Another observation is that the best topic coherence of ProdLDA is achieved as the topic number is 15, which is close to the number of effective topics discovered by iTM-VAE-Prod.

5.3. DOCUMENT RETRIEVAL EVALUATION:

The document retrieval task is to evaluate the discriminative power of the document representations learned by each model. We compare iTM-VAE and iTM-VAE-Prod with LDA, DocNADE, NVLDA and ProdLDA. 7 The setup of the retrieval task is as follows. The documents in the training/validation sets are used as the database for retrieval, and the test set is used as the query set. Given a query document, documents in the database are ranked according to the cosine similarities to the query. Precision/Recall (PR) curves can then be computed by comparing the label of the query with those of the database documents. For documents who have multiple labels (e.g. RCV1-V2), the PR curves for each of its labels are computed individually and then averaged for each query document. Finally, the global average of these curves is computed to compare the retrieval performance of each model. 

5.4. ADAPTING α AUTOMATICALLY BY ITM-VAE-G:

As mentioned in Section 4.4, the concentration parameter α of GEM(α) has significant impact on the growth of the number of topics, which affects the performance of iTM-VAE significantly. As a result, α has to be chosen by cross-validation for better performance. Figure 3 (a) also confirms that the number of effective topics increases with α. Consequently, iTM-VAE-G, which places a Gamma(s 1 , s 2 ) prior on α, is proposed to enable the model to adapt α to data automatically.

Another commonly mentioned problem of the optimization under VAE framework is that the latent representation will tend to collapse to the prior (Bowman et al.; Sønderby et al., 2016; Chen et al., 2017) . In our model, this means the choice of α will control the number of the learned topics very tightly when the decoder is strong (e.g. iTM-VAE-Prod with large factor dimension H), which might cause the model to be lack of adaptive power. Common tricks to alleviate the training problem are annealing the relative weight of the KL divergence term (Sønderby et al., 2016) , or regularizing the decoder (Bowman et al.) . Rather than regularizing the decoder, iTM-VAE-G can be regarded as a relaxation on the prior placed on the latent space, which is more effective in improving the adaptive power of our model.

To verify this, we compare the adaptive power of iTM-VAE-Prod with KL annealing and decoder regularization, to iTM-VAE-G, on several subsets of 20News dataset, which contain 1, 2, 5, 10 and 20 (the whole dataset) classes, respectively. For these two models, we use a MLP of two layer of 256 units as the encoder, and the factor dimension of the decoder is H = 200. For iTM-VAE-Prod, we set α = 4, and try different tricks to alleviate the collapse problem: KL annealing where the relative weight of the KL divergence term in the ELBO at epoch n is min(0.005n, 1); Decoder regularization where the weight of the L2 regularization on the decoder is set to 0.1. For iTM-VAE-G, we add a relatively non-informative prior Gamma(1, 0.25) on α, and initialize the global variational parameters γ 1 and γ 2 of Equation 13 the same as the non-informative prior. A SGD optimizer with a learning rate of 0.01 is used to optimize γ 1 and γ 2 . No KL annealing and decoder regularization is used for iTM-VAE-G.

The number of effective topics learned by iTM-VAE-Prod on subsets of 20News dataset is shown in Table 2 . We can see that training tricks like KL-annealing and regularizing the decoder do not help much when the decoder is strong. However, by placing a prior on the concentration parameter α, iTM-VAE-G can increase the adaptive power of the model.

The corpus-level variational posterior of α and the number of effective topics learned by iTM-VAE-G is shown in Table 3 . As for iTM-VAE-G, before training, E q(α|γ1,γ2) [α], the expectation of α given the variational posterior q(α|γ 1 , γ 2 ) is 4. Once the training is done, E q(α|γ1,γ2) [α] will be adjusted to the training set. Table 3 illustrates γ 1 , γ 2 , E q(α|γ1,γ2) [α] and the number of effective topics that are learned from data. We can see that, if the training set contains only 1 class of documents, E q(α|γ1,γ2) [α] will drop to 2.35, and only 6 effective topics are used to model the dataset. Whereas, when the training set consists of 10 classes of documents, E q(α|γ1,γ2) [α] increases to 6.62, and 11 effective topics are discovered by the model to explain the dataset. This indicates that iTM-VAE-G can learn to adjust α to data.

Figure 3 (b) illustrates the topic coverage w.r.t the number of topics when the training set contains 1, 2, 5, 10 and 20 classes, respectively. To this end, we compute the top-1 significant topic for each training document, and sort the topics according to the frequency that it is assigned as top-1. The topic coverage is then defined as the cumulative sum of these frequencies. 

6. CONCLUSION:

In this paper, we propose iTM-VAE, which, to our best knowledge, is the first Bayesian nonparametric topic model that is modeled by Variational Auto-Encoders. Specifically, a stick-breaking prior is used to generate the mixture weights of countably infinite topics and the Kumaraswamy distribution is exploited such that the model can be optimized by AEVB algorithm. Two variants of iTM-VAE are also proposed in this work. One is iTM-VAE-Prod, which replaces the mixture of multinomials assumption of iTM-VAE with a product of experts for better performance. The other one is iTM-VAE-G which places a Gamma prior on the concentration parameter of the stick-breaking process such that the model can adapt the concentration parameter to data automatically. The advantage of iTM-VAE and its variants over the other Bayesian nonparametric topics models is that the inference As shown in Table 4 , iTM-VAE-Prod can learn topics that are diverse and of high quality. One possible reason is that the stick-breaking prior for the document-specific π encourages the model to learn sparse representation, and the model can adjust the number of topics according to the data. Thus the topics can be sufficiently trained and of high diversity. The comparison of representation sparsity is illustrated in Figure 4(a) .

In contrast, the topics learned by ProdLDA (Srivastava & Sutton, 2017) lack diversity. As we listed in Table 5 , there are a lot of redundant topics. As a result, the latent representation learned by ProdLDA is of poor discriminative power. 

7.2. THE EVIDENCE LOWER BOUND OF ITM-VAE:

In this section we show how to compute the Evidence Lower Bound (ELBO) of iTM-VAE which can be written as: • E q ψ (ν|w 1:N ) [log p(w 1:N |π, Φ)]: Similar to other VAE-based models, the SGVB estimator and reparameterization trick can be used to approximate this intractable expectation and propagate the gradient flow into the inference network g. Specifically, we have:

EQUATION

EQUATION

where L is the number of Monte Carlo samples in the SGVB estimator and can be set to 1. N is the number of words in the document.

According to Section 4.2, π (l) can be obtained by

EQUATION

where g(w 1:N ; ψ) is an inference network with parameters ψ, κ denotes the Kumaraswamy distribution and K ∈ R + is the truncation level. Here we omit the superscript * (l) for simplicity.

According to the generative procedure in Section 4.1, p(w i |π (l) , Φ) can be computed by

EQUATION

where t k ∈ R H is the k-th topic factor vector, W ∈ R H×V is the word factor matrix, H is the factor dimension, V is the vocabulary size and σ(•) is the softmax function. • KL (q ψ (ν|w 1:N )||p(ν|α)): By applying the KL divergence of a Kumaraswamy distribution κ(ν; a k , b k ) from a beta distribution p(ν; 1, α), we have: KL (q ψ (ν|w 1:N )||p(ν|α)) = K-1 k=1 KL (q ψ (ν k |w 1:N )||p(ν k |α))

= K-1 k=1 a k -1 a k -γ -Ψ(b k ) - 1 b k + log a k b k + log B (1, α) + (α -1) ∞ m=1 b k m + a k b k B m a k , b k - b k -1 b k (20)

where B(•) is the Beta function and γ is the Euler's constant.

7.3. THE EVIDENCE LOWER BOUND OF ITM-VAE-G:

In this section we show how to compute the Evidence Lower Bound (ELBO) of iTM-VAE-G which can be written as: L(w 1:N |Φ, ψ) = E q ψ (ν|w 1:N ) [log p(w 1:N |π, Φ)] + E q ψ (ν|w 1:N )q(α|γ1,γ2) [log p(ν|α)]

-E q ψ (ν|w 1:N [log q ψ (ν|w 1:N )] -KL(q(α|γ 1 , γ 2 )||p(α|s 1 , s 2 )) (21) Specifically, each item in Equation 21 can be obtained as follows:

• E q ψ (ν|w 1:N ) [log p(w 1:N |π, Φ)]:

The derivation is exactly the same as Appendix 7.2. • E q ψ (ν|w 1:N )q(α|γ1,γ2) [log p(ν|α)]:

Recall that the prior of the stick length variable ν k is Beta(1,α): p(v k |α) = α(1 -ν k ) α-1 and the variational posterior of the concentration parameter α is a gamma distribution q(α; γ 1 , γ 2 ), we have E q ψ (ν|w 1:N )q(α|γ1,γ2) [log p(ν|α)] = E q ψ (ν|w 1:N ) K-1 k=1 E q(α|γ1,γ2) [log α + (α -1) log(1 -ν k )] = (K -1)E q(α|γ1,γ2) [log α] +

EQUATION

Now, we provide more details about the calculation of these two expectations in Equation 22 as follows:

• E q(α|γ1,γ2) [log α]: Fisrt, we can write the gamma distribution q(α; γ 1 , γ 2 ) in its exponential family form:

EQUATION

Considering the general fact that the derivative of the log normalizor log Γ(γ 1 ) -γ 1 log γ 2 of a exponential family distribution with respect to its natural parameter γ 1 is equal to the expectation of the sufficient statistic log α, we can compute E q(α|γ1,γ2) [log α] in the first term of Equation 22 as follows:

E q(α|γ1,γ2) [log α] = Ψ(γ 1 ) -log γ 2 (24) where Ψ is the digamma function, the first derivative of the log Gamma function.

• E q ψ (ν k |w 1:N ) [log(1 -v k )]:

By applying the Taylor expansion, E q(ν k |w 1:N ) [log(1 -v k )] can be written as the infinite sum of the Kumaraswamy's mth moment:

E q ψ (ν k |w 1:N ) [log(1 -v k )] = - ∞ m=1 1 m E q ψ (ν k |w 1:N ) [v m k ] = - ∞ m=1 b k m + a k b k B( m a k , b k ) (25)

where B(•) is the Beta function.

By substituting Equation 24and Equation 25 into Equation 22, we can obtain:

E q ψ (ν|w 1:N )q(α|γ1,γ2) [log p(ν|α)]

EQUATION

• -E q ψ (ν|w 1:N ) [log q ψ (ν|w 1:N )]:

According to Section 4.11 of (Michalowicz et al., 2013) , the Kumaraswamy's entropy is given as -E q ψ (ν|w 1:N ) [log q ψ (ν|w 1:N )] = -

EQUATION

)

where γ is the Euler's constant. • KL(q(α|γ 1 , γ 2 )||p(α|s 1 , s 2 )):

The KL divergence of one gamma distribution q(α; γ 1 , γ 2 ) from another gamma distribution p(α; s 1 , s 2 ) evaluates to

EQUATION

:

Ideally, Beta distribution is the most suitable probability candidate, since iTM-VAE assumes π is drawn from a GEM distribution in the generative process. However, as Beta does not satisfy the DNCP requirement of SGVB, we use the Kumaraswamy distribution, which is described in Section 3.3, for iTM-VAE.

Since there are no labels for the datasets provided by(Srivastava & Sutton, 2017) and(Miao et al., 2017), we used a version from(Srivastava et al., 2013) for document retrieval tasks, which contains label information.

In these baselines, at most 200 topics are used. Please refer Table 1 for details.

The definition of Effective Topic can be found in Section

5.2.5 We use the code provided by(Lau et al., 2014) at https://github.com/jhlau/topic_ interpretability/6 This means the ineffective topics are the ones act as top-1 important topic less than 0.5% times among all documents in the training set.

Since Miao et al. (2017) did not test the document retrieval performance and the code is not available now, we cannot compare with it during the draft preparation.

We use the code provided by(Srivastava & Sutton, 2017) to get their sparsity curve.

Figures and Tables:

Figure 1: Perplexity and topic coherence with different number of topics. The lines for iTM-VAE and iTM-VAE-Prod are horizontal as the topics used by the models are dynamic and adapted to data.

Figure 2 illustrates the PR curves of different models with hidden representations of length 128. Specifically, the mean of the variational Gaussian posterior is used as the features for NVLDA and ProdLDA. Since the effective topic numbers of iTM-VAE-Prod and iTM-VAE are dynamic and usually much smaller than 128 on both datasets, we use a weighted sum of topic factor vectors over the topic distributions, where the factor dimension is 128. As shown in Figure 2(a) and Figure 2(b), iTM-VAE-Prod always yields competitive results on both datasets, and outperforms the others in most cases. We also map the latent representations learned by iTM-VAE-Prod to a 2D space by TSNE (Maaten & Hinton, 2008) and visualize the representations in Figure 2(c).

Figure 2: Precision-Recall curves for document retrieval task on 20News (a) and RCV1-V2 (b). TSNE-visualization of the representations learned by iTM-VAE-Prod on 20News (c).

Figure3(b) illustrates the topic coverage w.r.t the number of topics when the training set contains 1, 2, 5, 10 and 20 classes, respectively. To this end, we compute the top-1 significant topic for each training document, and sort the topics according to the frequency that it is assigned as top-1. The topic coverage is then defined as the cumulative sum of these frequencies. Figure3(b) shows that, with the increasing of the number of classes, more topics are utilized by iTM-VAE-G to reach the same level of topic coverage, which indicates that the model has the ability to adapt to data. #classes H = 200 H = 200 with KL annealing H = 200 with decoder L2 regularization 1 10 10 9 2 10 9 10 5 9 10 10 10 9 9 10 20 10 9 10

Figure 3: (a) Number of effective topics w.r.t α on 20News. (b) Topic coverage w.r.t number of topics learned by iTM-VAE-G.

Figure 4(c) shows the TSNE-visualization of the representations learned by ProdLDA with the best topic coherence on 20News.

Figure 4: (a) Representation sparsity of different models on 20News. We sample one topic assignment π for each document, sort and then average across the test set. 9 (b) The TSNE-visualization of the representation learned by by iTM-VAE-Prod. (c) The TSNE-visualization of the representation learned by ProdLDA (Srivastava & Sutton, 2017) with the best topic coherence on 20News (K = 50).

Number of effective topics learned by iTM-VAE-Prod on subsets of 20News dataset.

Table content: <table><tr><td>#classes</td><td>γ 1</td><td>γ 2</td><td colspan="2">E q(α|γ1,γ2) [α] #Effective topics learned by</td></tr><tr><td/><td/><td/><td/><td>iTM-VAE-G H = 200</td></tr><tr><td>1</td><td colspan="2">18.30 7.80</td><td>2.35</td><td>6</td></tr><tr><td>2</td><td colspan="2">25.03 6.38</td><td>3.92</td><td>9</td></tr><tr><td>5</td><td colspan="2">34.10 6.19</td><td>5.51</td><td>9</td></tr><tr><td>10</td><td colspan="2">42.96 6.49</td><td>6.62</td><td>11</td></tr><tr><td>20</td><td colspan="2">52.97 7.23</td><td>7.33</td><td>14</td></tr></table>

Learned posterior distribution of α and number of effective topics learned by iTM-VAE-G on subsets of 20News dataset.

Table content: <table/>

Hardware scsi ide scsus motherboard ram controller upgrade meg cache floppy Encryption ripem escrow rsa des encrypt cipher privacy crypto chip nsa Trade shipping sale annual manual tor vs det cd price excellent X system window xterm font colormap server xlib widget xt windows toolkit Hockey † det tor buf cal pit que mon pt vs calgary Health msg ‡ patient disease symptom doctor food pain mouse cancer hospital Circuit voltage puck connector signal amp input circuit pin wire connect Lawsuit gun homicide militia weapon amendment handgun criminal firearm crime knife Traffic bike brake car tire ride engine honda bmw rear motorcycle †: All these words are about hockey teams of different cities, e.g. "que" means Quebec. ‡: "msg" means monosodium glutamate.

Table content: <table><tr><td colspan="2">7 APPENDIX</td></tr><tr><td colspan="2">7.1 LEARNED TOPICS AND DISCUSSION</td></tr><tr><td colspan="2">Geography turkish armenians turks armenia armenian turkey azerbaijan greek greece village</td></tr><tr><td>Sports</td><td>season team player nhl score playoff hockey game coach hitter</td></tr><tr><td>Religion</td><td>jesus bible god faith scripture christ doctrine belief eternal church</td></tr><tr><td>Space</td><td>orbit shuttle launch lunar spacecraft nasa satellite probe rocket moon</td></tr></table>

Top 10 words of topics learned by iTM-VAE-Prod without cherry picking.

Table content: <table/>

Top 10 words of some redundant topics learned by ProdLDA.

Table content: <table><tr><td/><td>1 jesus christian scripture faith god christ heaven christianity verse resurrection</td></tr><tr><td>Topics about Religion</td><td>2 jesus christ doctrine revelation verse scripture satan christian interpretation god 3 belief god passage scripture moral atheist christian truth principle jesus 4 god belief existence faith jesus atheist bible christian religion sin</td></tr><tr><td/><td>5 jesus son holy christ father god doctrine heaven spirit prophet</td></tr><tr><td/><td>6 homosexual marriage belief islam moral</td></tr><tr><td/><td>christianity truth islamic religion god</td></tr><tr><td/><td>7 floppy controller scsus ide scsi ram hd mb cache isa</td></tr><tr><td>Topics</td><td>8 printer meg adapter scsi motherboard windows modem mhz vga hd</td></tr><tr><td>about</td><td>9 ide mb connector controller isa scsi scsus floppy jumper disk</td></tr><tr><td>Hardware</td><td>10 mb controller bio rom interface mhz scsus scsi floppy ide</td></tr><tr><td/><td>11 ide meg motherboard shipping adapter simm hd mhz monitor scsi</td></tr><tr><td/><td>12 ram controller dos bio windows disk scsi rom scsus meg</td></tr><tr><td/><td>13 honda motherboard bike amp quadra hd brake apple upgrade meg</td></tr><tr><td/><td>14 decision stephanopoulos president armenian gay</td></tr><tr><td>Topics about Politics</td><td>package congress myers february armenians 15 armenian turkish genocide armenians turks jesus massacre muslim armenia muslims 16 armenians father gang soldier neighbor apartment girl armenian troops rape</td></tr><tr><td/><td>17 muslim greek turks turkish armenian muslims village genocide armenia jews</td></tr><tr><td/><td>18 armenian turks armenians armenia turkish</td></tr><tr><td/><td>muslim massacre village turkey greek</td></tr><tr><td/><td>19 armenians armenian neighbor apartment woman</td></tr><tr><td/><td>soviet kill bus azerbaijan hide</td></tr><tr><td>Topics</td><td>20 morality truth moral objective absolute</td></tr><tr><td>about</td><td>belief murder existence principle human</td></tr><tr><td>Lawsuit</td><td>21 homicide vancouver seattle handgun firearm child states percent study file</td></tr><tr><td/><td>22 murder moral constitution morality criminal objective rights gun law weapon</td></tr><tr><td>. . .</td><td>. . . . . .</td></tr></table>

Peer review (iclr2018):
 "topic modeling of text documents one of most important tasks" Does this claim have any backing? "inference of HDP is more complicated and not easy to be applied to new models" Really an artifact of the misguided nature of earlier work. The posterior for the of a elements of DP or HDP can be made a Dirichlet, made finite by keeping a "remainder" term and appropriate augmentation. Hughes, Kim and Sudderth (2015) have avoided stick-breaking and CRPs altogether, as have others in earlier work. Extensive models building on simple HDP doing all sorts of things have been developed. Variational stick-breaking methods never seemed to have worked well. I suspect you could achieve better results by replacing them as well, but you would have to replace the tree of betas and extend your Kumaraswamy distribution, so it may not work. Anyway, perhaps an avenue for future work. "infinite topic models" I*ve always taken the view that the use of the word "infinite" in machine learning is a kind of NIPSian machismo. In HDP-LDA at least, the major benefit in model performance comes from fitting what you call , which is uniform in vanilla LDA, and note that the number of topics "found" by a HDP-LDA sampler can be made to vary quite widely by varying what you call , so any statement about the "right" number of topics is questionable. So the claim in 3rd paragraph of Section 2, "superior" and "self-determined topic number" I*d say are misguided. Plenty of experimental work to support this. In Related Work, you seem to only mention HDP for non-parametric topic models. More work exists, for instance using Pitman-Yor distributions for modelling words and using Gibbs samplers that are efficient and don*t rely on the memory hungry HCRP. Good to see a prior is placed on the concentration parameter. Very important and not well done in the community, usually. ADDED: Originally done by Teh et al for HDP-LDA, and subsequently done by several, including Kim et al 2016. Others stress the importance of this. You need to cite at least Teh et al. in 5.4 to show this isn*t new and the importance is well known. The Prod version is a very nice idea. Great results. This looks original, but I*m not expert enough in the huge masses of new deep neural network research popping up. You*ve upped the standard a bit by doing good experimental work. Oftentimes this is done poorly and one is left wondering. A lot of effort went into this. ADDED: usually like to see more data sets experimented with What code is used for HDP-LDA? Teh*s original Matlab HCRP sampler does pretty well because at least he samples hyperparameters and can scale to 100k documents (yes, I tried). The comparison with LDA makes me suspicious. For instance, on 20News, a good non-parametric LDA will find well over 400 topics and roundly beat LDA on just 50 or 200. If reporting LDA, or HDP-LDA, it should be standard to do hyperparameter fitting and you need to mention what you did as this makes a big difference. ADDED: 20News results still poor for HPD, but its probably the implementation used ... their online variational algorithm only has advantages for large data sets Pros: * interesting new prod model with good results * alternative "deep" approach to a HDL-LDA model * good(-ish) experimental work Cons: * could do with a competitive non-parametric LDA implementation ADDED: good review responses generally