Title: WHAI: WEIBULL HYBRID AUTOENCODING INFERENCE FOR DEEP TOPIC MODELING

Abstract: To train an inference network jointly with a deep generative topic model, making it both scalable to big corpora and fast in out-of-sample prediction, we develop Weibull hybrid autoencoding inference (WHAI) for deep latent Dirichlet allocation, which infers posterior samples via a hybrid of stochastic-gradient MCMC and autoencoding variational Bayes. The generative network of WHAI has a hierarchy of gamma distributions, while the inference network of WHAI is a Weibull upward-downward variational autoencoder, which integrates a deterministicupward deep neural network, and a stochastic-downward deep generative model based on a hierarchy of Weibull distributions. The Weibull distribution can be used to well approximate a gamma distribution with an analytic Kullback-Leibler divergence, and has a simple reparameterization via the uniform noise, which help efficiently compute the gradients of the evidence lower bound with respect to the parameters of the inference network. The effectiveness and efficiency of WHAI are illustrated with experiments on big corpora.

1. INTRODUCTION:

There is a surge of research interest in multilayer representation learning for documents. To analyze the term-document count matrix of a text corpus, Srivastava et al. (2013) extend the deep Boltzmann machine (DBM) with the replicated softmax topic model of Salakhutdinov & Hinton (2009) to infer a multilayer representation with binary hidden units, but its inference network is not trained to match the true posterior (Mnih & Gregor, 2014) and the higher-layer neurons learned by DBM are difficult to visualize. The deep Poisson factor models of Gan et al. (2015) are introduced to generalize Poisson factor analysis (Zhou et al., 2012) , with a deep structure restricted to model binary topic usage patterns. Deep exponential families (DEF) of Ranganath et al. (2015) construct more general probabilistic deep networks with non-binary hidden units, in which a count matrix can be factorized under the Poisson likelihood, with the gamma distributed hidden units of adjacent layers linked via the gamma scale parameters. The Poisson gamma belief network (PGBN) (Zhou et al., 2015; 2016) also factorizes a count matrix under the Poisson likelihood, but factorizes the shape parameters of the gamma distributed hidden units of each layer into the product of a connection weight matrix and the gamma hidden units of the next layer, resulting in strong nonlinearity and readily interpretable multilayer latent representations.

Those multilayer probabilistic models are often characterized by a top-down generative structure, with the distribution of a hidden layer typically acting as a prior for the layer below. Despite being able to infer a multilayer representation of a text corpus with scalable inference (Patterson & Teh, 2013; Ruiz et al., 2016; Cong et al., 2017a) , they usually rely on an iterative procedure to infer the latent representation of a new document at the testing stage, regardless of whether variational inference or Markov chain Monte Carlo (MCMC) is used. The potential need of a large number of iterations per testing document makes them unattractive when real-time processing is desired. For example, one may need to rapidly extract the topic-proportion vector of a document and use it for downstream analysis, such as identifying key topics and retrieving related documents. A potential solution is to construct a variational autoencoder (VAE) that learns the parameters of an inference network (recognition model or encoder) jointly with those of the generative model (decoder) (Kingma & Welling, 2014; Rezende et al., 2014) . However, most existing VAEs rely on Gaussian latent variables, with the neural networks (NNs) acting as nonlinear transforms between adjacent layers (Sonderby et al., 2016; Dai et al., 2016; Ishaan et al., 2017) . A primary reason is that there is a simple reparameterization trick for Gaussian latent variables that allows efficiently computing the noisy gradients of the evidence lower bound (ELBO) with respect to the NN parameters. Unfortunately, Gaussian based distributions often fail to well approximate the posterior distributions of sparse, nonnegative, and skewed document latent representations. For example, Srivastava & Sutton (2017) propose autoencoding variational inference for topic models (AVITM), as shown in Fig. 2b , which utilizes the logistic-normal distribution to approximate the posterior of the latent representation of a document; even though the generative model is latent Dirichlet allocation (LDA) (Blei et al., 2003) , a basic single-hidden-layer topic model, due to the insufficient ability of the logistic-normal distribution to model sparsity, AVITM has to rely on some heuristic to force the latent representation of a document to be sparse. Another common shortcoming of existing VAEs is that they often only provide a point estimate for the global parameters of the generative model, and hence their inference network is optimized to approximate the posteriors of the local parameters conditioning on the data and the point estimate, rather than a full posterior, of the global parameters. In addition, from the viewpoint of probabilistic modeling, the inference network of a VAE is often merely a shallow probabilistic model, whose parameters, though, are deterministically nonlinearly transformed from the observations via a non-probabilistic deep neural network.

To address these shortcomings and move beyond Gaussian latent variable based deep models and inference procedures, we develop Weibull hybrid autoencoding inference (WHAI), a hybrid Bayesian inference for deep topic modeling that integrates both stochastic-gradient MCMC (Welling & Teh, 2011; Ma et al., 2015; Cong et al., 2017a) and a multilayer Weibull distribution based VAE. WHAI is related to a VAE in having both a decoder and encoder, but differs from a usual VAE in the following ways: 1) deep latent Dirichlet allocation (DLDA), a probabilistic deep topic model equipped with a gamma belief network, acts as the generative model; 2) inspired by the upward-downward Gibbs sampler of DLDA, as sketched in Fig. 2c , the inference network of WHAI uses a upwarddownward structure, as shown in Fig. 2a , to combine a non-probabilistic bottom-up deep NN and a probabilistic top-down deep generative model, with the th hidden layer of the generative model linked to both the ( + 1)th hidden layer of itself and the th hidden layer of the deep NN; 3) a hybrid of stochastic-gradient MCMC and autoencoding variational inference is employed to infer both the posterior distribution of the global parameters, represented as collected posterior MCMC samples, and a VAE that approximates the posterior distribution of the local parameters given the data and a posterior sample (rather than a point estimate) of the global parameters; 4) we use the Weibull distributions in the inference network to approximate gamma distributed conditional posteriors, exploiting the fact that the Weibull and gamma distributions have similar probability density functions (PDFs), the Kullback-Leibler (KL) divergence from the Weibull to gamma distributions is analytic, and a Weibull random variable can be efficiently reparameterized with a uniform noise.

Note that we have also tried gamma hybrid autoencoding inference (GHAI), which directly uses the gamma distribution in the probabilistic top-down part of the inference network, while using rejection sampling variational inference (RSVI) of Naesseth et al. to approximately compute the gradient of the ELBO. While RSVI is a very general technique that can be applied to a wide variety of non-reparameterizable distributions, we find that for replacing the reparameterizable Weibull with non-reparameterizable gamma distributions in the inference network, the potential gains are overshadowed by the disadvantages of having to rely on an approximate reparameterization scheme guided by rejection sampling. In the experiments for deep topic modeling, we show that WHAI clearly outperforms GHAI, and both WHAI and GHAI outperform their counterparts that remove the top-down links of the inference network, referred to as WHAI-independent and GHAI-independent, respectively; WHAI is comparable to Gibbs sampling in terms performance, but is scalable to big training data via mini-batch stochastic-gradient based inference and is considerably fast in out-ofsample prediction via the use of an inference network.

2. WHAI FOR MULTILAYER DOCUMENT REPRESENTATION:

Below we first describe the decoder and encoder of WHAI, and then provide a hybrid stochasticgradient MCMC and autoencoding variational inference that is fast in both training and testing.

2.1. DOCUMENT DECODER: DEEP LATENT DIRICHLET ALLOCATION:

In order to capture the hierarchical document latent representation, WHAI uses the Poisson gamma belief network (PGBN) of Zhou et al. (2016) , a deep probabilistic topic model, as the generative network (encoder). Choosing a deep generative model as its decoder distinguishes WHAI from both AVITM, which uses a "shallow" LDA as its decoder, and a conventional VAE, which often uses as its decoder a "shallow" (transformed) Gaussian distribution, whose parameters are deterministically nonlinearly transformed from the observation via "black-box" deep neural networks. With all the gamma latent variables marginalized out, as shown in Cong et al. (2017a) , the PGBN can also be represented as deep LDA (DLDA). For simplicity, below we use DLDA to refer to both the PGBN and DLDA representations of the same underlying deep generative model, as briefly described below. Note the single-hidden-layer version of DLDA reduces to Poisson factor analysis of Zhou et al. (2012) , which is closely related to LDA. Let us denote Φ (1) ∈ R K0×K1 + and θ (1) n ∈ R K1 + as the factor loading and latent representation of the first hidden layer of DLDA, respectively, where R + = {x, x ≥ 0} and K 1 is the number of topics (factors) of the first layer. We further restrict that the sum of each column of Φ (1) is equal to one. To model high-dimensional multivariate sparse count vectors x n ∈ Z K0 , where Z = {0, 1, . . .}, under the Poisson likelihood, the DLDA generative model with L hidden layers, from top to bottom, can be expressed as

EQUATION

)

where the hidden units θ (l) n ∈ R K l + of layer l are factorized into the product of the factor loading

Φ (l) ∈ R K l-1 ×K l +

and hidden units of the next layer. It infers a multilayer data representation, and can visualize its topic φ (l) k at hidden layer l as

l-1 t=1 Φ (t) φ (l)

k , which tend to be very specific in the bottom layer and become increasingly more general when moving upward. The unsupervisedly extracted multilayer latent representations θ (l) n are well suited for additional downstream analysis, such as document classification and retrieval.

The upward-downward Gibbs sampling for DLDA, as described in detail in Zhou et al. (2016) , is sketched in Fig. 2c , where Z l represent augmented latent counts that are sampled upward given the observations and model parameters. While having closed-form update equations, the Gibbs sampler requires processing all documents in each iteration and hence has limited scalability. Consequently, a topic-layer-adaptive stochastic gradient Riemannian (TLASGR) MCMC for DLDA, referred to as DLDA-TLASGR, is proposed to process big corpora (Cong et al., 2017a) . Different from AVITM (Srivastava & Sutton, 2017) that models a probabilistic simplex with the expanded-natural representation (Patterson & Teh, 2013) , DLDA-TLASGR uses a more elegant simplex constraint and increases the sampling efficiency via the use of the Fisher information matrix (FIM) (Cong et al., 2017a; b) , with adaptive step-sizes for the topics of different layers. Specifically, suppose φ

(l) k is the kth topic in layer with prior φ (l) k ∼ Dirichlet(η (l)

k ), sampling it can be efficiently realized as

EQUATION

where M k is calculated using the estimated FIM, both z:k• and z•k• come from the augmented latent counts Z, and [•] ∠ denotes a simplex constraint; more details about TLASGR-MCMC for DLDA can be found in Cong et al. (2017a) and are omitted here for brevity. The KL divergence from the inferred Weibull distribution to the target gamma one as (a) Gamma(0.05, 1), (b) Gamma(0.5, 1), and (c) Gamma(5, 1). Subplot (d) shows the KL divergence as a function of the gamma shape parameter, where the gamma scale parameter is fixed at 1. Despite the attractive properties, neither the Gibbs sampler nor TLASGR-MCMC of DLDA can avoid taking a potentially large number of MCMC iterations to infer the latent representation of a testing document, which hinders real-time processing of the incoming documents and motivates us to construct an inference network with fast out-of-sample prediction, as described below.

2.2. DOCUMENT ENCODER: WEIBULL UPWARD-DOWNWARD VARIATIONAL ENCODER:

A VAE uses an inference network to map the observations directly to their latent representations. However, their success so far is mostly restricted to Gaussian distributed latent variables, and does not generalize well to model sparse, nonnegative, and skewed latent document representations. To move beyond latent Gaussian models, below we propose Weibull upward-downward variational encoder (WUDVE) to efficiently produce a document's multilayer latent representation under DLDA.

Assuming the global parameters φ (l) k of DLDA shown in (1) are given and the task is to infer the local parameters θ (l+1) n , the usual strategy of mean-field variational Bayes (Jordan et al., 1999) is to maximize the ELBO that can be expressed as

EQUATION

where the expectations are taken with respect to (w.r.t.) a fully factorized distribution as

EQUATION

Instead of using a conventional latent Gaussian based VAE, in order to model sparse and nonnegative latent document representation, it might be more appropriate to use a gamma distribution based inference network defined as

q(θ n | x n ) = Gamma(f W (x n ), g W (x n ))

, where f and g are two related deep neural networks parameterized by W. However, it is hard to efficiently compute the gradient of the ELBO with respect to W, due to the difficulty to reparameterize a gamma distributed random variable (Kingma & Welling, 2014; Ruiz et al., 2016; Knowles, 2015) , motivating us to identify a surrogate distribution that can not only well approximate the gamma distribution, but also be easily reparameterized. Below we show the Weibull distribution is an ideal choice.

2.2.1. WEIBULL AND GAMMA DISTRIBUTIONS:

A main reason that we choose the Weibull distribution to construct the inference network is that the Weibull and gamma distributions have similar PDFs:

Weibull PDF:

P (x | k, λ) = k λ k x k-1 e (x/λ) k , Gamma PDF: P (x | α, β) = β α Γ(α) x α-1 e -βx ,

where x ∈ R + . Another reason is due to a simple reparameterization for x ∼ Weibull(k, λ) as

x = λ(-ln(1 -)) 1/k , ∼ Uniform(0, 1).

Moreover, its KL-divergence from the gamma distribution has an analytic expression as

KL(Weibull(k, λ)||Gamma(α, β)) = α ln λ- γα k -ln k -βλΓ 1+ 1 k +γ +1+α ln β -ln Γ(α).

Minimizing this KL divergence, one can identify the two parameters of a Weibull distribution to approximate a given gamma one. As shown in Fig. 1 , the inferred Weibull distribution in general quite accurately approximates the target gamma one, as long as the gamma shape parameter is neither too close to zero nor too large.

2.2.2. UPWARD-DOWNWARD INFORMATION PROPAGATION:

For the DLDA upward-downward Gibbs sampler sketched in Fig. 2c , the corresponding Gibbs sampling update equation for θ (l) n can be expressed as

EQUATION

where m

(l)(l+1) n and p (l)

n are latent random variables constituted by information upward propagated to layer l, as described in detail in Zhou et al. (2016) and hence omitted here for brevity. It is clear from (5) that the conditional posterior of θ (l) n is related to both the information at the higher (prior) layer, and that upward propagated to the current layer via a series of data augmentation and marginalization steps described in Zhou et al. (2016) . Inspired by this instructive upwarddownward information propagation in Gibbs sampling, as shown in Fig. 2a , we construct WUDVE, the inference network of our model, as

EQUATION

The Weibull distribution is used to approximate the gamma distributed conditional posterior, and its parameters k (l) n ∈ R K l and λ (l) n ∈ R K l are both deterministically transformed from the observation x n using the neural networks, as illustrated in Fig. 2a and specified as

EQUATION

EQUATION

EQUATION

where

h (0) n = log(1 + x n ), W (l) 1 ∈ R K l ×K l , W (l) 2 ∈ R K l ×K l , W (l) 3 ∈ R K l ×K l-1 , b (l) 1 ∈ R K l , b (l) 2 ∈ R K l , and b (l)

Comparing Figs. 2c and 2a show that in each iteration, both Gibbs sampling and WUDVE have not only an upward information propagation (orange arrows), but also a downward one (blue arrows), but their underlying implementations are distinct from each other. Gibbs sampling in Fig. 2c does not have an inference network and needs the local variables θ (l) n to help perform stochastic upward information propagation, whereas WUDVE in Fig. 2a uses its non-probabilistic part to perform deterministic upward information propagation, without relying on the local variables θ (l) n . It is also interesting to notice that the upward-downward structure of WUDVE, motivated by the upwarddownward Gibbs sampler of DLDA, is closely related to that used in the ladder VAE of Sonderby et al. (2016) . However, to combine the bottom-up and top-down information, ladder VAE relies on some heuristic restricted to Gaussian latent variables.

2.3. HYBRID MCMC/VAE INFERENCE:

In Section 2.1, we describe how to use TLASGR-MCMC of Cong et al. (2017a) , a stochasticgradient MCMC algorithm for DLDA, to sample the global parameters {Φ (l) } 1,L ; whereas in Section 2.2.2, we describe how to use WUDVE, an autoencoding variational inference network, to

h 3 h  1 Z 3  ( ) 2  ( ) X 2 Z  (1) 3 Z   2    3  3  ( ) 2  ( ) X  (1) 1 h   3  ( ) 2  ( )  (1) X   1    2    3    2    3  2 h  X 1 h X  (1)   1   (1) DLDA Inference Generative Inference Generative (a) WHAI 2 h 3 h  1 Z 3  ( ) 2  ( ) X 2 Z  (1) 3 Z   2    3  3  ( ) 2  ( ) X  (1) 1 h   3  ( ) 2  ( )  (1) X   1    2    3    2    3  2 h  X 1 h X  (1)   1   (1) DLDA Inference Generative Inference Generative (b) AVITM 2 h 3 h  1 Z 3  ( ) 2  ( ) X 2 Z  (1) 3 Z   2    3  3  ( ) 2  ( ) X  (1) 1 h   3  ( ) 2  ( )  (1) X   1    2    3    2    3  2 h  X 1 h X  (1)   1   (1)

Gibbs Sampling Inference Generative Inference Generative approximate the conditional posterior of the local parameters {θ (l) n } 1,L given {Φ (l) } 1,L and observation x n . Rather than merely finding a point estimate of the global parameters {Φ (l) } 1,L , we describe in Algorithm 1 how to combine TLASGR-MCMC and the proposed WUDVE into a hybrid MCMC/VAE inference algorithm, which infers posterior samples for both the global parameters {Φ (l) } 1,L of the generative network, and the corresponding neural network parameters

3  ( ) 2  ( )  (1) X   1    2    3  Generative (c) DLDA

Ω = {W (l) 1 , b (l) 1 , W (l) 2 , b (l) 2 , W (l) 3 , b (l)

3 } 1,L of the inference network. Being able to efficiently evaluating the gradient of the ELBO is important to the success of a variational inference algorithm (Hoffman et al., 2013; Paisley et al., 2012; Kingma & Welling, 2014; Mnih & Gregor, 2014; Ranganath et al., 2015; Ruiz et al., 2016; Rezende et al., 2014) . An important step of Algorithm 1 is calculating the gradient of the ELBO in (3) with respect to the NN parameters Ω. Thanks to the choice of the Weibull distribution, the second term of the ELBO in (3) is analytic, and due to simple reparameterization of the Weibull distribution, the gradient of the first term of the ELBO with respect to Ω can be accurately evaluated, achieving satisfactory performance using even a single Monte Carlo sample, as shown in our experimental results. Thanks to the architecture of WUDVE, using the inference network, for a new mini-batch, we can directly find the conditional posteriors of {θ (l) n } 1,L given {Φ (l) } 1,L and the stochastically updated Ω, with which we can sample the local parameters and then use TLASGR-MCMC to stochastically update the global parameters {Φ (l) } 1,L .

2.4. VARIATIONS OF WHAI:

To clearly understand how each component contributes to the overall performance of WHAI, below we consider two different variations: GHAI and WAI. We first consider gamma hybrid autoencoding inference (GHAI). In WUDVE, the inference network for WHAI, we have a deterministic-upward and stochastic-downward structure, where the reparameterizable Weilbull distribution is used to connect adjacent stochastic layers. Although we choose to use the Weibull distribution for the reasons specified in Section 2.2.1, one may also choose some other distribution in the downward structure. For example, one may choose the gamma distribution and replace (6) with

EQUATION

Even though the gamma distribution does not have a simple reparameteriation, one may use the RSVI of Naesseth et al. to define an approximate reparameterization procedure via rejection sampling. More specifically, following Naesseth et al., to generate a gamma random variable z ∼ Gamma(α, β), one may first use the rejection sampler of Marsaglia & Tsang (2000) to generate z ∼ Gamma(α + B, 1), for which the proposal distribution is expressed as 

z = α + B - 1 3 1 + ε 9(α + B) -3 3 , ε ∼ N (0, 1),

EQUATION

)

where S is the total number of collected samples and y

•• = V v=1 N n=1 y vn .

For the proposed model, we set the mini-batch size as 200, and use as burn-in 2000 mini-batches for both 20News and RCV1 and 3500 for wiki. We collect 3000 samples after burn-in to calculate perplexity. The hyperparameters of WHAI are set as: η (l) = 1/K l , r = 1, and c (l) n = 1. Table 1 lists for various algorithms both the perplexity and the average run time per testing document given a single sample (estimate) of the global parameters. Clearly, given the same generative network structure, DLDA-Gibbs performs the best in terms of predicting heldout word tokens, which is not surprising as this batch algorithm can sample from the true posteriors given enough Gibbs sampling iterations. DLDA-TLASGR is a mini-batch algorithm that is much more scalable in training than DLDA-Gibbs, at the expense of slighted degraded performance in out-of-sample prediction. Both DLDA-WAI, using SGD to infer the global parameters, and DLDA-WHAI, using a stochasticgradient MCMC to infer the global parameters, slightly underperform DLDA-TLASGR; all minibatch based algorithms are scalable to a big training corpus, but due to the use of the WUDVE inference network, both DLDA-GHAI and DLDA-WHAI, as well as their variations, are considerably fast in processing a testing document. In terms of perplexity, all algorithms with DLDA as the generative model clearly outperform both DPFA of Gan et al. (2015) and AVITM of Srivastava & Sutton (2017) , while in terms of the computational cost for testing, all algorithms with an inference network, such as AVITM, DLDA-GHAI, and DLDA-WHAI, clearly outperform these relying on an interactive procedure for out-of-sample prediction, including DPFA, DLDA-Gibbs, and DLDA-TLASGR. It is also clear that except for DLDA-GHAI-Independent and DLDA-WHAI-Independent that have no stochastic-downward components in their inference, all the other algorithms with DLDA as the generative model have a clear trend of improvement as the generative network becomes deeper, in- dicating the importance of having stochastic-downward information propagation during posterior inference; and DLDA-WHAI with a single hidden layer already clearly outperforms AVITM, indicating that using the Weibull distribution is more appropriate than using the logistic-normal distribution to model the document latent representation. Furthermore, thanks to the use of the stochastic gradient based TLASGR-MCMC rather than a simple SGD procedure, DLDA-WHAI consistently outperforms DLDA-WAI. Last but not least, while DLDA-GHAI that relies on RSVI to approximately reparameterize the gamma distributions clearly outperforms AVITM and DPFA, it clearly underperforms DLDA-WHAI that has simple reparameterizations for its Weibull distributions.

Below we examine how various inference algorithms progress over time during training, evaluated with per-holdout-word perplexity. As clearly shown in Fig. 3 , DLDA-WHAI outperforms DPFA and AVITM in providing lower perplexity as time progresses, which is not surprising as the DLDA multilayer generative model is good at document representation, while AVITM is only "deep" in the deterministic part of its inference network and DPFA is restricted to model binary topic usage patterns via its deep network. When DLDA is used as the generative model, in comparison to Gibbs sampling and TLASGR-MCMC on two large corpora, RCV1 and Wiki, the mini-batch based WHAI converges slightly slower than TLASGR-MCMC but much faster than Gibbs sampling; WHAI consistently outperforms WAI, which demonstrates the advantage of the hybrid MCMC/VAE inference; in addition, the RSVI based DLDA-GHAI clearly converges more slowly in time than DLDA-WHAI. Note that for all three datasets, the perplexity of TLASGR decreases at a fast rate, followed by closely by WHAI, while that of Gibbs sampling decreases slowly, especially for RCV1 and Wiki, as shown in Figs. 3(b-c ). This is expected as both RCV1 and Wiki are much larger corpora, for which a mini-batch based inference algorithm can already make significant progress in inferring the global model parameters, before a batch-learning Gibbs sampler finishes a single iteration that needs to go through all documents. We also notice that although AVITM is fast for testing via the use of a VAE, its representation power is limited due to not only the use of a shallow topic model, but also the use of a latent Gaussian based inference network that is not naturally suited to model document latent representation.

3.2. TOPIC HIERARCHY AND MANIFOLD:

In addition to quantitative evaluations, we have also visually inspected the inferred topics at different layers and the inferred connection weights between the topics of adjacent layers. Distinct from many existing deep learning models that build nonlinearity via "black-box" neural networks, we can easily visualize the whole stochastic network, whose hidden units of layer l -1 and those of layer l are connected by φ (l) k k that are sparse. In particular, we can understand the meaning of each hidden unit by projecting it back to the original data space via l-1 t=1 Φ (t) φ (l) k . We show in Fig. 4 a subnetwork, originating from units 16, 19, and 24 of the top hidden layer, taken from the generative network of size 128-64-32 inferred on Wiki. The semantic meaning of each topic and the connections between different topics are highly interpretable. We provide several additional topic hierarchies for Wiki in the Appendix. To further illustrate the effectiveness of our multilayer representation in our model, we apply a threehidden-layer WHAI to MNIST digits and present the learned dictionary atoms. We use the Poisson likelihood directly to model the MNIST digit pixel values that are nonnegative integers ranging from 0 to 255. As shown in Figs. 5a-5c , it is clear that the factors at layers one to three represent localized points, strokes, and digit components, respectively, that cover increasingly larger spatial regions. This type of hierarchical visual representation is difficult to achieve with other types of deep neural networks (Srivastava et al., 2013; Kingma & Welling, 2014; Rezende et al., 2014; Sonderby et al., 2016) .

WUDVE, the inference network of WHAI, has a deterministic-upward-stochastic-downward structure, in contrast to a conventional VAE that often has a pure deterministic bottom-up structure. Here, we further visualize the importance of the stochastic-downward part of WUDVE through a simple experiment. We remove the stochastic-downward part of WUDVE shown in ( 6) and define the inference network as q(θ (l) 

n | h (l) n ) = Weibull(k (l) n , λ (l) n ),

B MANIFOLD ON DOCUMENTS:

From a sci.medicine document to an eci.space one 1. com, writes, article, edu, medical, pitt, pain, blood, disease, doctor, medicine, treatment, patients, health, ibm 2. com, writes, article, edu, space, medical, pitt, pain, blood, disease, doctor, data, treatment, patients, health 3. space, com, writes, article, edu, data, medical, launch, earth, states, blood, moon, disease, satellite, medicine, 4. space, data, com, writes, article, edu, launch, earth, states, moon, satellite, shuttle, nasa, price, lunar 5. space, data, launch, earth, states, moon, satellite, case, com, shuttle, price, nasa, price, lunar, writes, 6. space, data, launch, earth, states, moon, orbit, satellite, case, shuttle, price, nasa, system, lunar, spacecraft From a alt.atheism document to a soc.religion.christian one . god, just, want, moral, believe, religion, atheists, atheism, christian, make, atheist, good, say, bible, faith 2. god, just, want, believe, jesus, christian, atheists, bible, atheism faith, say, make, religious, christians, atheist 3. god, jesus, just, faith, believe, christian, bible, want, church, say, religion, moral, lord, world, writes 4. god, jesus, faith, just, bible, church, christ, believe, say, writes, lord, religion, world, want, sin 5. god, jesus, faith, church, christ, bible, christian, say, write, lord, believe, truth, world, human, holy 6. god, jesus, faith, church, christ, bible, writes, say, christian, lord, sin, human, father, spirit, truth From a com.graphics document to a comp.sys.ibm.pc.hardware one 1. image, color, windows, files, image, thanks, jpeg, gif, card, bit, window, win, help, colors, format 2. image, windows, color, files, card, images, jpeg, thanks, gif, bit, window, win, colors, monitor, program 3. windows, image, color, card, files, gov, writes, nasa, article, images, program, jpeg, vidio, display, monitor 4. windows, gov, writes, nasa, article, card, going, program, image, color, memory, files, software, know, screen 5. gov, windows, writes, nasa, article, going, dos, card, memory, know, display, says, screen, work, ram 6. gov, writes, nasa, windows, article, going, dos, program, card, memory, software, says, ram, work, running 

1

:

∈ R K l . This upward-downward inference network is distinct from that of a usual VAE, where it is common that the inference network has a pure bottom-up structure and only interacts with the generative model via theELBO (Kingma & Welling, 2014;Ishaan et al., 2017). Note that WUDVE no longer follows mean-field variational Bayes to make a fully factorized assumption as in (4).

Figures and Tables:

Figure1: The KL divergence from the inferred Weibull distribution to the target gamma one as (a) Gamma(0.05, 1), (b) Gamma(0.5, 1), and (c) Gamma(5, 1). Subplot (d) shows the KL divergence as a function of the gamma shape parameter, where the gamma scale parameter is fixed at 1.

Figure 2: (a-b): Inference (or encoder/recognition) and generative (or decoder) models for (a) WHAI and (b) AVITM; (c) the generative model and a sketch of the upward-downward Gibbs sampler of DLDA, where Z l are augmented latent counts that are upward sampled in each Gibbs sampling iteration. Circles are stochastic variables and squares are deterministic variables. The orange and blue arrows denote the upward and downward information propagation respectively, and the red ones denote the data generation.

Figure 3: Plot of per-heldout-word perplexity as a function of time for (a) 20News, (b) RCV1, and (c) Wiki. Except for AVITM that has a single hidden layer with 128 topics, all the other algorithms have the same network size of 128-64-32 for their deep generative models.

Figure 4: An example of hierarchical topics learned from Wiki by a three-hidden-layer WHAI of size 128-64-32.

Figure 5: Learned topics on MNIST digits with a three-hidden-layer WHAI of size 128-64-32. Shown in (a)-(c) are example topics for layers 1, 2 and 3, respectively, learned with a deterministic-upward-stochastic-downward encoder, and shown in (d)-(f) are the ones learned with a deterministic-upward encoder.

Figure 8: An example of hierarchical topics learned from Wiki by a four-hidden-layer WHAI of size 256-128-64-32.

Comparison of per-heldout-word perplexity and testing time (average seconds per document) on three different datasets.

Table content: <table><tr><td>Model</td><td>Size</td><td colspan="6">Perplexity 20News RCV1 Wiki 20News RCV1 Wiki Test Time</td></tr><tr><td>DLDA-Gibbs</td><td>128-64-32</td><td>571</td><td>938</td><td>966</td><td>10.46</td><td colspan="2">23.38 23.69</td></tr><tr><td>DLDA-Gibbs</td><td>128-64</td><td>573</td><td>942</td><td>968</td><td>8.73</td><td colspan="2">18.50 19.79</td></tr><tr><td>DLDA-Gibbs</td><td>128</td><td>584</td><td>951</td><td>981</td><td>4.69</td><td colspan="2">12.57 13.31</td></tr><tr><td>DLDA-TLASGR</td><td>128-64-32</td><td>579</td><td>950</td><td>978</td><td>10.46</td><td colspan="2">23.38 23.69</td></tr><tr><td>DLDA-TLASGR</td><td>128-64</td><td>581</td><td>955</td><td>979</td><td>8.73</td><td colspan="2">18.50 19.79</td></tr><tr><td>DLDA-TLASGR</td><td>128</td><td>590</td><td>963</td><td>993</td><td>4.69</td><td colspan="2">12.57 13.31</td></tr><tr><td>DPFA</td><td>128-64-32</td><td>637</td><td colspan="2">1041 1056</td><td>20.12</td><td colspan="2">34.21 35.41</td></tr><tr><td>AVITM</td><td>128</td><td>654</td><td colspan="2">1062 1088</td><td>0.23</td><td>0.68</td><td>0.80</td></tr><tr><td colspan="2">DLDA-GHAI-Independent 128-64-32</td><td>613</td><td>970</td><td>999</td><td>0.62</td><td>1.22</td><td>1.47</td></tr><tr><td>DLDA-GHAI-Independent</td><td>128-64</td><td>614</td><td>970</td><td>1000</td><td>0.41</td><td>0.94</td><td>1.01</td></tr><tr><td>DLDA-GHAI-Independent</td><td>128</td><td>615</td><td>972</td><td>1003</td><td>0.22</td><td>0.69</td><td>0.80</td></tr><tr><td>DLDA-GHAI</td><td>128-64-32</td><td>604</td><td>963</td><td>994</td><td>0.66</td><td>1.25</td><td>1.49</td></tr><tr><td>DLDA-GHAI</td><td>128-64</td><td>608</td><td>965</td><td>997</td><td>0.44</td><td>0.96</td><td>1.05</td></tr><tr><td>DLDA-GHAI</td><td>128</td><td>615</td><td>972</td><td>1003</td><td>0.22</td><td>0.69</td><td>0.80</td></tr><tr><td colspan="2">DLDA-WHAI-Independent 128-64-32</td><td>588</td><td>964</td><td>990</td><td>0.58</td><td>1.15</td><td>1.38</td></tr><tr><td>DLDA-WHAI-Independent</td><td>128-64</td><td>589</td><td>965</td><td>992</td><td>0.38</td><td>0.87</td><td>0.97</td></tr><tr><td>DLDA-WHAI-Independent</td><td>128</td><td>592</td><td>966</td><td>996</td><td>0.20</td><td>0.66</td><td>0.78</td></tr><tr><td>DLDA-WAI</td><td>128-64-32</td><td>581</td><td>954</td><td>984</td><td>0.63</td><td>1.20</td><td>1.43</td></tr><tr><td>DLDA-WAI</td><td>128-64</td><td>583</td><td>958</td><td>986</td><td>0.42</td><td>0.91</td><td>1.02</td></tr><tr><td>DLDA-WAI</td><td>128</td><td>593</td><td>967</td><td>999</td><td>0.20</td><td>0.66</td><td>0.78</td></tr><tr><td>DLDA-WHAI</td><td>128-64-32</td><td>581</td><td>953</td><td>980</td><td>0.63</td><td>1.20</td><td>1.43</td></tr><tr><td>DLDA-WHAI</td><td>128-64</td><td>582</td><td>957</td><td>982</td><td>0.42</td><td>0.91</td><td>1.02</td></tr><tr><td>DLDA-WHAI</td><td>128</td><td>591</td><td>965</td><td>996</td><td>0.20</td><td>0.66</td><td>0.78</td></tr></table>

in other words, we ignore the top-down information. As shown in Figs. 5d-5f, although some latent structures are learned, the hierarchical new

Table content: <table><tr><td>85</td><td/><td/><td>117</td><td/><td/><td>78</td><td>86</td><td/><td>125</td><td>62</td></tr><tr><td>international international</td><td/><td/><td>art art</td><td/><td/><td/><td>new</td><td/><td>man man</td><td>character character</td></tr><tr><td>world world</td><td/><td/><td>work work</td><td/><td/><td/><td>like like</td><td/><td>king king</td><td>characters characters</td></tr><tr><td>founded founded</td><td/><td/><td>artist artist</td><td/><td/><td/><td>fact fact</td><td/><td>death death</td><td>back back</td></tr><tr><td>community community</td><td/><td/><td>prize prize</td><td/><td/><td/><td>second second</td><td/><td>vol vol</td><td>like like</td></tr><tr><td>development development</td><td/><td/><td>artists artists</td><td/><td/><td/><td>idea idea</td><td/><td>god god</td><td>world world</td></tr><tr><td>national national</td><td/><td/><td>arts arts</td><td/><td/><td/><td>possible possible</td><td/><td>dead dead</td><td>takes takes</td></tr><tr><td>program program</td><td/><td/><td colspan="2">academy academy</td><td/><td/><td>states states</td><td/><td>great great</td><td>see see</td></tr><tr><td>foundation foundation</td><td/><td/><td>painter painter</td><td/><td/><td/><td colspan="2">influence influence</td><td>sun sun</td><td>appears appears</td></tr><tr><td>based based</td><td/><td/><td>style style</td><td/><td/><td/><td colspan="2">described described</td><td>lord lord</td><td>secret secret</td></tr><tr><td>organization organization</td><td/><td/><td>design design</td><td/><td/><td/><td>word word</td><td/><td>magic magic</td><td>girl girl</td></tr><tr><td>social social</td><td/><td/><td colspan="2">painting painting</td><td/><td/><td colspan="2">approach approach</td><td>power power</td><td>named named</td></tr><tr><td colspan="11">Figure 7: An example of hierarchical topics learned from Wiki by a three-hidden-layer WHAI of</td></tr><tr><td colspan="2">size 128-64-32.</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td>6</td><td/><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td colspan="2">war government state</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td colspan="2">first time new national</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td colspan="2">two law general united</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td colspan="3">5 government state national law public members act people union department united</td><td colspan="2">28 war army government military time battle general law forces two union</td><td/><td/><td/></tr><tr><td>19</td><td/><td>53</td><td/><td>125</td><td/><td>77</td><td>52</td><td/><td>50</td><td/></tr><tr><td colspan="2">business federal project act bank financial tax office private market report</td><td colspan="2">law court act rights legal state justice police judge case commission</td><td colspan="2">government union war president law party political minister office general rights</td><td>left days sent two battle came union continued forced attack lost</td><td colspan="2">killed days police men death found left attack dead sent said</td><td colspan="2">war army battle military forces attack troops force civil commander</td></tr><tr><td>144 Federal bank credit financial Exchange Tax billion costs paid Income workers</td><td>116 report agreement security stated reported plan reports new due plans signed</td><td>101 court legal Decision legal bill Practice murder arrested Criminal Court lawyer</td><td>142 Rights law state government human public constitution right political Civil laws</td><td>153 political movement religious people freedom politics peace press war led nation</td><td>102 government economic trade capital minister economy political governments ministry nations relations</td><td>176 territory treaty power independence land led colonial rule region government provinces</td><td>191 days found decided Occurred believed months turned saw incident taken leave</td><td>65 killed death men dead shot killing people escape murder Man kill</td><td>179 battle army fort soldiers killed fought troops command wounded captain siege</td><td>133 war military army civil general soldiers forces major wars camp front</td></tr></table>

Peer review (iclr2018):
 The paper presents a deep Poisson model where the last layer is the vector of word counts generated by a vector Poisson. This is parameterized by a matrix vector product, and the vector in this parameterizeation is itself generated by a vector Gamma with a matrix-vector parameterization. From there the vectors are all Gammas with matrix-vector parameterizations in a typical deep setup. While the model is reasonable, the purpose was not clear to me. If only the last layer generates a document, then what use is the deep structure? For example, learning hierarchical topics as in Figure 4 doesn*t seem so useful here since only the last layer matters. Also, since no input is being mapped to an output, what does going deeper mean? It doesn*t look like any linear mapping is being learned from the input to output spaces, so ultimately the document itself is coming from a simple linear Poisson model just like LDA and other non-deep methods. The experiments are otherwise thorough and convincing that quantitative performance is improved over previous attempts at the problem.