Title: DIRICHLET VARIATIONAL AUTOENCODER

Abstract: This paper proposes Dirichlet Variational Autoencoder (DirVAE) using a Dirichlet prior for a continuous latent variable that exhibits the characteristic of the categorical probabilities. To infer the parameters of DirVAE, we utilize the stochastic gradient method by approximating the Gamma distribution, which is a component of the Dirichlet distribution, with the inverse Gamma CDF approximation. Additionally, we reshape the component collapsing issue by investigating two problem sources, which are decoder weight collapsing and latent value collapsing, and we show that DirVAE has no component collapsing; while Gaussian VAE exhibits the decoder weight collapsing and Stick-Breaking VAE shows the latent value collapsing. The experimental results show that 1) DirVAE models the latent representation result with the best log-likelihood compared to the baselines; and 2) DirVAE produces more interpretable latent values with no collapsing issues which the baseline models suffer from. Also, we show that the learned latent representation from the DirVAE achieves the best classification accuracy in the semi-supervised and the supervised classification tasks on MNIST, OMNIGLOT, and SVHN compared to the baseline VAEs. Finally, we demonstrated that the DirVAE augmented topic models show better performances in most cases.

1. INTRODUCTION:

A Variational Autoencoder (VAE) (Kingma & Welling, 2014c) brought success in deep generative models (DGMs) with a Gaussian distribution as a prior distribution (Jiang et al., 2017; Miao et al., 2016; 2017; Srivastava & Sutton, 2017 ). If we focus on the VAE, the VAE assumes the prior distribution to be N (0, I) with the learning on the approximated μ and Σ. Also, Stick-Breaking VAE (SBVAE) (Nalisnick & Smyth, 2017) is a nonparametric version of the VAE, which modeled the latent dimension to be infinite using a stick-breaking process (Ishwaran & James, 2001) .

While these VAEs assume that the prior distribution of the latent variables to be continuous random variables, recent studies introduce the approximations on discrete priors with continuous random variables (Jang et al., 2017; Maddison et al., 2017; Rolfe, 2017) . The key of these approximations is enabling the backpropagation with the reparametrization technique, or the stochastic gradient variational Bayes (SGVB) estimator, while the modeled prior follows a discrete distribution. The applications of these approximations on discrete priors include the prior modeling of a multinomial distribution which is frequently used in the probabilistic graphical models (PGMs). Inherently, the multinomial distributions can take a Dirichlet distribution as a conjugate prior, and the demands on such prior have motivated the works like Jang et al. (2017) ; Maddison et al. (2017) ; Rolfe (2017) that support the multinomial distribution posterior without explicit modeling on a Dirichlet prior.

When we survey the work with a explicit modeling on the Dirichlet prior, we found a frequent approach such as utilizing a softmax Laplace approximation (Srivastava & Sutton, 2017) . We argue that this approach has a limitation from the multi-modality perspective. The Dirichlet distribution can exhibit a multi-modal distribution with parameter settings, see Figure 1 , which is infeasible to generate with the Gaussian distribution with a softmax function. Therefore, the previous continuous domain VAEs cannot be a perfect substitute for the direct approximation on the Dirichlet distribution.

Utilizing a Dirichlet distribution as a conjugate prior to a multinomial distribution has an advantage compared to the usage of a softmax function on a Gaussian distribution. For instance, Figure 1 illustrates the potential difficulties in utilizing the softmax function with the Gaussian distribution. Unlike the Gaussian-Softmax or the GEM distribution, the Dirichlet distribution is able to capture the multi-modality that illustrates multiple peaks at the vertices of the probability simplex.

Given the three-dimensional probability simplex, the Gaussian-Softmax distribution cannot generate the illustrated case of the Dirichlet distribution with a high probability measure at the vertices of the simplex, i.e. the multi-modality where the necessity was emphasized in Hoffman & Johnson (2016) . Additionally, the Griffiths-Engen-McCloskey (GEM) distribution (Pitman, 2002) , which is the prior distribution of the SBVAE, is difficult to model the multi-modality because the sampling procedure of the GEM distribution is affected by the rich-get-richer phenomenon, so a few components tend to dominate the weight of the samples. This is different from the Dirichlet distribution that does not exhibit such phenomenon, and the Dirichlet distribution can fairly distribute the weights to the components, and the Dirichlet distribution is more likely to capture the multi-modality by controlling the prior hyper-parameter (Blei et al., 2003) . Then, we conjecture that an enhanced modeling on Dirichlet prior is still needed 1) because there are cases that the Gaussian-Softmax approaches, or the softmax Laplace approximation, cannot imitate the Dirichlet distribution; and 2) because the nonparametric approaches could be influenced by the biases that the Dirichlet distribution does not suffer from.

Given these motivations for modeling the Dirichlet distribution with the SGVB estimator, this paper introduces the Dirichlet Variational Autoencoder (DirVAE) that shows the same characteristics of the Dirichlet distribution. The DirVAE is able to model the multi-modal distribution that was not possible with the Gaussian-Softmax and the GEM approaches. These characteristics allow the DirVAE to be the prior of the discrete latent distribution, as the original Dirichlet distribution is.

Introducing the DirVAE requires the configuration of the SGVB estimator on the Dirichlet distribution. Specifically, the Dirichlet distribution is a composition of the Gamma random variables, so we approximate the inverse Gamma cumulative distribution function (CDF) with the asymptotic approximation. This approximation on the inverse Gamma CDF becomes the component of approximating the Dirichlet distribution. We compared this approach to the previously suggested approximations, i.e. approaches with the Weibull distribution and with the softmax Gaussian distribution, and our approximation shows the best log-likelihood among the compared approximations.

Moreover, we report that we had to investigate the component collapsing along with the research on DirVAE. It has been known that the component collapsing issue is resolved by the SBVAE because of the meaningful decoder weights from the latent layer to the next layer. However, we found that SBVAE has latent value collapsing issue resulting in many near-zero values on the latent dimensions that leads to the incomplete utilization of the latent dimension. Hence, we argue that Gaussian VAE (GVAE) suffers from the decoder weight collapsing, previously limitedly defined as component collapsing; and SBVAE has a problem of the latent value collapsing. Finally, we suggest that the definition of component collapsing should be expanded to represent both cases of decoder weight and latent value collapsings. The proposed DirVAE shows neither the near-zero decoder weights nor the near-zero latent values, so the reconstruction uses the full latent dimension information in most cases. We investigated this issue because our performance gain comes from resolving the expanded version of the component collapsing. Due to the component collapsing issues, the existing VAEs have less meaningful latent values or could not effectively use its latent representation. Meanwhile, DirVAE does not have component collapsing due to the multi-modal prior which possibly leads to superior qualitative and quantitative performances. We experimentally showed that the DirVAE has more meaningful or disentangled latent representation by image generation and latent value visualizations.

Technically, the new approximation provides the closed-form loss function derived from the evidence lower bound (ELBO) of the DirVAE. The optimization on the ELBO enables the representation learning with the DirVAE, and we test the learned representation from the DirVAE in two folds. Firstly, we test the representation learning quality by performing the supervised and the semi-supervised classification tasks on MNIST, OMNIGLOT, and SVHN. These classification tasks conclude that DirVAE has the best classification performances with its learned representation. Secondly, we test the applicability of DirVAE to the existing models, such as topic models with DirVAE priors on 20Newsgroup and RCV1-v2. This experiment shows that the augmentation of DirVAE to the existing neural variational topic models improves the perplexity and the topic coherence, and most of best performers were DirVAE augmented.

2. PRELIMINARIES 2.1 VARIATIONAL AUTOENCODERS:

A VAE is composed of two parts: a generative sub-model and an inference sub-model. In the generative part, a probabilistic decoder reproduces x close to an observation x from a latent variable z ∼ p(z), i.e. x ∼ p θ (x|z) = p θ (x|ζ) where ζ = MLP(z) is obtained from a latent variable z by a multilayer perceptron (MLP). In the inference part, a probabilistic encoder outputs a latent variable z ∼ q φ (z|x) = q φ (z|η) where η = MLP(x) is computed from the observation x by a MLP. Model parameters, θ and φ, are jointly learned by optimizing the below ELBO with the stochastic gradient method through the backpropagations as the ordinary neural networks by using the SGVB estimators on the random nodes.

EQUATION

In GVAE (Kingma & Welling, 2014c) , the prior distribution of p(z) is assumed to be a standard Gaussian distribution. In SBVAE (Nalisnick & Smyth, 2017) , the prior distribution becomes a GEM distribution that produces samples with a Beta distribution and a stick-breaking algorithm.

2.2. DIRICHLET DISTRIBUTION AS A COMPOSITION OF GAMMA RANDOM VARIABLES:

The Dirichlet distribution is a composition of multiple Gamma random variables. Note that the probability density functions (PDFs) of Dirichlet and Gamma distributions are as follows:

EQUATION

where α k , α, β > 0. In detail, if there are K independent random variables following the Gamma distributions

X k ∼ Gamma(α k , β) or X ∼ MultiGamma(α, β • 1 K ) where α k , β > 0 for k = 1, • • • , K, then we have Y ∼ Dirichlet(α) where Y k = X k / X i .

It should be noted that the rate parameter, β, should be the same for every Gamma distribution in the composition. Then, the KL divergence can be derived as the following:

KL(Q||P ) = log Γ(α k ) -log Γ(α k ) + (α k -α k )ψ(α k ) (3) for P = MultiGamma(α, β •1 K ) and Q = MultiGamma( α, β •1 K ) where ψ is a digamma function.

The detailed derivation is provided in Appendix B.

2.3. SGVB FOR GAMMA RANDOM VARIABLE AND APPROXIMATION ON DIRICHLET:



DISTRIBUTION:

This section discusses several ways of approximating the Dirichlet random variable; or the SGVB estimators for the Gamma random variables which compose a Dirichlet distribution. Utilizing SGVB requires a differentiable non-centered parametrization (DNCP) for the distribution (Kingma & Welling, 2014d) . The main SGVB for Gamma random variables, used in DirVAE, is using the inverse Gamma CDF approximation explained in the next section. Prior works include two approaches: the use of the Weibull distribution and the softmax Gaussian distribution, and the two approaches are explained in this section.

Approximation with Weibull distribution. Because of the similar PDFs between the Weibull distribution and the Gamma distribution, some prior works used the Weibull distribution as a posterior distribution of the prior Gamma distribution (Zhang et al., 2018) :

Weibull(x; k, λ) = k λ x λ k-1 e -(x/λ) k

where k, λ > 0 .

(4)

The paper Zhang et al. (2018) pointed out that there are two useful characteristics when approximating the Gamma distribution with the Weibull distribution. One useful property is that the KL divergence expressed in a closed form, and the other is the simple reparametrization trick with a closed form of the inverse CDF from the Weibull distribution. However, we noticed that the Weibull distribution has a component of e -(x/λ) k , and the Gamma distribution does not have the additional power term of k in the component. Since k is placed in the exponential component, small changes on k can cause a significant difference that limits the optimization.

Approximation with softmax Gaussian distribution. As in MacKay (1998); Srivastava & Sutton (2017) , a Dirichlet distribution can be approximated by a softmax Gaussian distribution by using a softmax Laplace approximation. The relation between the Dirichlet parameter α and the Gaussian parameters µ, Σ is explained as the following:

EQUATION

)

where Σ is assumed to be a diagonal matrix, and we use the reparametrization trick in the usual GVAE for the SGVB estimator.

3. MODEL DESCRIPTION:

Along with the inverse Gamma CDF approximation, we describe two sub-models in this section: the generative sub-model and the inference sub-model. Figure 2 describes the graphical notations of various VAEs and the neural network view of our model. Generative sub-model. The key difference between the generative models between the DirVAE and the GVAE is the prior distribution assumption on the latent variable z. Instead of using the standard Gaussian distribution, we use the Dirichlet distribution which is a conjugate prior distribution of the multinomial distribution.

EQUATION

Inference sub-model. The probabilistic encoder with an approximating posterior distribution q φ (z|x) is designed to be Dirichlet( α). The approximated posterior parameter α is derived by the MLP from the observation x with the softplus output function, so the outputs can be positive values constrained by the Dirichlet distribution. Here, we do not directly sample z from the Dirichlet distribution. Instead, we use the Gamma composition method described in Section 2.2. Firstly, we draw v ∼ MultiGamma(α, β • 1 K ). Afterwards, we normalize v with its summation v i .

The objective function to optimize the model parameters, θ and φ, is composed of Equation ( 1) and

(3). Equation ( 7) is the loss function to optimize after the composition. The inverse Gamma CDF method explained in the next paragraph enables the backpropagation flows to the input with the stochastic gradient method. Here, for the fair comparison of expressing the Dirichlet distribution between the inverse Gamma CDF approximation method and the softmax Gaussian method, we set α k = 1 -1/K when µ k = 0 and Σ k = 1 by using Equation ( 5); and β = 1.

EQUATION

Approximation with inverse Gamma CDF. A previous work Knowles (2015) suggested that, if X ∼ Gamma(α, β), and if F (x; α, β) is a CDF of the random variable X, the inverse CDF can be approximated as

F -1 (u; α, β) ≈ β -1 (uαΓ(α)) 1/α

. Hence, we can introduce an auxiliary variable u ∼ Uniform(0, 1) to take over all the randomness of X, and we treat the Gamma sampled X as a deterministic value in terms of α and β.

It should be noted that there has been a practice of utilizing the combination of decomposing a Dirichlet distribution and approximating each Gamma component with inverse Gamma CDF. However, such practices have not been examined with its learning properties and applicabilities. The following section shows a new aspect of component collapsing that can be remedied by this combination on Dirichlet prior in VAE, and the section illustrates the performance gains in a certain set of applications, i.e. topic modeling.

4. EXPERIMENTAL RESULTS:

This section reports the experimental results with the following experiment settings: 1) a pure VAE model; 2) a semi-supervised classification task with VAEs; 3) a supervised classification task with VAEs; and 4) topic models with DirVAE augmentations.

4.1. EXPERIMENTS FOR REPRESENTATION LEARNING OF VAES:

Baseline models. We select the following models as baseline alternatives of the DirVAE: 1) the standard GVAE; 2) the GVAE with softmax (GVAE-Softmax) approximating the Dirichlet distribution with the softmax Gaussian distribution; 3) the SBVAE with the Kumaraswamy distribution (SBVAE-Kuma) & the Gamma composition (SBVAE-Gamma) described in Nalisnick & Smyth (2017) ; and 4) the DirVAE with the Weibull distribution (DirVAE-Weibull) approximating the Gamma distribution with the Weibull distribution described in Zhang et al. (2018) . We use the following benchmark datasets for the experiments: 1) MNIST; 2) MNIST with rotations (MNIST+rot); 3) OMNIGLOT; and 4) SVHN with PCA transformation. We provide the details on the datasets in Appendix D.1.

Experimental setting. As a pure VAE model, we compare the DirVAE with the following models: GVAE, GVAE-Softmax, SBVAE-Kuma, SBVAE-Gamma, and DirVAE-Weibull. We use 50dimension and 100-dimension latent variables for MNIST and OMNIGLOT, respectively. We provide the details of the network structure and optimization in Appendix D.2. We set α = 0.98 • 1 50 for MNIST and α = 0.99 • 1 100 for OMNIGLOT for the fair comparison to GVAEs by using Equation ( 5). All experiments use the Adam optimizer (Kingma & Ba, 2014a) for the parameter learning. Finally, we acknowledge that the hyper-parameter could be updated as Appendix C, and the experiment result with the update is separately reported in Appendix D.2.

Quantitative result. For the quantitative comparison among the VAEs, we calculated the Monte-Carlo estimation on the marginal negative log-likelihood, the negative ELBO, and the reconstruction loss. The marginal log-likelihood is approximated as p(x) ≈ i p(x|zi)p(zi) q(zi)

for single instance x where q(z) is a posterior distribution of a prior distribution p(z), which is further derived in Appendix A. Table 1 shows the overall performance of the alternative VAEs. The DirVAE outperforms all baselines in both datasets from the log-likelihood perspective. The value of DirVAE comes from the better encoding of the latent variables that can be used for classification tasks which we examine in the next experiments. While the DirVAE-Weibull follows the prior modeling with the Dirichlet distribution, the Weibull based approximation can be improved by adopting the proposed approach with the inverse Gamma CDF. Qualitative result. As a qualitative result, we report the latent dimension-wise reconstructions which are decoder outputs with each one-hot vector in the latent dimension. Figure 3a shows 50 reconstructed images corresponding to each latent dimension from GVAE-Softmax, SBVAE, and DirVAE. We manually ordered the digit-like figures in the ascending order for GVAE-Softmax and DirVAE. We can see that the GVAE-Softmax and the SBVAE have components without significant semantic information, which we will discuss further in Section 4.2, and the DirVAE has interpretable latent dimensions in most of the latent dimensions. Figure 3b also supports the quality of the latent values from DirVAE by visualizing learned latent values through t-SNE (Maaten & Hinton, 2008) . Decoder weight collapsing, a.k.a. component collapsing. One main issue of GVAE is component collapsing that there are a significant number of near-zero decoder weights from the latent neurons to the next decoder neurons. If these weights become near-zero, the values of the latent dimensions loose influence to the next decoder, and this means an inefficient learning given a neural network structure. The same issue occurs when we use the GVAE-Softmax. We rename this component collapsing phenomenon as decoder weight collapsing to specifically address the collapsing source.

Latent value collapsing. SBVAE claims that SBVAE solved the decoder weight collapsing by learning the meaningful weights as shown in Figure 4a . However, we notice that SBVAE produces the output values, not the weight parameters, from the latent dimension to be near-zero in many latent dimensions after averaging many samples obtained from the test dataset. Figure 4b shows the properties of DirVAE and SBVAE from the perspective of the latent value collapsing, which SBVAE shows many near-zero average means and near-zero average variances, while DirVAE does not. The average Fisher kurtosis and average skewness of DirVAE are 5.76 and 2.03, respectively over the dataset, while SBVAE has 20.85 and 4.35, which states that the latent output distribution from SBVAE is more skewed than that of DirVAE. We found out that these near-zero latent values prevent learning on decoder weights, which we introduce as another type of collapsing problem, as latent value collapsing that is different from the decoder weight collapsing. These results mean that SBVAE distributes the non-near-zero latent values sparsely over a few dimensions while DirVAE samples relatively dense latent values. In other words, DirVAE utilizes the full spectrum of latent dimensions compared to SBVAE, and DirVAE has a better learning capability in the decoder network. Figure 3a supports the argument on the latent value collapsing by activating each and single latent dimension with a one-hot vector through the decoder. The non-changing latent dimension- Figure 4 : Sub-figure 4a shows GVAE and GVAE-Softmax have component collapsing issue, while SBVAE and DirVAE do not. Sub-figure 4b shows that SBVAE has many near-zero output values in the latent dimensions.

wise images of SBVAE proves that there were no generation differences between the two differently activated one-hot latent values.

4.3. APPLICATION 1. EXPERIMENTS OF (SEMI-)SUPERVISED CLASSIFICATION WITH VAES:

Semi-supervised classification task with VAEs. There is a previous work demonstrating that the SBVAE outperforms the GVAE in semi-supervised classification task (Nalisnick & Smyth, 2017 ). The overall model structure for this semi-supervised classification task uses a VAE with separate random variables of z and y, which is introduced as the M2 model in the original VAE work (Kingma et al., 2014b) . The detailed settings of the semi-supervised classification tasks are enumerated in Appendix D.3. Fundamentally, we applied the same experimental settings to GVAE, SBVAE, and DirVAE in this experiment, as specified by the authors in Nalisnick & Smyth (2017) .

Table 2 enumerates the performances of the GVAE, the SBVAE, and the DirVAE, and the result shows that the error rate of classification result using 10%, 5% and 1% of labeled data for each dataset. In general, the experiment shows that the DirVAE has the best performance out of three alternative VAEs. Also, it should be noted that the performance of the DirVAE is more improved in the most complex task with the SVHN dataset. Supervised classification task with latent values of VAEs. Also, we tested the performance of the supervised classification task with the learned latent representation from the VAEs. We applied the vanilla version of VAEs to the datasets, and we classified the latent representation of instances with k-Nearest Neighbor (kNN) which is one of the simplest classification algorithms. Hence, this experiment can better distinguish the performance of the representation learning in the classification task. Further experimental details can be found in Appendix D.4.

Table 3 enumerates the performances from the experimented VAEs in the datasets of MNIST and OMNIGLOT. Both datasets indicated that the DirVAE shows the best performance in reducing the classification error, which we conjecture that the performance is gathered from the better representation learning. It should be noted that, to our knowledge, this is the first reported comparison of latent representation learning on VAEs with kNN in the supervised classification using OMNIGLOT dataset. We identified that the classification with OMNIGLOT is difficult given that the kNN error rates with the raw original data are as high as 69.94%, 69.41%, and 70.10%. This high error rate mainly originates from the number of classification categories which is 50 categories in our test setting of OMNIGLOT, compared to 10 categories in MNIST.

Table 3 : The error rate of kNN with the latent representations of VAEs.

MNIST (K = 50) OMNIGLOT (K = 100) One usefulness of the Dirichlet distribution is being a conjugate prior to the multinomial distribution, so it has been widely used in the field of topic modeling, such as Latent Dirichlet Allocation (LDA) (Blei et al., 2003) . Recently, some neural variational topic (or document) models have been suggested, for example, ProdLDA (Srivastava & Sutton, 2017) , NVDM (Miao et al., 2016) , and GSM (Miao et al., 2017) . NVDM used the GVAE, and the GSM used the GVAE-Softmax to make the sum-to-one positive topic vectors. Meanwhile, ProdLDA assume the prior distribution to be the Dirichlet distribution with the softmax Laplace approximation. To verify the usefulness of the DirVAE, we replace the probabilistic encoder part of the DirVAE to each model. Two popular performance measures in the topic model fields, which are perplexity and topic coherence via normalized pointwise mutual information (NPMI) (Lau et al., 2014) , have been used with 20Newsgroups and RCV1-v2 datasets. Further details of the experiments can be found in Appendix D.5. Table 4 indicates that the augmentation of DirVAE improves the performance in general. Additionally, the best performers from the two measurements are always the experiment cell with DirVAE augmentation except for the perplexity of RCV1-v2, which still remains competent. 

k = 3 k = 5 k = 10 k = 3 k = 5 k = 10 GVAE (

5. CONCLUSION:

Recent advances in VAEs have become one of the cornerstones in the field of DGMs. The VAEs infer the parameters of explicitly described latent variables, so the VAEs are easily included in the conventional PGMs. While this merit has motivated the diverse cases of merging the VAEs to the graphical models, we ask the fundamental quality of utilizing the GVAE where many models have latent values to be categorical probabilities. The softmax function cannot reproduce the multi-modal distribution that the Dirichlet distribution can. Recognizing this problem, there have been some previous works that approximated the Dirichlet distribution in the VAE settings by utilizing the Weibull distribution or the softmax Gaussian distribution, but the DirVAE with the inverse Gamma CDF shows the better learning performance in our experiments of the representation: the semi-supervised, the supervised classifications, and the topic models. Moreover, DirVAE shows no component collapsing and it leads to better latent representation and performance gain. The proposed DirVAE can be widely used if we recall the popularity of the conjugate relation between the multinomial and the Dirichlet distributions because the proposed DirVAE can be a brick to the construction of complex probabilistic models with neural networks.

APPENDIX:

This is an appendix for Dirichlet Variational Autoencoder. Here, we describe the derivations of key equations and experimental setting details which were used in the body of the paper. The detailed information such as model names, parameter names, or experiment assumptions is based on the main paper.

A MONTE-CARLO ESTIMATION ON THE MARGINAL LIKELIHOOD:

Proposition A.1. The marginal log-likelihood is approximated as p(x) ≈ i p(x|zi)p(zi) q(zi)

, where q(z) is a posterior distribution of a prior distribution p(z).

Proof.

p(x) = z p(x, z)dz = z p(x, z) q(z) q(z) dz = z p(x|z)p(z) q(z) q(z) dz = z p(x|z)p(z) q(z) q(z)dz ≈ i p(x|z i )p(z i ) q(z i ) where z i ∼ q(z) B KL DIVERGENCE OF TWO MULTI-GAMMA DISTRIBUTIONS Proposition B.1. Define X = (X 1 , • • • , X K ) ∼ MultiGamma(α, β • 1 K ) as a vector of K in- dependent Gamma random variables X k ∼ Gamma(α k , β) where α k , β > 0 for k = 1, • • • , K.

The KL divergence between two MultiGamma distributions P = MultiGamma(α, β • 1 K ) and Q = MultiGamma( α, β • 1 K ) can be derived as the following:

EQUATION

)

where ψ is a digamma function.

Proof. Note that the derivative of a Gamma-like function Γ(α) β α can be derived as follows:

d dα Γ(α) β α = β -α (Γ (α) -Γ(α) log β) = ∞ 0

x α-1 e -βx log x dx .

Then, we have the following.

KL(Q||P ) = D q(x) log q(x) p(x) dx = ∞ 0 • • • ∞ 0 Gamma(α k , β) log β αk Γ -1 ( αk )e -β x k x αk -1 k β α k Γ -1 (α k )e -β x k x α k -1 k dx = ∞ 0 • • • ∞ 0 Gamma(α k , β) × (α k -α k ) log β + log Γ(α k ) - log Γ(α k ) + (α k -α k ) log x k dx = (α k -α k ) log β + log Γ(α k ) - log Γ(α k ) + ∞ 0 • • • ∞ 0 β αk Γ(α k ) e -β x k x αk -1 k (α k -α k ) log x k dx = (α k -α k ) log β + log Γ(α k ) - log Γ(α k ) + (α k -α k )β αk Γ -1 (α k )β -αk Γ (α k ) -Γ(α k ) log β = (α k -α k ) log β + log Γ(α k ) - log Γ(α k ) + (α k -α k )(ψ(α k ) -log β) = log Γ(α k ) - log Γ(α k ) + (α k -α k )ψ(α k ) C HYPER-PARAMETER α LEARNING STRATEGY

In this section, we introduce the method of moment estimator (MME) to update the Dirichlet prior parameter α. Suppose we have a set of sum-to-one proportions D = {p 1 , • • • , p N } sampled from Dirichlet(α), then the MME update rule is as the following:

EQUATION

After the burn-in period for stabilizing the neural network parameters, we use the MME for the hyper-parameter learning using the sampled latent values during training. We alternatively update the neural network parameters and hyper-parameter α. We choose this estimator because of its closed form nature and consistency (Minka, 2000) . The usefulness of the hyper-parameter update can be found in Appendix D.2.

Proposition C.1. Given a proportion set D = {p 1 , • • • , p N } sampled from Dirichlet(α), MME of the hyper-parameter α is as the following:

α k ← S N n p n,k where S = 1 K k μ1,k -μ2,k μ2,k -μ2 1,k for μj,k = 1 N n p j n,k .

Proof. Define µ j,k = E[p j k ] as the j th moment of the k th dimension of Dirichlet distribution with prior α. Then, by the law of large number, µ j,k ≈ μj,k . It can be easily shown that µ MNIST (K = 50) OMNIGLOT (K = 100) 

1,k = α k i αi and µ 2,k = α k i αi 1+α k 1+ i αi = µ 1,k 1+α k 1+ i αi so that numerator µ 1,k -µ 2,k µ 2,k -µ 2 1,k = α k i α i - α k i α i 1 + α k 1 + i α i = α k ( i =k α i ) ( i α i )(1 + i α i ) denominator µ 1,k -µ 2,k µ 2,k -µ 2 1,k = α k i α i 1 + α k 1 + i α i - α k i α i 2 = α k ( i =k α i ) ( i α i ) 2 (1 + i α i ) holds for each k = 1, • • • , K. Therefore, i α i = µ 1,k -µ 2,k µ 2,k -µ 2 1,k ≈ 1 K k µ 1,k -µ 2,k µ 2,k -µ 2 1,k ≈ 1 K k μ1,k -μ2,k μ2,k -μ2 1,k and hence, αk = ( i α i )μ 1,k = S N n p n,k .

k = 3 k = 5 k = 10 k = 3 k = 5 k = 10 GVAE (

D.5 TOPIC MODEL AUGMENTATION WITH DIRVAE:

For the topic model augmentation experiment, two popular performance measures in the topic model fields, which are perplexity and topic coherence via normalized pointwise mutual information (NPMI) (Lau et al., 2014) , have been used with 20Newsgroups6 and RCV1-v27 datasets. 20Newsgroups has 11, 258 train data and 7, 487 test data with vocabulary size 1, 995. For the RCV1-v2 dataset, due to the massive size of the whole data, we randomly sampled 20, 000 train data and 10, 000 test data with vocabulary size 10, 000. The lower is better for the perplexity, and the higher is better for the NPMI.

The specific model structures can be found in the original papers, Srivastava & Sutton (2017); Miao et al. (2016; 2017) . We replace the model prior to that of DirVAE to each model and search the hyper-parameter as Table 7 with 1, 000 randomly selected test data. We use 500-dimension hidden layers and 50 topics for 20Newsgroups, and 1, 000-dimension hidden layers and 100 topics for RCV1-v2.

Table 8 shows top-10 high probability words per topic by activating single latent dimensions in the case of 20Newsgroups. Also, we visualized the latent embeddings of documents by t-SNE in Figure 7 ,8, and 9. 

:

http://www.iro.umontreal.ca/ lisa/twiki/bin/view.cgi/Public/MnistVariations

https://github.com/yburda/iwae/tree/master/datasets/OMNIGLOT

http://ufldl.stanford.edu/housenumbers/

https://github.com/enalisnick/stick-breaking dgms

https://www.ics.uci.edu/∼enalisni/sb dgm supp mat.pdf

https://github.com/akashgit/autoencoding vi for topic models

http://scikit-learn.org/stable/datasets/rcv1.html

Figures and Tables:

Figure 1: Illustrated probability simplex with Gaussian-Softmax, GEM, and Dirichlet distributions.Unlike the Gaussian-Softmax or the GEM distribution, the Dirichlet distribution is able to capture the multi-modality that illustrates multiple peaks at the vertices of the probability simplex.

Figure 2: Sub-figures 2a, 2b, and 2c are the graphical notations of the VAEs as latent variable models. The solid lines indicate the generative sub-models where the waved lines denote a prior distribution of the latent variables. The dotted lines indicate the inference sub-models. Sub-figure 2d denotes a neural network structure corresponding to Sub-figure 2c. Red nodes denote the random nodes which allow the backpropagation flows to the input.

(a) Latent dimension-wise reconstructions of GVAE-Softmax, SBVAE, and DirVAE. The DirVAE shows more meaningful latent dimensions than other VAEs. (b) (Left) GVAE, (Middle) SBVAE, (Right) DirVAE.

Figure 3: Latent dimension visualization with reconstruction images and t-SNE latent embeddings.

Figure 6: The optimized dimension-wise α values from DirVAE-Learning with MNIST.

genocide armenians turkish militia massacre village turks Topic 2 arab arabs israeli palestinian jews soldier turks nazi massacre jew Topic 3 resurrection bible christianity doctrine scripture eternal belief christian faith jesus Topic 4 hitter season braves pitcher baseball pitch game player defensive team Topic 5 directory file compile variable update ftp version site copy host Topic 6 performance speed faster mhz rate clock processor average twice fast Topic 7 windows microsoft driver dos nt graphic vga card virtual upgrade Topic 8 seat gear rear tire honda oil front mile wheel engine Topic 9 patient disease doctor treatment symptom medical health hospital pain medicine Topic 10 pt la det tor pit pp vs van cal nj (b) DirVAE augmentation to NVDM GSM+DirVAE Topic 1 turkish armenian armenians people one turkey armenia turks greek history Topic 2 israel israeli jews attack world jewish article arab peace land Topic 3 god jesus christian religion truth believe bible church christ belief Topic 4 team play game hockey nhl score first division go win Topic 5 drive video mac card port pc system modem memory speed Topic 6 image software file version server program system ftp package support Topic 7 space launch orbit earth nasa moon satellite mission project center Topic 8 law state gun government right rights case court police crime Topic 9 price sell new sale offer pay buy good condition money Topic 10 internet mail computer send list fax phone email address information (c) DirVAE augmentation to GSM

Figure 7: 20Newsgroups latent document embedding visulaization with t-SNE by replacing the model prior to the Dirichlet. (Left) ProdLDA+DirVAE, (Middle) NVDM+DirVAE, (Right) GSM+DirVAE.

Figure 8: 20Newsgroups latent document embedding visulaization with t-SNE by replacing the model prior to the Stick-Breaking. (Left) ProdLDA+SBVAE, (Middle) NVDM+SBVAE, (Right) GSM+SBVAE.

Figure 9: 20Newsgroups latent document embedding visulaization with t-SNE of original models. (Left) ProdLDA, (Middle) NVDM, (Right) GSM.

Negative log-likelihood, negative ELBO, and reconstruction loss of the VAEs for MNIST and OMNIGLOT dataset. The lower values are the better for all measures.

Table content: <table><tr><td/><td/><td>MNIST (K = 50)</td><td/><td colspan="3">OMNIGLOT (K = 100)</td></tr><tr><td/><td>Neg. LL</td><td colspan="2">Neg. ELBO Reconst. Loss</td><td>Neg. LL</td><td>Neg. ELBO</td><td>Reconst. Loss</td></tr><tr><td>GVAE (Nalisnick &amp; Smyth, 2017)</td><td>96.80</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>SBVAE-Kuma (Nalisnick &amp; Smyth, 2017)</td><td>98.01</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>SBVAE-Gamma (Nalisnick &amp; Smyth, 2017)</td><td>100.74</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>GVAE</td><td>94.54 ±0.79</td><td>98.58 ±0.04</td><td>74.31 ±0.13</td><td>119.29 ±0.44</td><td>126.42 ±0.24</td><td>98.90 ±0.36</td></tr><tr><td>GVAE-Softmax</td><td>98.18 ±0.61</td><td>103.49 ±0.16</td><td>79.36 ±0.82</td><td>130.01 ±1.16</td><td>139.73 ±0.81</td><td>123.34 ±1.43</td></tr><tr><td>SBVAE-Kuma</td><td>99.27 ±0.48</td><td>102.60 ±1.81</td><td>83.90 ±0.82</td><td>130.73 ±2.17</td><td>132.86 ±3.03</td><td>119.25 ±1.00</td></tr><tr><td>SBVAE-Gamma</td><td colspan="2">102.14 ±0.69 135.30 ±0.24</td><td>113.89 ±0.25</td><td>128.82 ±1.82</td><td>149.30 ±0.82</td><td>136.36 ±1.53</td></tr><tr><td>DirVAE-Weibull</td><td colspan="2">114.59 ±11.15 183.33 ±2.96</td><td>150.92 ±3.70</td><td>140.89 ±3.21</td><td>198.01 ±2.46</td><td>145.52 ±3.13</td></tr><tr><td>DirVAE</td><td colspan="2">87.64 ±0.64 100.47 ±0.35</td><td>81.50 ±0.27</td><td colspan="2">108.24 ±0.42 120.06 ±0.35</td><td>99.78 ±0.36</td></tr></table>

The error rate of semi-supervised classification task using VAEs.

Table content: <table><tr><td/><td colspan="3">MNIST (K = 50)</td><td colspan="3">MNIST+rot (K = 50)</td><td/><td>SVHN (K = 50)</td></tr><tr><td/><td>10%</td><td>5%</td><td>1%</td><td>10%</td><td>5%</td><td>1%</td><td>10%</td><td>5%</td><td>1%</td></tr><tr><td>GVAE (Nalisnick &amp; Smyth, 2017)</td><td colspan="4">3.95±0.15 4.74±0.43 11.55±2.28 21.78±0.73</td><td>27.72±0.69</td><td>38.13±0.95</td><td>36.08±1.49</td><td>48.75±1.47</td><td>69.58±1.64</td></tr><tr><td colspan="2">SBVAE (Nalisnick &amp; Smyth, 2017) 4.86±0.14</td><td>5.29±0.39</td><td>7.34±0.47</td><td>11.78±0.39</td><td>14.27±0.58</td><td>27.67±1.39</td><td>32.08±4.00</td><td>37.07±5.22</td><td>61.37±3.60</td></tr><tr><td>DirVAE</td><td>4.60±0.07</td><td colspan="8">5.05±0.18 7.00±0.17 11.18±0.32 13.53±0.46 26.20±0.66 24.81±1.13 28.45±1.14 55.99±3.30</td></tr></table>

Topic modeling performances of perpexity and NPMI with DirVAE augmentations.

Table content: <table><tr><td/><td/><td/><td colspan="2">20Newsgroups (K = 50)</td><td/><td/><td colspan="2">RCV1-v2 (K = 100)</td></tr><tr><td/><td/><td>ProdLDA</td><td>NVDM</td><td>GSM</td><td>LDA (Gibbs)</td><td>ProdLDA</td><td>NVDM</td><td>GSM</td><td>LDA (Gibbs)</td></tr><tr><td/><td>Reported</td><td>1172</td><td>837</td><td>822</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td/><td>Reproduced</td><td>1219±8.87</td><td>810±2.60</td><td>954±1.22</td><td>1314±18.50</td><td>1190±45.24</td><td>796±6.24</td><td>1386±21.06</td><td>1126±12.66</td></tr><tr><td colspan="2">Perplexity Add SBVAE</td><td>1164±2.55</td><td>878±14.21</td><td>980±13.50</td><td>-</td><td>1077±22.57</td><td>1050±12.19</td><td>1670±4.78</td><td>-</td></tr><tr><td/><td>Add DirVAE</td><td>1114±2.30</td><td>752±12.17</td><td>916±1.64</td><td>-</td><td>992±2.19</td><td>809±12.60</td><td>1526±6.11</td><td>-</td></tr><tr><td/><td>Reported</td><td>0.240</td><td>0.186</td><td>0.121</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td/><td>Reproduced</td><td>0.273±0.019</td><td colspan="2">0.119±0.003 0.199±0.006</td><td>0.225±0.002</td><td colspan="2">0.194±0.005 0.023±0.002</td><td>0.267±0.019</td><td>0.266±0.006</td></tr><tr><td>NPMI</td><td>Add SBVAE</td><td>0.247±0.015</td><td colspan="2">0.162±0.007 0.162±0.006</td><td>-</td><td colspan="2">0.190±0.006 0.116±0.016</td><td>0.207±0.004</td><td>-</td></tr><tr><td/><td colspan="4">Add DirVAE 0.359±0.026 0.247±0.010 0.201±0.003</td><td>-</td><td colspan="3">0.193±0.004 0.131±0.015 0.308±0.005</td><td>-</td></tr></table>

The error rate of kNN with the latent representations of VAEs.

Table content: <table/>

Hyper-parameter selections for DirVAE augmentations. .98 • 1 50 0.95 • 1 50 0.20 • 1 50 0.99 • 1 100 0.90 • 1 100 0.01 • 1 100

Table content: <table><tr><td colspan="3">20Newsgroups (K = 50)</td><td colspan="2">RCV1-v2 (K = 100)</td><td/></tr><tr><td>ProdLDA</td><td>NVDM</td><td>GSM</td><td>ProdLDA</td><td>NVDM</td><td>GSM</td></tr><tr><td>Add DirVAE 0</td><td/><td/><td/><td/><td/></tr></table>

Sample of learned per topic top-10 high probability words from 20Newsgroups with DirVAE augmentation by activating single latent dimensions. ProdLDA+DirVAE Topic 1 turks turkish armenian genocide village armenia armenians muslims turkey greece Topic 2 doctrine jesus god faith christ scripture belief eternal holy bible Topic 3 season defensive puck playoff coach score flyers nhl team ice Topic 4 pitcher braves hitter coach pen defensive injury roger pitch player Topic 5 ide scsi scsus controller motherboard isa cache mb floppy ram Topic 6 toolkit widget workstation xlib jpeg xt vendor colormap interface pixel Topic 7 spacecraft satellite solar shuttle nasa mission professor lunar orbit rocket Topic 8 knife handgun assault homicide batf criminal gun firearm police apartment Topic 9 enforcement privacy encrypt encryption ripem wiretap rsa cipher cryptography escrow Topic 10 min detroit tor det calgary rangers leafs montreal philadelphia cal

Table content: <table/>

Peer review (iclr2019):
 This paper proposes DirVAE, a variational autoencoder with Dirichlet prior on latent variables. The advantage of using Dirichlet distribution is that due the nature of Dirichlet distribution the model does not suffer from decoder weight collapsing and latent value collapsing. Stochastic gradient variational Bayes with inverse CDF reparametrization of gamma distribution is presented. The motivation behind using Dirichlet instead of GEM makes sense, but other than that I fail to find any novelty in the paper. The authors should tone down the statement "to our knowledge, combining the two statistical results is the first finding in the machine learning field". Even though left unpublished, I*ve been using this combination of inverse CDF gamma reparametrization and transformation to Dirichlet all the time for my own problems. It*s just trivial once we have both techniques. See also [2], where an improved way of reparametrizing gamma and Dirichlet distribution is presented. The observation that DirVAE does not suffer from latent value collapsing is interesting, but not really surprising. Minor question - What is the difference between negative LL*s and reconstruction losses in experiments? - The approximation for inverse CDF of gamma works well only when alpha << 1. How did you treat the regime alpha > 1? References [1] Diederik P Kingma, Max Welling, Auto-encoding variational Bayes, ICLR, 2014. [2] Michael Figurnov, Shakir Mohamed, Andriy Mnih, Implicit reparametrization gradients, arXiv, 2018.