Year: 2018

Abstract:
This paper proposes Dirichlet Variational Autoencoder (DirVAE) using a Dirichlet prior for a continuous latent variable that exhibits the characteristic of the categorical probabilities. To infer the parameters of DirVAE, we utilize the stochastic gradient method by approximating the Gamma distribution, which is a component of the Dirichlet distribution, with the inverse Gamma CDF approximation. Additionally, we reshape the component collapsing issue by investigating two problem sources, which are decoder weight collapsing and latent value collapsing, and we show that DirVAE has no component collapsing; while Gaussian VAE exhibits the decoder weight collapsing and Stick-Breaking VAE shows the latent value collapsing. The experimental results show that 1) DirVAE models the latent representation result with the best log-likelihood compared to the baselines; and 2) DirVAE produces more interpretable latent values with no collapsing issues which the baseline models suffer from. Also, we show that the learned latent representation from the DirVAE achieves the best classification accuracy in the semi-supervised and the supervised classification tasks on MNIST, OMNIGLOT, and SVHN compared to the baseline VAEs. Finally, we demonstrated that the DirVAE augmented topic models show better performances in most cases.

heading: 1 INTRODUCTION
text: A Variational Autoencoder (VAE) (Kingma & Welling, 2014c) brought success in deep generative models (DGMs) with a Gaussian distribution as a prior distribution (Jiang et al., 2017; Miao et al., 2016; 2017; Srivastava & Sutton, 2017). If we focus on the VAE, the VAE assumes the prior distribution to be N (0, I) with the learning on the approximated µ̂ and Σ̂. Also, Stick-Breaking VAE (SBVAE) (Nalisnick & Smyth, 2017) is a nonparametric version of the VAE, which modeled the latent dimension to be infinite using a stick-breaking process (Ishwaran & James, 2001).
While these VAEs assume that the prior distribution of the latent variables to be continuous random variables, recent studies introduce the approximations on discrete priors with continuous random variables (Jang et al., 2017; Maddison et al., 2017; Rolfe, 2017). The key of these approximations is enabling the backpropagation with the reparametrization technique, or the stochastic gradient variational Bayes (SGVB) estimator, while the modeled prior follows a discrete distribution. The applications of these approximations on discrete priors include the prior modeling of a multinomial distribution which is frequently used in the probabilistic graphical models (PGMs). Inherently, the multinomial distributions can take a Dirichlet distribution as a conjugate prior, and the demands on such prior have motivated the works like Jang et al. (2017); Maddison et al. (2017); Rolfe (2017) that support the multinomial distribution posterior without explicit modeling on a Dirichlet prior.
When we survey the work with a explicit modeling on the Dirichlet prior, we found a frequent approach such as utilizing a softmax Laplace approximation (Srivastava & Sutton, 2017). We argue that this approach has a limitation from the multi-modality perspective. The Dirichlet distribution can exhibit a multi-modal distribution with parameter settings, see Figure 1, which is infeasible to generate with the Gaussian distribution with a softmax function. Therefore, the previous continuous domain VAEs cannot be a perfect substitute for the direct approximation on the Dirichlet distribution.
Utilizing a Dirichlet distribution as a conjugate prior to a multinomial distribution has an advantage compared to the usage of a softmax function on a Gaussian distribution. For instance, Figure 1 illustrates the potential difficulties in utilizing the softmax function with the Gaussian distribution.
Given the three-dimensional probability simplex, the Gaussian-Softmax distribution cannot generate the illustrated case of the Dirichlet distribution with a high probability measure at the vertices of the simplex, i.e. the multi-modality where the necessity was emphasized in Hoffman & Johnson (2016). Additionally, the Griffiths-Engen-McCloskey (GEM) distribution (Pitman, 2002), which is the prior distribution of the SBVAE, is difficult to model the multi-modality because the sampling procedure of the GEM distribution is affected by the rich-get-richer phenomenon, so a few components tend to dominate the weight of the samples. This is different from the Dirichlet distribution that does not exhibit such phenomenon, and the Dirichlet distribution can fairly distribute the weights to the components, and the Dirichlet distribution is more likely to capture the multi-modality by controlling the prior hyper-parameter (Blei et al., 2003). Then, we conjecture that an enhanced modeling on Dirichlet prior is still needed 1) because there are cases that the Gaussian-Softmax approaches, or the softmax Laplace approximation, cannot imitate the Dirichlet distribution; and 2) because the nonparametric approaches could be influenced by the biases that the Dirichlet distribution does not suffer from.
Given these motivations for modeling the Dirichlet distribution with the SGVB estimator, this paper introduces the Dirichlet Variational Autoencoder (DirVAE) that shows the same characteristics of the Dirichlet distribution. The DirVAE is able to model the multi-modal distribution that was not possible with the Gaussian-Softmax and the GEM approaches. These characteristics allow the DirVAE to be the prior of the discrete latent distribution, as the original Dirichlet distribution is.
Introducing the DirVAE requires the configuration of the SGVB estimator on the Dirichlet distribution. Specifically, the Dirichlet distribution is a composition of the Gamma random variables, so we approximate the inverse Gamma cumulative distribution function (CDF) with the asymptotic approximation. This approximation on the inverse Gamma CDF becomes the component of approximating the Dirichlet distribution. We compared this approach to the previously suggested approximations, i.e. approaches with the Weibull distribution and with the softmax Gaussian distribution, and our approximation shows the best log-likelihood among the compared approximations.
Moreover, we report that we had to investigate the component collapsing along with the research on DirVAE. It has been known that the component collapsing issue is resolved by the SBVAE because of the meaningful decoder weights from the latent layer to the next layer. However, we found that SBVAE has latent value collapsing issue resulting in many near-zero values on the latent dimensions that leads to the incomplete utilization of the latent dimension. Hence, we argue that Gaussian VAE (GVAE) suffers from the decoder weight collapsing, previously limitedly defined as component collapsing; and SBVAE has a problem of the latent value collapsing. Finally, we suggest that the definition of component collapsing should be expanded to represent both cases of decoder weight and latent value collapsings. The proposed DirVAE shows neither the near-zero decoder weights nor the near-zero latent values, so the reconstruction uses the full latent dimension information in most cases. We investigated this issue because our performance gain comes from resolving the expanded version of the component collapsing. Due to the component collapsing issues, the existing VAEs have less meaningful latent values or could not effectively use its latent representation. Meanwhile, DirVAE does not have component collapsing due to the multi-modal prior which possibly leads to superior qualitative and quantitative performances. We experimentally showed that the DirVAE has more meaningful or disentangled latent representation by image generation and latent value visualizations.
Technically, the new approximation provides the closed-form loss function derived from the evidence lower bound (ELBO) of the DirVAE. The optimization on the ELBO enables the representation learning with the DirVAE, and we test the learned representation from the DirVAE in two folds. Firstly, we test the representation learning quality by performing the supervised and the semi-supervised classification tasks on MNIST, OMNIGLOT, and SVHN. These classification tasks conclude that DirVAE has the best classification performances with its learned representation. Secondly, we test the applicability of DirVAE to the existing models, such as topic models with DirVAE priors on 20Newsgroup and RCV1-v2. This experiment shows that the augmentation of DirVAE to the existing neural variational topic models improves the perplexity and the topic coherence, and most of best performers were DirVAE augmented.

heading: 2 PRELIMINARIES

heading: 2.1 VARIATIONAL AUTOENCODERS
text: A VAE is composed of two parts: a generative sub-model and an inference sub-model. In the generative part, a probabilistic decoder reproduces x̂ close to an observation x from a latent variable z ∼ p(z), i.e. x ∼ pθ(x|z) = pθ(x|ζ) where ζ = MLP(z) is obtained from a latent variable z by a multilayer perceptron (MLP). In the inference part, a probabilistic encoder outputs a latent variable z ∼ qφ(z|x) = qφ(z|η) where η = MLP(x) is computed from the observation x by a MLP. Model parameters, θ and φ, are jointly learned by optimizing the below ELBO with the stochastic gradient method through the backpropagations as the ordinary neural networks by using the SGVB estimators on the random nodes.
log p(x) ≥ L(x) = Eqφ(z|x) [log pθ(x|z)]− KL(qφ(z|x)||pθ(z)) (1)
In GVAE (Kingma & Welling, 2014c), the prior distribution of p(z) is assumed to be a standard Gaussian distribution. In SBVAE (Nalisnick & Smyth, 2017), the prior distribution becomes a GEM distribution that produces samples with a Beta distribution and a stick-breaking algorithm.

heading: 2.2 DIRICHLET DISTRIBUTION AS A COMPOSITION OF GAMMA RANDOM VARIABLES
text: The Dirichlet distribution is a composition of multiple Gamma random variables. Note that the probability density functions (PDFs) of Dirichlet and Gamma distributions are as follows:
Dirichlet(x;α) = Γ( ∑ αk)∏
Γ(αk)
∏ xαk−1k , Gamma(x;α, β) = βα
Γ(α) xα−1e−βx (2)
where αk, α, β > 0. In detail, if there are K independent random variables following the Gamma distributions Xk ∼ Gamma(αk, β) or X ∼ MultiGamma(α, β · 1K) where αk, β > 0 for k = 1, · · · ,K, then we have Y ∼ Dirichlet(α) where Yk = Xk/ ∑ Xi. It should be noted that the rate parameter, β, should be the same for every Gamma distribution in the composition. Then, the KL divergence can be derived as the following:
KL(Q||P ) = ∑ log Γ(αk)− ∑ log Γ(α̂k) + ∑ (α̂k − αk)ψ(α̂k) (3)
for P = MultiGamma(α, β ·1K) andQ = MultiGamma(α̂, β ·1K) where ψ is a digamma function. The detailed derivation is provided in Appendix B.

heading: 2.3 SGVB FOR GAMMA RANDOM VARIABLE AND APPROXIMATION ON DIRICHLET DISTRIBUTION
text: This section discusses several ways of approximating the Dirichlet random variable; or the SGVB estimators for the Gamma random variables which compose a Dirichlet distribution. Utilizing SGVB requires a differentiable non-centered parametrization (DNCP) for the distribution (Kingma & Welling, 2014d). The main SGVB for Gamma random variables, used in DirVAE, is using the inverse Gamma CDF approximation explained in the next section. Prior works include two approaches: the use of the Weibull distribution and the softmax Gaussian distribution, and the two approaches are explained in this section.
Approximation with Weibull distribution. Because of the similar PDFs between the Weibull distribution and the Gamma distribution, some prior works used the Weibull distribution as a posterior distribution of the prior Gamma distribution (Zhang et al., 2018):
Weibull(x; k, λ) = k
λ (x λ )k−1 e−(x/λ) k where k, λ > 0 . (4)
The paper Zhang et al. (2018) pointed out that there are two useful characteristics when approximating the Gamma distribution with the Weibull distribution. One useful property is that the KL divergence expressed in a closed form, and the other is the simple reparametrization trick with a closed form of the inverse CDF from the Weibull distribution. However, we noticed that the Weibull distribution has a component of e−(x/λ) k
, and the Gamma distribution does not have the additional power term of k in the component. Since k is placed in the exponential component, small changes on k can cause a significant difference that limits the optimization.
Approximation with softmax Gaussian distribution. As in MacKay (1998); Srivastava & Sutton (2017), a Dirichlet distribution can be approximated by a softmax Gaussian distribution by using a softmax Laplace approximation. The relation between the Dirichlet parameter α and the Gaussian parameters µ,Σ is explained as the following:
µk = logαk − 1
K ∑ i logαi, Σk = 1 αk ( 1− 2 K ) + 1 K2 ∑ i 1 αi , (5)
where Σ is assumed to be a diagonal matrix, and we use the reparametrization trick in the usual GVAE for the SGVB estimator.

heading: 3 MODEL DESCRIPTION
text: Along with the inverse Gamma CDF approximation, we describe two sub-models in this section: the generative sub-model and the inference sub-model. Figure 2 describes the graphical notations of various VAEs and the neural network view of our model.
Generative sub-model. The key difference between the generative models between the DirVAE and the GVAE is the prior distribution assumption on the latent variable z. Instead of using the standard Gaussian distribution, we use the Dirichlet distribution which is a conjugate prior distribution of the multinomial distribution.
z ∼ p(z) = Dirichlet(α), x ∼ pθ(x|z) (6)
Inference sub-model. The probabilistic encoder with an approximating posterior distribution qφ(z|x) is designed to be Dirichlet(α̂). The approximated posterior parameter α̂ is derived by the MLP from the observation x with the softplus output function, so the outputs can be positive values constrained by the Dirichlet distribution. Here, we do not directly sample z from the Dirichlet distribution. Instead, we use the Gamma composition method described in Section 2.2. Firstly, we draw v ∼ MultiGamma(α, β · 1K). Afterwards, we normalize v with its summation ∑ vi.
The objective function to optimize the model parameters, θ and φ, is composed of Equation (1) and (3). Equation (7) is the loss function to optimize after the composition. The inverse Gamma CDF method explained in the next paragraph enables the backpropagation flows to the input with the stochastic gradient method. Here, for the fair comparison of expressing the Dirichlet distribution between the inverse Gamma CDF approximation method and the softmax Gaussian method, we set αk = 1− 1/K when µk = 0 and Σk = 1 by using Equation (5); and β = 1.
L(x) = Eqφ(z|x) [log pθ(x|z)]− ( ∑ log Γ(αk)− ∑ log Γ(α̂k) + ∑ (α̂k − αk)ψ(α̂k)) (7)
Approximation with inverse Gamma CDF. A previous work Knowles (2015) suggested that, if X ∼ Gamma(α, β), and if F (x;α, β) is a CDF of the random variable X , the inverse CDF can be approximated as F−1(u;α, β) ≈ β−1(uαΓ(α))1/α. Hence, we can introduce an auxiliary variable u ∼ Uniform(0, 1) to take over all the randomness of X , and we treat the Gamma sampled X as a deterministic value in terms of α and β.
It should be noted that there has been a practice of utilizing the combination of decomposing a Dirichlet distribution and approximating each Gamma component with inverse Gamma CDF. However, such practices have not been examined with its learning properties and applicabilities. The following section shows a new aspect of component collapsing that can be remedied by this combination on Dirichlet prior in VAE, and the section illustrates the performance gains in a certain set of applications, i.e. topic modeling.

heading: 4 EXPERIMENTAL RESULTS
text: This section reports the experimental results with the following experiment settings: 1) a pure VAE model; 2) a semi-supervised classification task with VAEs; 3) a supervised classification task with VAEs; and 4) topic models with DirVAE augmentations.

heading: 4.1 EXPERIMENTS FOR REPRESENTATION LEARNING OF VAES
text: Baseline models. We select the following models as baseline alternatives of the DirVAE: 1) the standard GVAE; 2) the GVAE with softmax (GVAE-Softmax) approximating the Dirichlet distribution with the softmax Gaussian distribution; 3) the SBVAE with the Kumaraswamy distribution (SBVAE-Kuma) & the Gamma composition (SBVAE-Gamma) described in Nalisnick & Smyth (2017); and 4) the DirVAE with the Weibull distribution (DirVAE-Weibull) approximating the Gamma distribution with the Weibull distribution described in Zhang et al. (2018). We use the following benchmark datasets for the experiments: 1) MNIST; 2) MNIST with rotations (MNIST+rot); 3) OMNIGLOT; and 4) SVHN with PCA transformation. We provide the details on the datasets in Appendix D.1.
Experimental setting. As a pure VAE model, we compare the DirVAE with the following models: GVAE, GVAE-Softmax, SBVAE-Kuma, SBVAE-Gamma, and DirVAE-Weibull. We use 50- dimension and 100-dimension latent variables for MNIST and OMNIGLOT, respectively. We provide the details of the network structure and optimization in Appendix D.2. We set α = 0.98 · 150 for MNIST and α = 0.99 · 1100 for OMNIGLOT for the fair comparison to GVAEs by using Equation (5). All experiments use the Adam optimizer (Kingma & Ba, 2014a) for the parameter learning. Finally, we acknowledge that the hyper-parameter could be updated as Appendix C, and the experiment result with the update is separately reported in Appendix D.2.
Quantitative result. For the quantitative comparison among the VAEs, we calculated the MonteCarlo estimation on the marginal negative log-likelihood, the negative ELBO, and the reconstruction loss. The marginal log-likelihood is approximated as p(x) ≈ ∑ i p(x|zi)p(zi) q(zi) for single instance x where q(z) is a posterior distribution of a prior distribution p(z), which is further derived in Appendix A. Table 1 shows the overall performance of the alternative VAEs. The DirVAE outperforms all baselines in both datasets from the log-likelihood perspective. The value of DirVAE comes from the better encoding of the latent variables that can be used for classification tasks which we examine in the next experiments. While the DirVAE-Weibull follows the prior modeling with the Dirichlet distribution, the Weibull based approximation can be improved by adopting the proposed approach with the inverse Gamma CDF.
Qualitative result. As a qualitative result, we report the latent dimension-wise reconstructions which are decoder outputs with each one-hot vector in the latent dimension. Figure 3a shows 50 reconstructed images corresponding to each latent dimension from GVAE-Softmax, SBVAE, and DirVAE. We manually ordered the digit-like figures in the ascending order for GVAE-Softmax and DirVAE. We can see that the GVAE-Softmax and the SBVAE have components without significant semantic information, which we will discuss further in Section 4.2, and the DirVAE has interpretable latent dimensions in most of the latent dimensions. Figure 3b also supports the quality of the latent values from DirVAE by visualizing learned latent values through t-SNE (Maaten & Hinton, 2008).

heading: 4.2 DISCUSSION ON COMPONENT COLLAPSING
text: Decoder weight collapsing, a.k.a. component collapsing. One main issue of GVAE is component collapsing that there are a significant number of near-zero decoder weights from the latent neurons to the next decoder neurons. If these weights become near-zero, the values of the latent dimensions loose influence to the next decoder, and this means an inefficient learning given a neural network structure. The same issue occurs when we use the GVAE-Softmax. We rename this component collapsing phenomenon as decoder weight collapsing to specifically address the collapsing source.
Latent value collapsing. SBVAE claims that SBVAE solved the decoder weight collapsing by learning the meaningful weights as shown in Figure 4a. However, we notice that SBVAE produces the output values, not the weight parameters, from the latent dimension to be near-zero in many latent dimensions after averaging many samples obtained from the test dataset. Figure 4b shows the properties of DirVAE and SBVAE from the perspective of the latent value collapsing, which SBVAE shows many near-zero average means and near-zero average variances, while DirVAE does not. The average Fisher kurtosis and average skewness of DirVAE are 5.76 and 2.03, respectively over the dataset, while SBVAE has 20.85 and 4.35, which states that the latent output distribution from SBVAE is more skewed than that of DirVAE. We found out that these near-zero latent values prevent learning on decoder weights, which we introduce as another type of collapsing problem, as latent value collapsing that is different from the decoder weight collapsing. These results mean that SBVAE distributes the non-near-zero latent values sparsely over a few dimensions while DirVAE samples relatively dense latent values. In other words, DirVAE utilizes the full spectrum of latent dimensions compared to SBVAE, and DirVAE has a better learning capability in the decoder network. Figure 3a supports the argument on the latent value collapsing by activating each and single latent dimension with a one-hot vector through the decoder. The non-changing latent dimension-
wise images of SBVAE proves that there were no generation differences between the two differently activated one-hot latent values.

heading: 4.3 APPLICATION 1. EXPERIMENTS OF (SEMI-)SUPERVISED CLASSIFICATION WITH VAES
text: Semi-supervised classification task with VAEs. There is a previous work demonstrating that the SBVAE outperforms the GVAE in semi-supervised classification task (Nalisnick & Smyth, 2017). The overall model structure for this semi-supervised classification task uses a VAE with separate random variables of z and y, which is introduced as the M2 model in the original VAE work (Kingma et al., 2014b). The detailed settings of the semi-supervised classification tasks are enumerated in Appendix D.3. Fundamentally, we applied the same experimental settings to GVAE, SBVAE, and DirVAE in this experiment, as specified by the authors in Nalisnick & Smyth (2017).
Table 2 enumerates the performances of the GVAE, the SBVAE, and the DirVAE, and the result shows that the error rate of classification result using 10%, 5% and 1% of labeled data for each dataset. In general, the experiment shows that the DirVAE has the best performance out of three alternative VAEs. Also, it should be noted that the performance of the DirVAE is more improved in the most complex task with the SVHN dataset.
GVAE (Nalisnick & Smyth, 2017) 3.95±0.15 4.74±0.43 11.55±2.28 21.78±0.73 27.72±0.69 38.13±0.95 36.08±1.49 48.75±1.47 69.58±1.64 SBVAE (Nalisnick & Smyth, 2017) 4.86±0.14 5.29±0.39 7.34±0.47 11.78±0.39 14.27±0.58 27.67±1.39 32.08±4.00 37.07±5.22 61.37±3.60 DirVAE 4.60±0.07 5.05±0.18 7.00±0.17 11.18±0.32 13.53±0.46 26.20±0.66 24.81±1.13 28.45±1.14 55.99±3.30
Supervised classification task with latent values of VAEs. Also, we tested the performance of the supervised classification task with the learned latent representation from the VAEs. We applied the vanilla version of VAEs to the datasets, and we classified the latent representation of instances with k-Nearest Neighbor (kNN) which is one of the simplest classification algorithms. Hence, this experiment can better distinguish the performance of the representation learning in the classification task. Further experimental details can be found in Appendix D.4.
Table 3 enumerates the performances from the experimented VAEs in the datasets of MNIST and OMNIGLOT. Both datasets indicated that the DirVAE shows the best performance in reducing the classification error, which we conjecture that the performance is gathered from the better representation learning. It should be noted that, to our knowledge, this is the first reported comparison of latent representation learning on VAEs with kNN in the supervised classification using OMNIGLOT dataset. We identified that the classification with OMNIGLOT is difficult given that the kNN error
rates with the raw original data are as high as 69.94%, 69.41%, and 70.10%. This high error rate mainly originates from the number of classification categories which is 50 categories in our test setting of OMNIGLOT, compared to 10 categories in MNIST.

heading: 4.4 APPLICATION 2. EXPERIMENTS OF TOPIC MODEL AUGMENTATION WITH DIRVAE
text: One usefulness of the Dirichlet distribution is being a conjugate prior to the multinomial distribution, so it has been widely used in the field of topic modeling, such as Latent Dirichlet Allocation (LDA) (Blei et al., 2003). Recently, some neural variational topic (or document) models have been suggested, for example, ProdLDA (Srivastava & Sutton, 2017), NVDM (Miao et al., 2016), and GSM (Miao et al., 2017). NVDM used the GVAE, and the GSM used the GVAE-Softmax to make the sum-to-one positive topic vectors. Meanwhile, ProdLDA assume the prior distribution to be the Dirichlet distribution with the softmax Laplace approximation. To verify the usefulness of the DirVAE, we replace the probabilistic encoder part of the DirVAE to each model. Two popular performance measures in the topic model fields, which are perplexity and topic coherence via normalized pointwise mutual information (NPMI) (Lau et al., 2014), have been used with 20Newsgroups and RCV1-v2 datasets. Further details of the experiments can be found in Appendix D.5. Table 4 indicates that the augmentation of DirVAE improves the performance in general. Additionally, the best performers from the two measurements are always the experiment cell with DirVAE augmentation except for the perplexity of RCV1-v2, which still remains competent.

heading: 5 CONCLUSION
text: Recent advances in VAEs have become one of the cornerstones in the field of DGMs. The VAEs infer the parameters of explicitly described latent variables, so the VAEs are easily included in the conventional PGMs. While this merit has motivated the diverse cases of merging the VAEs to the graphical models, we ask the fundamental quality of utilizing the GVAE where many models have latent values to be categorical probabilities. The softmax function cannot reproduce the multi-modal distribution that the Dirichlet distribution can. Recognizing this problem, there have been some previous works that approximated the Dirichlet distribution in the VAE settings by utilizing the Weibull distribution or the softmax Gaussian distribution, but the DirVAE with the inverse Gamma CDF shows the better learning performance in our experiments of the representation: the semi-supervised, the supervised classifications, and the topic models. Moreover, DirVAE shows no component collapsing and it leads to better latent representation and performance gain. The proposed DirVAE can be widely used if we recall the popularity of the conjugate relation between the multinomial and the Dirichlet distributions because the proposed DirVAE can be a brick to the construction of complex probabilistic models with neural networks.

heading: A MONTE-CARLO ESTIMATION ON THE MARGINAL LIKELIHOOD
text: Proposition A.1. The marginal log-likelihood is approximated as p(x) ≈ ∑ i p(x|zi)p(zi) q(zi) , where q(z) is a posterior distribution of a prior distribution p(z).
Proof.
p(x) = ∫ z p(x, z)dz = ∫ z p(x, z) q(z) q(z) dz
= ∫ z p(x|z)p(z)q(z) q(z) dz = ∫ z p(x|z)p(z) q(z) q(z)dz
≈ ∑ i p(x|zi)p(zi) q(zi) where zi ∼ q(z)

heading: B KL DIVERGENCE OF TWO MULTI-GAMMA DISTRIBUTIONS
text: Proposition B.1. Define X = (X1, · · · , XK) ∼ MultiGamma(α, β · 1K) as a vector of K independent Gamma random variables Xk ∼ Gamma(αk, β) where αk, β > 0 for k = 1, · · · ,K. The KL divergence between two MultiGamma distributions P = MultiGamma(α, β · 1K) and Q = MultiGamma(α̂, β · 1K) can be derived as the following:
KL(Q||P ) = ∑ log Γ(αk)− ∑ log Γ(α̂k) + ∑ (α̂k − αk)ψ(α̂k) , (8)
where ψ is a digamma function.
Proof. Note that the derivative of a Gamma-like function Γ(α)βα can be derived as follows:
d
dα
Γ(α)
βα = β−α(Γ′(α)− Γ(α) log β) = ∫ ∞ 0 xα−1e−βx log x dx .
Then, we have the following. KL(Q||P ) = ∫ D q(x) log q(x) p(x) dx
= ∫ ∞ 0 · · · ∫ ∞ 0 ∏ Gamma(α̂k, β) log β ∑ α̂k ∏ Γ−1(α̂k)e −β ∑ xk ∏ x α̂k−1 k β ∑ αk ∏ Γ−1(αk)e −β ∑ xk ∏ x αk−1 k dx
= ∫ ∞ 0 · · · ∫ ∞ 0 ∏ Gamma(α̂k, β)
× [∑ (α̂k − αk) log β + ∑ log Γ(αk)− ∑ log Γ(α̂k) + ∑ (α̂k − αk) log xk ] dx
= [∑ (α̂k − αk) log β + ∑ log Γ(αk)− ∑ log Γ(α̂k) ]
+ ∫ ∞ 0 · · · ∫ ∞ 0 βα̂k∏ Γ(α̂k) e−β ∑ xk ∏ xα̂k−1k (∑ (α̂k − αk) log xk ) dx
= [∑ (α̂k − αk) log β + ∑ log Γ(αk)− ∑ log Γ(α̂k) ]
+ ∑ (α̂k − αk)βα̂kΓ−1(α̂k)β−α̂k ( Γ′(α̂k)− Γ(α̂k) log β ) = ∑ (α̂k − αk) log β + ∑ log Γ(αk)− ∑ log Γ(α̂k) + ∑ (α̂k − αk)(ψ(α̂k)− log β)
= ∑ log Γ(αk)− ∑ log Γ(α̂k) + ∑ (α̂k − αk)ψ(α̂k)
C HYPER-PARAMETER α LEARNING STRATEGY
In this section, we introduce the method of moment estimator (MME) to update the Dirichlet prior parameter α. Suppose we have a set of sum-to-one proportions D = {p1, · · · ,pN} sampled from Dirichlet(α), then the MME update rule is as the following:
αk ← S
N ∑ n pn,k where S = 1 K ∑ k µ̃1,k − µ̃2,k µ̃2,k − µ̃21,k for µ̃j,k = 1 N ∑ n pjn,k . (9)
After the burn-in period for stabilizing the neural network parameters, we use the MME for the hyper-parameter learning using the sampled latent values during training. We alternatively update the neural network parameters and hyper-parameter α. We choose this estimator because of its closed form nature and consistency (Minka, 2000). The usefulness of the hyper-parameter update can be found in Appendix D.2.
Proposition C.1. Given a proportion set D = {p1, · · · ,pN} sampled from Dirichlet(α), MME of the hyper-parameter α is as the following:
αk ← S
N ∑ n pn,k where S = 1 K ∑ k µ̃1,k − µ̃2,k µ̃2,k − µ̃21,k for µ̃j,k = 1 N ∑ n pjn,k .
Proof. Define µj,k = E[pjk] as the jth moment of the kth dimension of Dirichlet distribution with prior α. Then, by the law of large number, µj,k ≈ µ̃j,k. It can be easily shown that µ1,k = αk∑
i αi and µ2,k = αk∑ i αi 1+αk 1+ ∑ i αi = µ1,k 1+αk 1+ ∑ i αi so that
numerator (µ1,k − µ2,k µ2,k − µ21,k ) = αk∑ i αi − αk∑ i αi 1 + αk 1 + ∑ i αi
= αk( ∑ i 6=k αi)
( ∑ i αi)(1 + ∑ i αi)
denominator (µ1,k − µ2,k µ2,k − µ21,k ) = αk∑ i αi 1 + αk 1 + ∑ i αi − ( αk∑ i αi )2 = αk( ∑ i 6=k αi)
( ∑ i αi) 2(1 + ∑ i αi)
holds for each k = 1, · · · ,K. Therefore,∑ i αi = µ1,k − µ2,k µ2,k − µ21,k ≈ 1 K ∑ k µ1,k − µ2,k µ2,k − µ21,k ≈ 1 K ∑ k µ̃1,k − µ̃2,k µ̃2,k − µ̃21,k
and hence,
α̂k = ( ∑ i αi)µ̃1,k = S N ∑ n pn,k.

heading: D EXPERIMENTAL SETTINGS
text: In this section, we support Section 4 in the original paper with more detailed experimental settings. Our Tensorflow implementation is available at https://TO BE RELEASED.
D.1 DATASET DESCRIPTION
We use the following benchmark datasets for the experiments in the original paper: 1) MNIST; 2) MNIST with rotations (MNIST+rot); 3) OMNIGLOT; and 4) SVHN with PCA transformation. MNIST (LeCun et al., 1998) is a hand-written digit image dataset of size 28 × 28 with 10 labels, consists of 60, 000 training data and 10, 000 testing data. MNIST+rot data is reproduced by the authors of Nalisnick & Smyth (2017) consists of MNIST and rotated MNIST1. OMNIGLOT2 (Lake et al., 2013; Snderby et al., 2016) is another hand-written image dataset of characters with 28 × 28 size and 50 labels, consists of 24, 345 training data and 8, 070 testing data. SVHN3 is a Street View House Numbers image dataset with the dimension-reduction by PCA into 500 dimensions (Nalisnick & Smyth, 2017).
D.2 REPRESENTATION LEARNING OF VAES
We divided the datasets into {train,valid,test} as the following: MNIST = {45, 000 : 5, 000 : 10, 000} and OMNIGLOT = {22, 095 : 2, 250 : 8, 070}. For MNIST, we use 50-dimension latent variables with two hidden layers in the encoder and one hidden layer in the decoder of 500 dimensions. We set α = 0.98 · 150 for the fair comparison to GVAEs using the Equation (5). The batch size was set to be 100. For OMNIGLOT, we use 100- dimension latent variables with two hidden layers in the encoder and one hidden layer in the decoder of 500 dimensions. We assume α = 0.99 · 1100 for the fair comparison to the GVAEs using the Equation (5). The batch size was set to be 15.
For both datasets, the gradient clipping is used; ReLU function (Nair & Hinton, 2010) is used as an activation function in hidden layers; Xavier initialization (Glorot & Bengio, 2010) is used for the neural network parameter initialization; and the Adam optimizer (Kingma & Ba, 2014a) is used as an optimizer with learning rate 5e-4 for all VAEs except 3e-4 for the SBVAEs. The prior assumptions for each VAE is the following: 1) N (0, I) for the GVAE and the GVAE-Softmax; 2) GEM(5) for the SBVAEs; and 3) Dirichlet(0.98 · 150) (MNIST) and Dirichlet(0.99 · 1100) (OMNIGLOT) for the DirVAE-Weibull. Finally, to compute the marginal log-likelihood, we used 100 samples for each 1, 000 randomly selected from the test data.
We add the result of VAE with 20 normalizing flows (GVAE-NF20) (Rezende & Mohamed, 2015) as a baseline in Table 5. Also, latent dimension-wise decoder weight norm and t-SNE visualization on latent embeddings of MNIST is given in Figure 5a and 5b which correspond to Figure 4a and 3, respectively.
Additionally, DirVAE-Learning use the same α for the initial value, but the DirVAE-Learning optimizes hyper-parameter α by the following stages through the learning iterations using the MME method in Appendix C: 1) the burn-in period for stabilizing the neural network parameters; 2) the alternative update period for the neural network parameters and α; and 3) the update period for the neural network parameters with the fixed learned hyper-parameter α. Table 5 shows that there are improvements in the marginal log-likelihood, ELBO, and reconstruction loss with DirVAE-Learning in both datasets. We also give the learned hyper-parameter α in Figure 6.
D.3 SEMI-SUPERVISED CLASSIFICATION TASK WITH VAES
The overall model structure for this semi-supervised classification task uses a VAE with a separate random variable of z and y, which is introduced as the M2 model in the original VAE work (Kingma et al., 2014b). However, the same task with the SBVAE uses a different model modified to ignore
1http://www.iro.umontreal.ca/ lisa/twiki/bin/view.cgi/Public/MnistVariations 2https://github.com/yburda/iwae/tree/master/datasets/OMNIGLOT 3http://ufldl.stanford.edu/housenumbers/
the relation between the class label variable y and the latent variable z, but they still share the same parent nodes: qφ(z,y|x) = qφ(z|x)qφ(y|x) where qφ(y|x) is a discrimitive network for the unseen labels. We follow the structure of SBVAE. Finally, the below are the objective functions to optimize for the labeled and the unlabeled instances of the semi-supervised classification task, respectively:
log p(x,y) ≥ Llabeled(x,y) = Eqφ(z|x) [log pθ(x|z,y)]−KL(qφ(z|x)||pθ(z))+log qφ(y|x) , (10)
log p(x) ≥ Lunlabeled(x) = Eqφ(z,y|x) [log pθ(x|z,y) + H(qφ(y|x))]− KL(qφ(z|x)||pθ(z)) . (11)
In the above, H is an entropy function. The actual training on the semi-supervised learning optimizes the weighted sum of Equation (10) and (11) with a ratio hyper-parameter 0 < λ < 1.
The datasets are divided into {train, valid, test} as the following: MNIST = {45, 000 : 5, 000 : 10, 000}, MNIST+rot = {70, 000 : 10, 000 : 20, 000}, and SVHN = {65, 000 : 8, 257 : 26, 032}. For SVHN, dimension reduction into 500 dimensions by PCA is applied as preprocessing.
Fundamentally, we applied the same experimental settings to GVAE, SBVAE and DirVAE in this experiment, as specified by the authors in Nalisnick & Smyth (2017).4,5 Specifically, the three VAEs used the same network structures of 1) a hidden layer of 500 dimension for MNIST; and 2) four hidden layers of 500 dimensions for MNIST+rot and SVHN with the residual network for the last three hidden layers. The latent variables have 50 dimensions for all settings. The ratio parameter λ is set to be 0.375 for the MNISTs, and 0.45 for SVHN. ReLU function is used as an activation function in hidden layers, and the neural network parameters were initialized by sampling from N (0, 0.001). The Adam optimizer is used with learning rate 3e-4 and the batch size was set to be 100. Finally, the DirVAE sets α = 0.98 · 150 by using Equation (5).
4https://github.com/enalisnick/stick-breaking dgms 5https://www.ics.uci.edu/∼enalisni/sb dgm supp mat.pdf
D.4 SUPERVISED CLASSIFICATION TASK WITH LATENT VALUES OF VAES
For the supervised classification task on the latent representation of the VAEs, we used exactly the same experimental settings as in D.2. Since DLGMM is basically a Gaussian mixture model with the SBVAE, DLGMM is a more complex model than the VAE alternatives. We only report the authors’ result from Nalisnick et al. (2016) for the comparison purposes. Additionally, we omit the comparison with the VaDE (Jiang et al., 2017) because the VaDE is more customized to be a clustering model rather than the ordinary VAEs that we choose as baselines.
D.5 TOPIC MODEL AUGMENTATION WITH DIRVAE
For the topic model augmentation experiment, two popular performance measures in the topic model fields, which are perplexity and topic coherence via normalized pointwise mutual information (NPMI) (Lau et al., 2014), have been used with 20Newsgroups6 and RCV1-v27 datasets. 20Newsgroups has 11, 258 train data and 7, 487 test data with vocabulary size 1, 995. For the RCV1-v2 dataset, due to the massive size of the whole data, we randomly sampled 20, 000 train data and 10, 000 test data with vocabulary size 10, 000. The lower is better for the perplexity, and the higher is better for the NPMI.
The specific model structures can be found in the original papers, Srivastava & Sutton (2017); Miao et al. (2016; 2017). We replace the model prior to that of DirVAE to each model and search the hyper-parameter as Table 7 with 1, 000 randomly selected test data. We use 500-dimension hidden layers and 50 topics for 20Newsgroups, and 1, 000-dimension hidden layers and 100 topics for RCV1-v2.
Table 8 shows top-10 high probability words per topic by activating single latent dimensions in the case of 20Newsgroups. Also, we visualized the latent embeddings of documents by t-SNE in Figure 7,8, and 9.
6https://github.com/akashgit/autoencoding vi for topic models 7http://scikit-learn.org/stable/datasets/rcv1.html

Review summaries:
In this paper, authors proposes an algorithm to use Dirichlet prior on the variational auto-encoder (VAE). They used this prior as natural conjugate to likelihood distributtion of multinomial (categorical). The paper proposes a way to use scalability power of VAE for data distributed by categorical distribution. In order to apply reparametrization trick, authors have used iid Gamma random variable to construct draw from Dirichlet distribution and have used approximation with inverse gamma CDF,  it is discussed how this method has better performance than other approximations method for gamma distribution such as Weibull and logistic Gaussian.

This paper proposes to change the typical Gaussian posterior distribution (and prior) for the latent features z associated to an image x that is used in Variational Autoencoders by a Dirichlet distribution. The work improves over previous attempts based on a soft-max + Gaussian distribution and the soft-max + Weibull distribution. The trick proposed to make feasible training the model includes approximating the inverse CDF of the gamma distribution and using the fact that the Dirichlet distribution can also be obtained as a normalized sum of gamma random variables. The method is compared in several problems. Some analysis of the reasons why it performs better is also carried out.