The advantage of using Student's t-priors in variational autoencoders
Authors: Najmeh Abiri
Mattias Ohlsson
Venue:  (2020)
Keywords: Variational Autoencoders
DLVMs
Posterior Collapse

Abstract: Is it optimal to use the standard Gaussian prior in variational autoencoders? With Gaussian distributions, which are not weakly informative priors, variational autoencoders struggle to reconstruct the actual data. We provide numerical evidence that encourages using Student's t-distributions as default priors in variational autoencoders, and we challenge the usual setup for the variational autoencoder structure by comparing Gaussian and Student's t-distribution priors with different forms of the covariance matrix.

Under review as a conference paper at ICLR 2020
THE ADVANTAGE OF USING STUDENT'S S T-PRIORS IN VARIATIONAL AUTOENCODERS
Anonymous authors
Paper under double-blind review
ABSTRACT
Is it optimal to use the standard Gaussian prior in variational autoencoders? With Gaussian distributions, which are not weakly informative priors, variational au- toencoders struggle to reconstruct the actual data. We provide numerical evi- dence that encourages using Student's t-distributions tional autoencoders, and we challenge the usual setup for the variational autoen- coder structure by comparing Gaussian and Student's t-distribution priors with different forms the covariance matrix.
default priors in varia-
INTRODUCTION
Variational autoencoders (VAEs) (Kingma & Welling 2013 Rezende et al.] 2014) are generative models that, by learning a joint distribution on all variables, aim reveal the process of how datasets are generated. Consider input data X = {xi...., X, where x E Rd and d is the feature dimension, and is-dimensional latent variables z. VAEs consist of two neural networks, inference and generative
networks (encoder qo (z(x) and decoder po px(x(z) that are jointly trained to maximize a variational lower bound log likelihood the data,
(og(po(x(z))) - DKL -=x_(o((2)x)]loof2 C(A, 6; = ()(2
(1)
The KI divergence term as the regularizer that minimizes the distance between (z(x) (ap- proximated posterior) and the suggested prior, p(z).
Despite the recent successes of VABs (Kingma al. 2014 Gregor et al. 2015 Kulkarni al. 2015
Hsu et al. 2017). there is still room for improvement concerning the approximation of the true pos- terior distribution. The main difficulty in VAE is the "posterior collapse". in which the term ds x) stays in the initial state, equivalent to random samples from the prior (Bowman et al.) 2015 Chen al.
2016 Kingma al. 2016) that results in uninformative latent variables. Various extensions to VAE models have been proposed to tackle this problem, such as weakening the decoder (Bowman
2015 Chen et al., 2016 Sonderby et al. 2016) or by changing the objective of VAEs (Chao 2017 Higgins et al. (017). Many of these studies aim to learn more expressive posterior, which results a tighter variational bound.
al.
al.
As a solution to the problem of the posterior collapse, we will explore different prior distribu- tions for the VAE, specifically various forms prior's covariance matrices. The standard prior, in connection with VAE, is the multivariate Gaussian with diagonal covariance. ((z(x) (<=]p((x, diag( (a" (x))). Using Gaussian diagonal covariance results in Mean-Field Variational Inference (MFVI). MFVI is an approximation method that was adopted from mean-field theory (Op- per & Saad 2001) and introduced by (Peterson) 1987) when training neural networks. It considers a
fully factorized variational distribution, in the VAE case (z) II do (2;), ), that is sufficient to ap- proximate the posterior and provide a tractable approximation being the size of the latent space). However, these models may lead to the poor hidden representation since they ignore all correlation between latent variables (Hoffman & Johnson 2016).
Recently, it has shown how to use the Student's t-distribution prior when training VAE's and that it has positive effect on the reconstruction ability (Abiri & Ohlsson 2019). The covariance matrix used for that study limited to diagonal form. Here, we examine the effect of the covariance in priors by extending the VAE to allow for full covariance in both Gaussian and Student's families of priors. The models tested two different datasets with the intent highlight the advantages of

---

Under review as a conference paper at ICLR 2020
using a prior different from the standard diagonal Gaussian. Using the Student's t-distribution prior, the degree of freedom parameter can be shared between the prior and the posterior. Hence, we have an empirical Bayes method that provides the ability to update the prior.
2
METHODS
We investigate two possible families of priors for VAE, Gaussian, and Student's t. All priors in this work are standardized, i.e location zero and scale one. For the Gaussian family, besides the centered isotropic multivariate Gaussian, we also analyze the effect of multivariate Gaussian (MVN) with full-covariance on the model (both forms have been mentioned in the original VAE paper (Kingma & Welling ))
To set up the network to use full-covariance MVN, instead of optimizing the diagonal vector a with dimension d, we need to optimize the covariance matrix E with the dimension (d x d). Since E positive semi-definite we can use the Cholesky decomposition, Σ LL", " to only optimize the lower (or upper) triangular matrix L with the dimension of d(d+1) The KL. divergence of an MVN and normal prior is,
D L. (2)xx))) - log((E]) + tr(E) + p-
where j is the location of the distribution q, (z)x). This expression allows for a straightforward implementation where the full covariance matrix Σ can be updated using standard stochastic gradient descent techniques. Using the full covariance prior increases the number of parameters, and it grows as d², compared to linear growth for the standard diagonal Gaussian prior.
The next family is the Student's t-distribution. has been shown that (Abiri Ohlsson 2019), with the use of implicit differentiation of the cumulative function of the Gamma distribution (Figurnov) jet al.) 2018). it is possible to use a Student's t-distribution, which belongs to the location- seale family, as a prior for the VAE. In this work, we considered the of multivariate t-distributions (MVT) with a diagonal covariance, in which all the marginal distributions share the same degree of freedom parameter v. In this case, there is an analytical solution for the KI. divergence,
log( [) ("
+ d
- d
log(1+ + - 1) (3)
DKL (xL(g(z)x))lpof2
+ 2s
E
2
with ()))) being the digamma function, the latent dimension and Σ the diagonal covariance ma- trix. The expectation in equation can be approximated by numerical integration method (Abiri & Ohlsson 2019). Unlike Gaussian random variables where a diagonal covariance matrix cause independency, uncorrelated Student's t random variables are not in general independent (Kotz & Nadarajah 2004). This can be illustrated looking at the joint PDF of two Student's 's random vari- ables 21 and 22.
En
p(21,22) = St
22
122
0
E22
.V)
1+ 1
Σ (z+ - )
i (zs - pi)
x
1,1
which cannot factorize into a product of marginal densities. Similar to MVN, MVT can be extended to full covariance matrix by merely replacing in equation by the full covariance matrix Σ.
The last prior we consider in our study is batch of Student's 's t-distributions, obtained by stacking several independent t-distributions to form the latent space, each with separate parameters including the degree of freedom и. Since the joint probability of such batch does not have an analytical density function, we use Monte Carlo sampling to estimate the KI. divergence term,
logp(z) + log ((([x)
In this study we used single MC sample for this estimation.
Figure [ shows an illustration of the five different VAE priors, where a simple VAE with two inputs connected directly to a layer of size two. Each vertical row of (μ, Σ, v) parameters corresponds to one of the five priors. Table [ refer them by their abbreviations (shown by bold text).
shows a condensed summary of all five priors. From now on, we
2

---

Under review as a conference paper at ICLR 2020
X1
X1
x
X2
MVN-daa
MVT-5180
Batch-St
MVN-UUI
MVT-uIl
P
Figure 1: An illustrative comparison of all five priors for simple 2D input VAE. The latent di- mension two and the priors are parameterized using (j4, Σ, v), as indicated by the vertical rows. For the full covariance models Σ has three parameters that form lower/upper triangular positive semi-definite matrix. the marginals of MVT-ddig and MVT-full share one v, the Batch-St prior, on the other hand, has two independent v--parameters.
Table 1: Five different priors for VAE with their properties. For the posterior dimension, d refers to the size of the latent space.
VAE priors
Name
Posterior
Posterior dimension
KL divergence
MVN-diag
MVN with diagonal covariance
2d
Analytically available
MVN-full
MVN with full covariance
d(1
+
d+3
Analytically available
2
MVT--diag
MVT diagonal covariance
2d
+
Analytically available
MVT-full
MVT full covariance
d(1
+ d+3
+1
Analytically available
Batch-St
Batch of Student's
3d
sample MC
EXPERIMENTS AND DISCUSSIONS
We evaluated the five VAE priors with their specific structures on datasets:
Gaussian ovals: A 2D data set that contains 1500 samples generated data from bivariate normal distributions with diagonal covariances. The "clusters" are different in both shape and width.
Omniglot (Lake al. 2015): This data with overal dimensions of (32460, 105 105), contains characters from 50 different alphahets. 20 random handwritten samples represent each character. the original work, the data divided into two groups of 30 alphabets for training and 20 for testing purposes. In our study, we used the same division into training and test.
For the Gaussian ovals, used root squared error (RMSE), and for Omniglot data, used the structural similarity index measure (SSIM) (Wang et al. 2004). Unlike other image difference measurements, SSIM measures the perceptual difference between two similar images. The final SSIM here is the average over all images differences.
3.1 GAUSSIAN OVALS
The Gaussian ovals a 2D dataset where each sample (21, x2) is generated from one of five Gaus- sian distributions with randomly picked scale and location parameters. The dataset contains 1500

---

Under review as a conference paper at ICLR 2020
samples. For the experiments. repeated cross-validation was used, with approximately 70% for training and the remaining for the test. Figure 2 (left plots) shows one instance the Gaussian ovals test set.
Since 21 and 22 are independent, a simple VAE network with a latent layer of dimension two and no hidden layers should be able to reconstruct the input with small RMSE. A VAE model with this structure and MVN-diag as prior will end up with only 18 parameters (with bias nodes). To see the effect of using different values for the standard deviation of the MVN-diag prior, we trained four VAE models with standard deviations of [1., 0.5, 0.1, 0.01]. Each model was evaluated on the test data 10 times with random split of training and test data.
True data
VAE(MVN-diag) with blas
a= 1
a 0.5
0 0.1
G 0.01
1
1
-2
-1
-1 0
-1
-1 0
MSE:
0.183=0.01
0.167=0.02
0.14320.003
0.147=0.01
True data
VAE(MVN-diag) without blas
a=1 1
a= 0.5
a= 0.1
a= 0.01
0
0
0
-2
-2
i
i
-1 0
- i
-1 0
MSE:
0.781#0.012
0.55
0.21
0.144
Figure 2: Best viewed in color (color-coded). The leftmost plots show the test data (denoted True data) and the plots on the right show reconstructed test data using different values for a. Upper row: Results from the VAE model with bias nodes, Lower row: Result from the VAE model without bias nodes (in total 12 parameters). The presented MSE reconstruction errors are estimated using cross- validation, repeated 10 times.
Figure 2 shows one test set reconstruction of the input with the different a used when training the VAE (MVN-diag) model. The upper row of the figure shows reconstruction results for the VAE model with bias nodes, whereas in the lower row, all bias nodes were removed from the model. For both cases (with and without bias nodes), decreasing standard deviation improves the reconstruction and separation of the ovals. In this case, the VAE tends to a deterministic autoencoder and learns manifold. Since some of the ovals have been generated from Gaussian with small standard deviation, a normal prior with a = 1 is too informative and results in poor network reconstruction. This is most apparent for the models without bias nodes. The MVN-diag prior, apart from being too informative, may need tuning of the parameter increase performance. In practice, the default is probably most used. Using a weakly informative prior, such as the Student's 's t-distribution, will improve the reconstruction results without any parameter tuning for the prior.
For the next set experiments on the Gaussian ovals, all five VAE priors from Table were tested. All VAE models were trained without any bias nodes, and from now on the prior's standard deviation was fixed to to compare the effect of using different prior distributions. Again MSE reconstruction emrrr were estimated using ten random training and test data splits.
4

---

Under review as a conference paper at ICLR 2020
True data
MVN-diag
MVN-full
MVT-diag
MVT-full
Batch-St
-2
-2
-2
-2
-2
-10
-10 1
-10 1
-10 0.12
-10 0.11#0.05
-10 0.0820.02
MSE:
0.75110.012
0.3+0.01
Figure 3: The leftmost plot shows the test data (denoted True data), and the plots the right show reconstructed test data using VAE models with five different priors. All models were im- plemented without any bias nodes. The presented MSE reconstruction errors estimated using cross-validation, repeated 10 times.
Figure 3 shows reconstructed test data (one instance) for all five prior distributions. The smallest average MSE was found for the batch of Student-t priors (Batch-St). Since the real variance of Batch-St for each 2j is oiv -ーー and vi. the degree of freedom for each of the Student's is shared between the prior and posterior, the model can update its prior during the optimization. This also the case for MVT-diag and MVT--full, except that all the marginal distributions share the same 1/, only single parameter updating in the prior, which is less powerful than Batch-St model. Although the reconstruction improved by using full covariance matrix for the Gaussian prior (MVN- full) compared to the standard MVN-diag prior, both Gaussian priors perform worse compared to Student's priors. We have done similar analysis with the presence bias nodes in all of the VAE models. With bias nodes, the error drops for all models with an RMSE s 0.02. By observing the posterior learned parameters, one can conclude that the two Gaussian priors stay close to the initial setting, while the Student's priors do learn even when bias nodes are present.
Finally, is intuitive to observe the distribution of latent space for all models. Figure shows the priors (top row) and the learned marginal distribution q(z) (bottom row) after training. All models, except MVN-diag, allow for a better fit between the posterior and the prior and improve the flexibility of qa (z)x), which results in better reconstruction that resembles the underlying input data clusters.
3.2 OMNIGLOT DATA
For the Omniglot data, we used the proposed fixed division training and test data. The training data contains 19280 samples (characters) from one group of alphabets, and the test data contains 13180 samples from another separate group of alphabets. This experiment aimed to study the effect of using the five different VAE prior distributions for more complicated image data set. The VAE structure still contained very simple encoder and decoder part to put the focus on learning an appropriate posterior distribution. Hence, the 2D input layer was directly connected to the latent layer, with selected size of 25, which was then directly connected to the 2D output layer. In case of, e.g., the Student's diagonal (MVT-diag) prior, this resulted latent layer with 51 parameters, 25 for each of the location (n) and scale (E) parameters and one common degree of freedom (v). No bias nodes were used in the VAE models. All models were trained with the same set of common hyperparameters, mini-batci size of 32, learning rate 10 and the number of epochs equal to 1000.
Figure (left plot) shows examples of four different characters: the original image and VAE gen- erated images for the different priors. Visually, the VAE with Gaussian diagonal prior (MVN-diap produces more blurry images compared to the other prior distributions. This can also be seen by comparing test set average SSIM values, where the MVN-diag prior clearly generating images with lower SSIM. Even though the difference between e.g., Satch-St and MVN-full is small, 0.793

---

Under review as a conference paper at ICLR 2020
MVN-diag
MVT-diag
Batch-St
MVN-full
MVT-full
Priors
True data
MVN-diag
MVT-diag
Batch-St
MVN-full
MVV-full
Latent codes
Figure 4: This figure shows the priors and posteriors of latent variable of all five models. Lower left shows the distribution of the data. The top row contains priors that have been used for the models, and the second row shows the marginal distribution q(z) of the models encoder after training.
Data
MVN-diag
MVN-full MVT-diag MVT-full Batch-Sc
0.8
0.5
6
5
5
0.2
MVN-dlag
MWW full
MVT-diag
MVT
Batch St
S5IM:
0. 747
D.79
0.79
0.791
0.793
Figure 5: Left figure shows distributions of SSIM (higher better) for each method on x-axis. Right figure shows reconstructed samples from four random characters from test data. The numbers on the last row are the SSIM between the original character and the reconstructed one.
compared 0.790, a paired Mann-Whitner U shows significant difference in favor of the Batch-St prior. Note that in this experiment, we did not optimize the VAE architecture for optimal SSIM between original and generated images. The purpose away from the MVN-diag prior, where either MVT-ddig or Batch-St prior distributions are likely to be better.
show the advantage of moving
One consequence of using the full covariance priors is the increase of parameters. We have timed the training and testing process for all models on the Omniglot data quantify the increase in computational cost when adding more parameters. All calculations were carried out machine with single NVIDIA GV100 (TITAN V) graphics card.
Table 2: Training and testing times for the all five VAE models using the Omniglot data.
Methods
MVN-diag
MVN-full
MVT-diag
MVT-full
Batch-St
Time (second)
1220
2070
1240
2200
1730

---

Under review as a conference paper at ICLR 2020
The result in 2 shows the approximate training and testing time for each method. As expected, the full covariance models requires more computational time due to more parameters in the latent layer. The Batch-St prior requires less parameters compared to MVN-full and MVT-full, but the computational cost depends heavily on the MC sampling of KI. divergence. Here we sample once, making Batch-St faster compared to the full covariance models.
CONCLUSION
We have investigated the effect of using different prior distributions for the variational autoencoder. Besides the standard diagonal Gaussian prior (with location zero and scale one), we have used Gaussian prior with full covariance, as well as the Student's 's prior, both the diagonal and full co- variance structure. Finally, tested batch of independent Student's priors. We have deliberately used very simple VAE structures to focus on the effect different priors in latent variable learning.
By comparing the two Gaussian priors, observed clear advantage of using the full covariance structure, with the drawback of requiring more parameters. Using a weakly informative prior such as the Student's with lower number of parameters compared Gaussian is advantageous for VAE. Furthermore, our setup of the Student's t-priors allows VAE update the prior's 's scale parameters, providing flexible posterior approximation. All three Student's 's t-priors show similar performance on the test data, with slight advantage of using batches of Student's t-priors (Batch-St). This distribution, however, has the drawback of needing Monte Carlo estimates of the KI. divergence term due to there being no analytical form of the density function. The diagonal Student's 's t-prior distribution in this respect better alternative, also requiring fewer parameters.
more informative full covariance
empirical Bayes model and results in a more
Regardless of the exact choice Student's t-prior, the main message is that standard diagonal Gaussian prior is far from optimal for the variational autoencoder. It is highly recommended to use a Student's 's t-distribution prior.
ACKNOWLEDGMENTS
The authors would like to thank Professor Carsten Peterson and Pontus Fyhr for valuable discussions and comments.
REFERENCES
Najmeh Abiri and Mattias Ohlsson. Variational auto-encoders with Student's t-prior. Proceed- ings 27th European Symposium Artificial Neural Networks, Computational Intelligence and Machine Learning, pp. 415 420. ESANN, March 2019.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Ben- gio. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349, 2015
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. arXiv preprint 1rX77.1611.02731, 2016.
Mikhail Figurnov, Shakir Mohamed, and Andriy Mnih. Implicit reparameteriizatin gradients. In Advances in Neural Information Processing Systems, pp. 441-452, 2018.
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. arXiv preprint rrXi::1522.04623 2015.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta- VAE: Learning Basic Visual Concepts with Constrained Variational Framework. ICLR, 2(5):6, 2017.
Matthew D Hoffman and Matthew J Johnson. Elbo surgery: yet another way to carve up the varia- tional evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS, volume 1, 2016.
7

---

Under review as a conference paper at ICLR 2020
Wei-Ning Hsu, Yu Zhang, and James Glass. Learning latent representations for speech generation and transformation. arXiv preprint arXiv:1704.04222, 2017.
Diederik P Kingma and Max Welling, arXiv: 14, 2013.
Auto-encoding variational Bayes. arXiv preprint
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pp- 3581-3589, 2014.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. In Advances in neural information processing systems, pp. 474,
Samuel Kotz and Saralees Nadarajah. Multivariate r-distributions and their applications. Cambridge University Press, 2004.
Tejas Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. In Advances in neural information processing systems, pp. 2539-2547, 2015.
Brenden Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 15006266) 2015.
Manfred Opper and David Saad. Advanced mean field methods: Theory and practice. MIT press, 2001.
Carsten Peterson. A mean field theory learning algorithm for neural networks. Complex systems, 1: 995-1019, 1987.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv: 1003:::4001408.. 2014.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In Advances in neural information processing systems, pp. 3738-3746, 2016.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 13(4):600- 612, 2004.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Information maximizing variational autoencoders. arXiv preprint 1rXiv:1706.02200 2017.
8


Reviews:
[{"review_id": "SygHTGnhYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes using more expressive priors in variational autoencoders (VAEs), such as the multivariate Student’s t-distribution. The experiments demonstrate a small improvement of SSIM metric on Omniglot.\n\nExploring various design choices of the VAE, including the prior distribution, is important, However, I feel that the paper does not have a sufficient contribution. The theoretical novelty is small; the experimental evaluation is lacking and missing crucial baselines, such as normalizing flows. The text is also not very well-written. Because of these factors, I think the paper should be rejected.\n\nIn more detail, I see the following issues with the current manuscript:\n1. The manuscript is not properly anonymized as it includes acknowledgements.\n2. The title of the paper is confusing, as it suggests that only the prior is being changed. However, Table 1 shows that the posterior is changed jointly with the prior.\n3. The novelty compared to (Abiri & Ohlsson, 2019) is very small. The difference is replacing diagonal covariance multivariate Student’s t-distribution with similar distributions, such as the full covariance Student’s t-distribution or a batch of Student’s t-distributions. The empirical improvement from these changes, based on Figure 5, is quite small.\n4. The text of the paper can be significantly improved. There are many typos, e.g. “lower bound log likelihood” instead of “lower bound on the log-likelihood”. There are also issues in the math, such as a missing negation in the KL divergence at the bottom of Page 2. Some things are not specified: how exactly the diagonal values of the Cholesky decomposition are enforced to be positive? what are the bias nodes mentioned in Figure 2? what is the multivariate Student’s t-distribution density function?\n5. One thing I didn’t understand is why the KL-divergence of a product of independent (“batch”) t-distributions cannot be computed analytically using equation (3), given that the KL(prod_i q_i || prod_i p_i) factorizes as \\sum_i KL(q_i || p_i).\n6. The experimental protocol is lacking. A key missing comparison is normalizing flows (https://arxiv.org/abs/1606.04934) which are typically used to make the VAE prior distributions more expressive. There are also VampPrior (https://arxiv.org/abs/1705.07120) and LARS (https://arxiv.org/abs/1810.11428) which also augment VAE’s prior in various ways. Another issue is that only the SSIM metric is reported, whereas the standard way of comparing generative models is the test log-likelihood.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJg7nEQatr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper advertised the use of student-t prior, instead of the standard normal prior, in variational autoencoders. The authors listed a few candidates, including the diagonal and full covariance matrices. Several experiments on two datasets were then performed.\n\nIn general, this paper is confusing and difficult to follow. For example, the same terminology has different meanings across this paper. The novelty in this paper is also quite limited, as the main technical part is basically to list Equations (2) and (3) to show the KL divergence for normal and student-t distributions. Finally, the experiments need to be significantly improved, as there are not sufficient takeaways.\n1. The authors used the word \"prior\" to denote different distributions, e.g., p (z) and q (z | x). This is pretty confusing, as q (z | x) is usually treated as the posterior. In Table 1, the posterior was even described under VAE priors. It would be better for the authors to read the original VAE papers (e.g., Kingma & Welling, ICLR 2014) carefully to understand these terminologies: https://arxiv.org/pdf/1312.6114.pdf\n2. The contributions of this paper are far from sufficient: In Section 2, the main takeaway for me is the list of KL divergence from different distributions, which is pretty standard and difficult to be treated as the authors' own contributions.\n3. I feel confused about the purposes of Figures 2 and 3. First, they were solely focused on reconstruction tasks, which may not be interesting enough, especially on such a toy dataset. Furthermore, the standard deviations in Figure 3 were set to be one, which may not be fair for MVN-Diag, as Figure 2 suggests that one is a bad choice. As a result, it's not convincing to conclude that student-t is the best candidate.\n4. Again, the \"prior\" used at the top row of Figure 4 is difficult to understand. How did you compute the scale parameter and plot the figure? ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1etdm5itr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:  This paper studies alternative priors for VAEs, comparing Normal distributions (diagonal and full covariance) against Student-t’s (diagonal and full covariance).   In particular, the paper is concerned with posterior collapse---i.e. posterior remains at the prior, limiting the model’s ability to reconstruct the data.  Experiments are performed on a synthetic 2D dataset, ‘Gaussian ovals,’ and on OMNIGLOT.  Results primarily take the form of visualizations of the reconstructed data and MSE / SSIM numbers. \n\nPros:  Systematic study and investigation of alternative priors for deep generative models is an under-studied area.  Moreover, heavy-tailed priors such as the student-t---while widely successful for robust regression---have not been explored as extensively for latent variable models, to the best of my knowledge.  This paper makes some steps towards solving these open problems.  \n\nCons:  I have two primary critiques of the paper: (i) the experimental hypothesis is unclear, (ii) no engagement with the results of Mathieu et al. [ICML 2019], who also study Student-t priors for reconstruction (and disentanglement).  \n\nRegarding (i), the paper seems to be testing two hypotheses simultaneously: the effect of diagonal vs full covariance matrices and exponential-tailed vs heavy-tailed priors.  The latter seems more crucial for purposes of reconstruction according to Figure 3 (since the diagonal St-t has good reconstruction).  Yet a proper study of the effect of tails vs reconstruction would report results as the degree of freedom parameter is gradually increased (as this controls the tails directly).  No careful ablation study of this sort is performed.  For comparison, see Figure 2 of Mathieu et al. [ICML 2019].  Moreover, the text simply calls the student-t a “weakly informative prior,” but it needs to be much more specific about what characteristics of the student-t are crucial.  If all we require is something “weakly informative,” why aren’t alternatives like a diffuse Gaussian or uniform also considered?\n\nRegarding (ii), Mathieu et al. [ICML 2019] also study priors formed by products of student-t marginals, but their work is not cited.  Mathieu et al. [ICML 2019] also show that, due to student-t’s not being rotationally invariant (unlike the diagonal Gaussian), they improve disentanglement with only a minor degradation of reconstruction.  As this work also studies reconstruction in student-t VAEs, it should include some discussion of Mathieu et al. [ICML 2019]’s results---if not direct engagement with their hypotheses.       \n\nFinal Evaluation:  While I like the general motivation for this work, there are no clear experimental hypotheses being tested in the experiments.  \n\nMathieu, E., Rainforth, T., Narayanaswamy, S. and Teh, Y.W., 2018. Disentangling Disentanglement in Variational Autoencoders. ICML 2019.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The consensus among all reviewers was to reject this paper, and the authors did not provide a rebuttal.
Decision: Reject