Shape your Space: A Gaussian Mixture Regularization Approach to Deterministic Autoencoders
Authors: Amrutha Saseendran
Kathrin Skubch
Stefan Falkner
Margret Keuper
Venue: NeurIPS 2021 Poster (2021)
Keywords: Generative Model
Variational Autoencoder
Gaussian mixture model
Regularization

Abstract: Variational Autoencoders (VAEs) are powerful probabilistic models to learn representations of complex data distributions. One important limitation of VAEs is the strong prior assumption that latent representations learned by the model follow a simple uni-modal Gaussian distribution. Further, the variational training procedure poses considerable practical challenges. Recently proposed regularized autoencoders offer a deterministic autoencoding framework, that simplifies the original VAE objective and is significantly easier to train. Since these models only provide weak control over the learned latent distribution, they require an ex-post density estimation step to generate samples comparable to those of VAEs. In this paper, we propose a simple and end-to-end trainable deterministic autoencoding framework, that efficiently shapes the latent space of the model during training and utilizes the capacity of expressive multi-modal latent distributions. The proposed training procedure provides direct evidence if the latent distribution adequately captures complex aspects of the encoded data. We show in experiments the expressiveness and sample quality of our model in various challenging continuous and discrete domains. An implementation is available at https://github.com/boschresearch/GMM_DAE.

Shape your Space: A Gaussian Mixture Regularization Approach to Deterministic Autoencoders
Amrutha Saseendran', Kathrin Skubch', Stefan Falkner and Margret Keuper Bosch Center for Artificial Intelligence University of Siegen, Max Planck Institute for Informatics, Saarland Informatics Campus Amrutha. Saseendran@de bosch. com com
Abstract
Variational Autoencoders (VABs) are powerful probabilistic models learn rep- resentations of complex data distributions. One important limitation of VAEs is the strong prior assumption that latent representations learned by the model follow a simple uni-modal Gaussian distribution. Further, the variational training procedure poses considerable practical challenges. Recently proposed regular- ized autoencoders offer deterministic autoencoding framework, that simplifies the original VAE objective and is significantly easier to train. Since these mod- els only provide weak control over the learned latent distribution, they require an ex-post density estimation step to generate samples comparable to those VAEs. In this paper, we propose simple and end-to-end trainable deterministic autoencoding framework, that efficiently shapes the latent space of the model during training and utilizes the capacity of expressive multi-modal latent distri- butions. The proposed training procedure provides direct evidence the latent distribution adequately captures complex aspects of the encoded data. We show in experiments the expressiveness and sample quality of our model in various challenging continuous and discrete domains. An implementation is available https: //github. com/bos wwchchssseechhhhceeece B/MM_DAGMMMDD
Introduction
Variational autoencoders (VAEs) constitute one of the popular generative learning frameworks widely used for applications such image understanding and generation, sentence modeling, and optimizing discrete data and graph-based structures 7 [3 40 48). The VAE framework elegantly combines autoencoders with variational inference [24] The encoder of the model maps the input data into a lower-dimensional latent space according to a given inference model. The decoder provides a mapping from the latent space back the original input space. Both are jointly optimized by maximizing a lower bound on the model evidence, regularizing the latent space towards a fixed prior distribution, usually uni-modal Gaussian. By sampling from the latent space prior, we can utilize the decoder network to efficiently generate new samples from the training distribution. Due the variational formulation, optimizing the VAE training objective poses significant practical challenges. Further, the over simplistic prior assumption often leads to an unsatisfying trade-off between the quality of reconstructed samples and the prior regularization (2] Recent work has shown that choosing more flexible priors helps to improve the generative performance of VAES
Since the initial introduction of VAEs, various novel training objectives have been proposed. One line of work focuses on different regularization techniques derived from alternative probabilistic metrics to shape the latent space of the model during training, e.g. using the Wasserstein distance [43] In contrast to the KL-divergence, the Wasserstein distance measure induces a metric on probability
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

distributions. Practically, this facilitates smoother convergence even for initially non-ovellap distributions. Further, it overcomes the regglaaalaiinnnoon effect in VAEs. To be precise, it prevents the undesired behaviour of multiple data points being mapped to the same latent representation by the encoder. Since closed-form solutions for metries like the Wasserstein distance can only be derived for very few prior distributions, these approaches rely on numerical approximations during training.
Recent work by Ghosh et al. [2] reinterprets deterministic autoencoders as variational models, even when trained with a deterministic loss. During training, this approach maximizes the negative log- marginal likelihood of the latent samples under Gaussian normal distribution as regularization in addition to minimizing the reconstruction loss. Experimental results show that this regularization alone does not suffice to generate high quality samples using the Gaussian prior. To overeome this, Gosh et al. propose to use a multi-modal Gaussian mixture model (GMM) to fit arbitrary, learned latent spaces. While this approach leads to good sampling efficiency and generalization if the post-hoc fit is reasonable, sampling quality can suffer significantly if the learned latent space can not be modeled well by a GMM.
In this work, we propose a deterministic training scheme for autoencoders that is applicable to expressive priors and overcomes the necessity of a post-hoc density estimation step for deterministic training. To be precise, we derive a deterministic regularization loss from the distance metric used in the non- non-parametric Kolmogorov-Smming -mirmoo (KS) test for equality of probability distributions. The resulting training objective can be derived in closed form for class of expressive multi- Modail prior distributions and provides a strong signal to efficiently shape the latent space of the model during training. We chose our experiments to evaluate the proposed approach in terms of sampling quality and expressiveness. In the first line of experiments, we compare the quality of newly generated and reconstructed samples from our model with those from variety of other VAE variants. In the second line, we investigate our method's s capability to model discrete and complex structured inputs such as arithmetic expressions and molecules. In these domains, VAEs have recently been proposed as a tool for dimensionality reduction in optimization. Applying our regularization scheme effectively utilizes multi-modal prior distributions in this context and significantly improves optimization performance.
2
Related Work
Since the introduction of VABs, many follow up works tried to overcome the practical and theoretical limitations of the framework, e.g. 2 43 44 and make them applicable to specific applications such as clustering 6 (39 or anomaly detection 49). We first review some seminal examples of VAE models with different priors and probability metrics for latent regularization. Since our proposed regularization term structures the latent space to a Gaussian mixture model, we also compare it to prior work on deep clustering. Lastly, we discuss VAEs in the context of black-box optimization approaches such as Bayesian Optimization (BO).
VAEs In the standard VAE framework, the prior distribution is commonly assumed to be a Gaussian normal distribution. This might lead to simplified representations Icarned by the model which is unable to represent the rich semantics in the data distribution. Several methods were proposed to introduce more flexible and expressive priors to the VAE formulation. Casale et al. employ Gaussian process priors to account for correlations between the data samples. In [15] a Bayesian non-parametrie prior is used with a hierarchical non- non-parametric variational autoencoder for video representation learning. Chen et al. [4] use an auto-regressive prior to achieve improved generative performance on image datasets.. Berger et al. (2) propose to replace the standard spherical Gaussian prior with a more general version with an arbitrary covariance matrix and learn the correlations by optimizing the evidence lower bound of the model. Although the proposed methods offer competitive performance, they often employ complex architectures 4 to achieve desired performance.
In another line of work, multi-modal priors were utilized in VAE models. Zong et al. 49 propose to use a GMM prior in autoendocers for unsupervised anomaly detection by training an additional network estimating the parameters of the GMM. Lee al. (27 address unsupervised meta-learning using a GMM prior in VAES to shape the latent space by employing an extension of the evidence lower bound to complex variational inference schemes. Tomezak et al. 44 propose to replace the GMM prior by coupling of the posterior and prior of the model. Adversarial autoencoders (31] improve the generative performance of VAEs by incorporating adversarial learning into the VAE framework and offer competitive performance in image generation at an increased computational complexity and

---

decreased training stability. To account for the over regularization effect of the KL divergence term in the standard VAE framework, 44 minimize the Wasserstein distance between the representations learned by the model and the target prior. Recently introduced regularized autoencoders [22 (R (RAs), question the variational framework adopted by the VAEs and propose a deterministic approach to achieve comparable or better image generation performance. The authors use the negative log likelihood for regularization, but require post-hoc step to derive a strong sampling procedure from the model. The state of the art VAE model for high fidelity image generation, VQ-VAE [37 4]], can be also considered as a deterministic autoencoder. Similar to RAEs, training VQ-VAE involves two stages of training relying complex discrete autoregressive density estimators. Moreover, the training loss of "Q-VAE is dindeefferentiblee due to the quantization of the latent vector.
Our approach elegantly combines the idea of new training objectives with the extension to multi- modal priors without increasing training complexity or compromising sampling quality. We derive a strong training signal which can be derived in closed form for multimodal priors. This ensures stable training and reliable regularization of the latent space, improving sampling quality.
Deep Clustering Deep Clustering approaches benefit from well structured latent spaces. Thus, several methods employ Gaussian mixture VAES for data encoding 6 39 or establish a GMM-like latent space structure through k-means models in the latent space. For example, Xie et al. [46] train an autoencoder and apply KL-divergence loss for better k-means clustering while Ghasedi et al. combine the autoencoder reconstruction loss with the relative cluster entropy. Similar approaches Caron al. [33 iteratively group points using k-means during optimization. In the context of clustering, generative adversarial networks have been considered in 11 35
have been proposed in the literature [8] 16 I8
While we are not considering the clustering task in this paper, we hypothesise that the proposed regularization can be beneficial in this context since it implicitly optimizes for mode assignments.
Structural VAËs and optimization High-dimensional optimization problems in structured discrete input domains are ubiquitous. VAEs have been used in this context to learn low dimensional, continu- ous representations of high dimensional, structured data like molecules arithmetic expressions. Recent work proposes use such representations to perform efficient optimization by running BO in the latent space of VAEs 22 30]. In this setting, prior knowledge of the structure of the latent space crucial to allow for an efficient exploration and generation of valid samples. Yet, as discussed above, VAEs can suffer from simplistic prior assumptions. Thus, sampling from the latent space of such models can result in invalid samples, reducing the sampling efficiency of BO [17.. Kusner et al. (25) overcome this issue if data follows specific grammar. Lu et al. [30] propose VAE that directly works on parse trees from context-free grammars to represent discrete data. Yet, those only work with unimodal priors which limits the generalization capabilities. Our approach can be readily
used to extend these models to better encode structural data and improve BO performance.
3 Method
We introduce novel loss function to regularize the latent representation learned by deterministic autoencoders towards a given prior distribution. The definition of our loss builds on the non paramerett statistical Kolmogorov-Smiroov (KS) test for equality of one-dimensional probability distributions. We propose a multivariate variant of the distance measure used in the KS test, that allows for gradient based optimization and can easily be applied to expressive multi-modal prior distributions. For ease of exposition, we start with introducing our regularization loss for unimodal Gaussian priors in section 3.1 and extend the formulation to expressive multi-modal Gaussian mixture models in Section 3.2 Finally, in Section 3.3 we provide an explicit way to estimate the weighting parameters of our loss.
3.1
Uni-Modal latent regularization
The KS test can be used to determine whether a collection of N, one-dimensional samples follow given reference distribution. It compares the cumulative distribution function (CDF) of the ref- erence distribution with the empirical CDF F(N) of the samples. It is often applied to the class of one-dimensional Gaussian distributions, which has important analytical properties. For spherical Gaussians, the one-dimensional KS test quantifies distance between the empirical distribution
3

---

function of the data and the cumulative distribution function
-(t -(t-1)) [=) 20
((2) 2V
cxp
dt
(I)
of the univariate Gaussian Z ~ (p, σ) as SuP (2) - - Extending this KS distance to higher dimension is particularly challenging, since it requires matching joint CDFs [10 14 33). Especially in higher dimensions this becomes infeasible [29]. The continuous ranked probability seore 133 shares the same theoretical basis the KS distance. However it tests whether two sets of samples are consistent with each other, i.e., they could originate from the same distribution, and is thus not suitable to regularize collection of latent samples towards given prior distribution. Alternative multi-variate normality tests, like the Mardia test [2] and the BEHP test [ suffer from slow convergence rates.
To derive a regularization loss from the KS distance, we propose to overcome this issue by taking into consideration the marginal CDFs and correlations in the prior distribution separately. Given d-dimensional latent samples Zi..... Zn. the empirical marginal CDF in dimension j is given by
F
(2)
1 2n,
We aim to regularize the latent space of our models by comparing the empirical marginal CDFs with the one-dimensional CDFs of the marginal distributions of the prior. To strengthen the training signal of our regularization scheme and make it suitable for gradient-based optimization, the supremum in the original KS distance by a smoother MSE loss, that compares the distances between those functions at the latent representations. For uni-modal Gaussian prior with mean μ and covariance matrix E, this results in
replace
1 Σ MSE d
Zj Σ j.
P3
(xs(Z2 NN)
(2:1i) and ((z)) ) is defined accordingly. This loss is minimized, if the empirical marginal CDFs of the latent samples match those of the uni-modal Gaussian prior. Using the above loss alone will not account for correlations between different latent dimensions. In the case of spherical Gaussian prior with identity covariance matrix for example, samples with perfectly correlated Gaussian components [z,], = [2, will also minimize this objective, see Figure To overcome this problem, we equip our loss with an additional term, that matches covariances between different latent distributions explicitly. Following similar reasoning to the MSE above, we define an additional loss term,
Here, (((N(2) (zj) denotes the vector with entries (x(N (
d
Lev(zi....))
DiI [Σ] [=),))
(4)
d
1J=1
where Σ is the empirical covariance matrix of the latent representations and Σ stands for the prior covariance. Compared the negative log marginal regularization proposed in [12), our loss will
actually enforce the latent representations to be spread across the entire support of the Gaussian prior, instead of being minimal when all latent collapse to the origin.
3.2 Multi-Modal latent regularization
One advantage of our approach the applicability more expressive, multi-modal prior distribu- tions. While the Gaussian distribution has important analytical properties, it suffers from significant limitations when modelling real data sets. In contrast, linear combination of Gaussians can give rise to very complex densities while still allowing for closed form computations of important quantities, like CDFs and covariances. d-dimensional K-modal Gaussian mixture model a weighted super- position of K Gaussian distributions in Rd, that are often referred to as the modes of the model. For & K, let b4 and Σg be the mean and covariance matrix of the k-th mode in the model. Further, let Pk > 0 be the weight of the k-th mode. Then, the marginal CDFs of a GMM model can be computed

---

perioctly comoloree Garsiann
weghel westrogggeeettooo
10 Saamm
Figure I: Uni-modal latent regularization in one and two dimensions for varying numbers of samples (x-axis) from different distributions: In two dimensions (right), the simplistic KS distance can not differentiate the target prior (blue) from other probability distributions. By contrast, proposed regularization scheme successfully matches correlations across different dimensions.
from the CDFs of univariate Gaussians as follows
K
[u
FGMM)(2)
Σ
Pa
(5)
Le. the marginal CDFs the GMM are weighted sums of CDFs of one-dimensional Gaussians. The covariance matrix of the GMM can be computed as
ΣGMM Σ Pk Σ
pk (1 (1) (14 -²)²
Extending our proposed regularization scheme to multimodal GMMs is straight forward. Our first loss term is defined as
LKS,K CKs,k(Z1.....))
Σ MSE (2)) (2.))
Similarly, the second loss term is defined be
1 [E - [EMM)).))
LCv, (ZI...Z
1J=1
The total loss of the model is combination of the reconstruction loss and a regularization loss, that enforces the latent representations of the encoded data to match predefined multi-modal prior distribution. The reconstruction loss LREC FREec(((.... (x'....v) equals the mean squared error between inputs x; and their reconstructions xt. Given positive weights AKs and λcv, final loss is given by
C(xi.....V) = ARECCREC(X + AKS (ZI....)) +
Formally, the weights ÄKS; λcy and ÀREC are hyperparameters of the model. Nevertheless, we propose an explicit way to set λςs and λcv and simple heuristic to estimate λREC to avoid an extensive optimization of these weights.
3.3 Loss weight estimation
Balancing the two regularization losses appropriately poses key challenge as they potentially vary on very different scales. For example, modes the GMM prior are far spread, the covariance Lcv. . loss will dominate the marginal CDF LKS,K loss by far. Nevertheless, given target GMM prior, the
5

---

dimension of the latent space and the batch size used during training, there is concise way to fix
those hyperparameters beforehand. To be precise, for m 1,..,, ,....,M. M samples z ((m) N the prior GMM, we propose to set
(m)
from
X M
LKS (2 (m)
X M Mfov (2im!..)
1
(10)
not overcome the necessity of tuning the weight of the reconstruction loss, which has significant impact on performance of the model. Nevertheless, reasonable approximation to can be obtained by training autoencoder model and using the inverse of the best obtained loss for AREC. Using this scaling, all loss terms in our regularization loss will ultimately converge to one the target prior is matched successfully.
Formally,
4 Experiments
With experiments strive investigate the potential of the proposed model when compared to other VAE variants in generating new samples, analyse the effect of the defined prior to effectively cluster the latent space and to shape the latent space efficiently in highly structured domains such as discrete spaces. We provide all the experimental settings and hyperparameters used in the Appendix. All experiments were run on a GPU cluster, with single GPU per individual experiments. Since the cluster is part of neetral framework, these experiments did not contribute to climate change.
4.1 Image generation
We consider four dataset, MNIST [66. FASHIONMNIST 45 SVHN [36 and CELEBA [28 to evaluate the proposed method in image generation experiments. The qualitative analysis of the generated samples for MNIST, SVHN amd CELEBA images are shown in Figure 2 along with the reconstructed samples and interpolated samples in the latent space of the trained model. In order to assess the quality of the generated images, we evaluate the Fréchet Inception Distance (FID) 19] for each dataset, see Table [ For baseline comparison, we evaluate the following models: vanilla variational autoencoder (VAE (24), Gaussian mixture variational autoencoder (GMVAE) [6), Wasserstein autoencoder (WAE) (33 with MMD loss, 2stage VAEs (2s- VAE) [5], constant variance- VAE (CV-VAE) [12] and regularized autoencoders (RAEs) [12. We consider the following evaluation metrics: 1. Sampling FID (Samp.)- FID score of the generated random samples (evaluated by generating random samples from the prior distribution of the respective models and by fitting& a Gaussian distribution to models trained without any prior assumptions). 2. reconstruction FID (Rec.) measured by computing the FID between the test samples and their corresponding reconstructions by the model and 3. interpolation FID (Inter.) - measured by computing the FID between the interpolated samples in the latent space and test samples. As pointed out by (112) fitting an ex-post density estimator on the learned embedding after training of VAEs further improves the generation quality. Hence, we also report the FID values by fitting GMM in the learned latent space of the trained model (GMM column in Table I not evaluated for 2s-VAI VAE as they perform ex- ex-post density estimation using another VAE).
As shown in Table T our method achieves better FIDs (Samp.) on all datasets considered, when compared to all considered baselines sampled by fitting a single Gaussian in the latent space. We also improved the generation quality as argued above by fitting a mixture of Gaussians in the latent space and achieve better FIDs in MÄIST, FASHION MNIST and CELEBA images, whereas for SVHN, WAES achieved the overall best score. It is also important to note that the proposed method performs comparably or even better without employing the ex-post density estimation. The proposed method also achieves better reconstruction quality than the other VAEs except for SVHN images where RAEs performs better. The interpolation FID indicates the overall structure of the learned latent space and the obtained FID values show that the proposed method shapes the latent space better than the other approaches except for the CELEBA images where RAEs performs slightly better than our method. For fair comparison, we use the same architecture and experimental settings in all the considered baseline evaluations. Please refer to the Appendix for more details on the experimental settings.
4.2 Unsupervised image clustering
We evaluate the potential of our method to naturally cluster the data points in the learned latent space in two dataset, MNIST and FASHION-MNIST. The Gaussian mixture model prior with k

---

MNIST
SVHN
CELEBA
8
7
(
0
6
6
12
69
6 190
8
4
8
0
4
7
50
74
7
09
S
9
5
8
3
8
2648
26
9
8
7
0
4
1
2
Y 2 4 1 6 6
7
106
6
1 15
15
Random samples
0
6
0
9
6
411
8
51
24
66
7
0
3
9
3
2
525
515
21
2
7
5 1
2
6
55
5
3
9
1666
4
5 7
10
16.9
6
2
10
4
2
6
477
3
6
12609
8
9 7
/ 6
1:2 2
347
14
3
Reconstructed Samples
8 9 7 5 /
6
2
3
7
4
3
5 2 41 / 5 2
4 1 /
3
7
66
14
55 1
6
3
6 14
8
9 0 8
9
5
9
0
ZA
1-
2
5 15
9 0 8
9
5
0
2
6
16
8
O 8
3
2
3
8
5
7
2
3
8
0
8
3
2
3
8
6
7
2
5
6
8
8
8
8
14
14
1
4
8
9
9
7
7
7
7 27
27
4
4
4
7
7
7
11:11:1111
Interpolated Samples
8
8
8
8
9
9
10
10 10
100
9
9
9
5
5
5
5
5
5
5
1
6
6
6
R
2
6
5
32
3
6
6
6
6
8
8
656666668
64
6
9
9
9
3
5
5 5
6
8
8
8
1
9
9
4
Figure 2: Qualitative analysis on image generation across datasets, MNIST, SVHN and CELEBA. Row 1 shows the randomly generated samples; row shows the reconstructed samples by the decoder on test dataset after training, first row each sections corresponds to the ground truth and the second one its corresponding reconstruction; row shows randomly interpolated samples in the learned latent space of our model.
components in our method could be considered as k different classes/clusters to which the data points are mapped by the encoder. We train the model with latent space dimension 10 for both dataset and visualize the random samples generated from each Gaussian component of our prior as shown in Figure The figure shows that visually similar images fall into the same cluster. For a quantitative analysis of the clustering performance, we evaluated the unsupervised classification accuracy (similar to 21) and compare the performance with JointVAE 9 and Cascade 21]. The observed values are reported in the table in Figure 4 We observed a comparable performance to both baselines. We also observed that the distance between the modes the GMM prior is deciding factor in better clustering performance. Figure @ (right) shows the performance comparison of both image generation
7

---

Table 1: Quantitative evaluation results across datasets. Samp. refers to the FID the generated samples from the prior distribution or by fitting Gaussian to the learned models trained without prior, GMM refers to the FID computed by fitting GMM on the learned model, Rec. refers to the reconstruction FID on test samples and Inter. refers to the Interpolation FID.
Dataset
MNIST
FASHION MNIST
Samp.
GMM
Rec.
Inter.
Samp.
GMM
Rec.
Inter.
VAE
27.27
20.52
21.59
21.05
50.50
36.22
33.33
14.12
GMVAE
21.35
20.61
20.21
10.23
38. 79
38.54
CV-VAE
32.12
28.62
29.61
30.76
57.57
38.28
35.10
47.73
26.99
23.77
22.13
46.47
31.93
41.06
RAE
17.72
14.15
14.69
15.57
47.26
29.59
24.54
34.77
Dataset
SVHN
CELEBA
Samp.
GMM
Roc.
Inter.
Samp.
GMM
Roc.
Inter.
VAB
61.01
58.23
59.13
50.29
68.01
61.63
52.55
58.39
GMVAE
49.74
48.65
47.15
65.35
64.22
64.92
WAF
58.08
34,87
29.62
27.16
58.91
49.17
41.14
47.08
CV-VAE
51.01
54.19
48.53
47.65
57.61
52.72
45.32
50.87
45.84
RAE
42.35
35.12
31.04
27:30
52.33
48.23
41.61
46.58
Ours
37.42
36.46
31.27
24.87
49.79
44.79
39.48
47.13
7
7
7
7
7
7
7
9
1
7
8
8
8
9
8
8
8
8
8
2
2
2
2
6
2
2
A
6
4
5
5
S
5
5
5
J
5
5 4 6
5
4
4
4
4
4
4
4
4
6
6
6
6
6
6
6
6
6 2 0 9
3
3
3
3
3
3
3
J
0
0
0
0
0
0
0
0
0 9
2
9
7
7
9
9
9
9
MNIST
FASHIOMNIST
Figure 3: Clustering performance on MNIST and FASHION-MNISY images with 10 component GMM prior. Each row the figure shows randomly generated images from different Gaussian components of the GMM prior. Similar looking images are mapped into same clusters.
and clustering performance with increasing distance between different modes in the GMM prior. The result shows that with increasing distance, the clustering performance improved wheres the quality of the generated images gets reduced. Our experimental analysis indicates that natural clustering happens with the multi-modal GMM prior in our method.
4.3 Modelling discrete data structures
In this section, we investigate the ability of our model generate complex discrete data structures such as arithmetic expressions and molecules. The objective of this experiments is to analyze the model performance on shaping the latent space such structured discrete spaces effectively. The learned latent space of the model is traversed to generate new samples with the desired properties by performing Bayesian Optimization (BO). We perform experiments in two sequence optimization problems similar to (25).

---

86
Method
Acc(t)
85
20
86-
MNIST
FASHION-MNIST
83
18
VAE
78.33
51.51
82:
Cascade VAE
84.19
57.72
16
Ours
85.53
56.24
80-
79
78
8
Distance between moces
Figure 4: Image clustering: (left) Unsupervised classification results on MNIST and FASHION- MNIST images, (right) Image Eaastering(Acuuray and generation performance(FID) on MNIST images with increase in the distance between modes in the GMM prior.
Arithmetic Expression Given dataset of 50, 000 univariate (functions of 2) arithmetic expressions following a formal grammar [25). the task is to find the expression that best fits a target dataset. This done by minimizing log(1 + MSF), where the MSE is computed between values of the generated expression and the target points. For our evaluation, we choose similar target data points as in [25).
Chemical Design Given the ZINC250K dataset of drug molechules [20]. the objective is generate new drug like molecules. The drug likeliness of molecule is quantified by the water-octanol partition coefficient, which is maximized in our line of experiments.
Results We extend the architecture and experimental settings of 25 to include our proposed losses during training. For baseline comparison, we consider Grammar VAE (GVAE) 255 Char- acter VAE (CVAF) [17], Grammar constant variance VAE (GCVVAE) [2] and Grammar based RAE (GRAE) 12] frameworks. The three best scores found by our method for arithmetic expressions and the molccule experiments are reported in Table Our model performs comparatively better than the considered baselines and achieves the best first score for both tasks. In addition to the optimization performance, it is also important to consider the validity of the new samples generated by the models. A well-structured latent space should yield valid samples following the defined grammar/rules of the used dataset. Our model achieves better validation and average scores as shown in Table 3 except for GCVVAE which achieves better average score in the arithmetic expression task. All reported values are evaluated by averaging across 5 BO trials.
Table 2: Best scores found by each method for arithmetic expression and molecule exper- iments. Baseline values reported from [1].
Table 3: Fraction of valid samples and their corre- sponding average scores for arithmetic expression and molecule experiments for each method. Baseline values reported from [12..
Method
Expressions
Molecules
Expressions
Molecules
1sk())
()
3rd(4)
Isk(t)
2nd(t) 3rd(?)
Frac, valid (f)
Avg. score (4) Frac, valid (t)
score (?)
score
GVAE
0.10
0.46
0.52
3.13
3.10
2.37
GVAE
0.99 0.01
3.26 0.20
0.28 0.04
-7.89 a 1.90
0.45
0.48
0.61
2.75
0.82
0.63
CVAR
0.07
-25.64 6.35
GCVVAE
0.39
0.40
0.43
3.22
2.83
2.63
GCVVAE
0.09
2,85 0.08
0.76 0.06
-6.40 + 0,80
GRAE
0.39
0.39
0.43
3.74
3.52
3.14
GRAE
1.00 0.00
3.22 0.03
0.72 2 0.09
-5.62 : 0.71
Ours
0.03
0.40
0.41
4.15
3.84
3.12
Ours
1.00 0.00
3.02
0.72 2 0.03
-5.08 1.30
4.4 Ablation study and hyperparameter sensitivity analysis
We perform ablation study the regularization loss terms in the proposed model. When the model is trained without the KS distance loss for MNIST images, we observed an FID of 49.82 and when trained without the covariance matching loss, observed an FID of 38.45. These values significantly worse than the FID that we achieve when training with the weighted combination of both regularization losses i.e 13.11. 11. These empirical evaluations show that the combination of the two regularization terms facilitates better prior-posterin match and hence better image generation. Please refer the Appendix for qualitative evaluation of the ablation study on MNIST images.

---

From a conceptual point of view, the most important hyperparameters of our model are (a) the weights of
40
the different terms in the training objective and (b) the number of components in the prior. section 3.4, we propose an explicit way how to fix the weights in the loss function. We investigate the sensitivity of our model performance to the number of compo- nents in the GMM prior. We trained our model on the MNIST dataset using GMM prior with 1, 5, 10, 15, 20 and 25 modes respectively. The observed FID scores for the respective number of components are shown in Figure The result show that with increas- ing number of components in the chosen prior, the performance of our model improves significantly. As consequence, choosing large number of compo- nents can be beneficial for practical considerations.
30
25
20
15-
10
Number components
Figure 5: Hyperparameter sensitivity analy- sis FID of the MNIST generated samples when model is trained with different number of components in the GMM prior.
5 Limitations and Future Work
One limitation of our work is the necessity to chose the prior distribution in advance. We showed that fixing suitable number of modes for the GMM is important to provide better sampling quality. Also, by considering marginal CDFs, we simplified the original distance metric from the KS test. While reducing computational complexity during training, this comes at the cost of additional loss term. Further, our proposed addition of the KS distance not suitable for matching higher order moments of the latent representations to the target prior, which at least from a conceptual point of view can lead to a mismatch to the prior. Further, our loss only facilitates matching empirical marginal CDFs of latent representations to the marginal CDFs of the prior evaluated at latent vectors. As consequence, our regularization loss might be less stable training signal for small batch sizes in high dimensions.
Using the Frobenius norm in the covariance matching loss reflects our assumption that the latent dimensions should all be independent from each other and should all simultaneously match the prior's values. The choice for the MSE for covariance matching loss is purely based on its prevalence in the literature. Additionally, this makes all three loss terms (reconstruction loss, KS distance loss and the covariance matching loss) behave similarly, as they are all squares. While we did not investigate any other metries for matrix comparison this scenario, exploring other options for the covariance matching is interesting area for future studies. We have not considered the case where there exists class imbalance in the dataset. We would expect the model to separate the classes the imbalance is weak and the classes are sufficiently different such that the reconstruction loss outweighs the regularization penalty for the mismatch. Extending our prior to accommodate for this by introducing weighted GMM prior is also very interesting direction for future work.
6 Conclusion
Recent studies have illustrated the effectiveness of flexible priors in VAES to learn more meaningful latent representations. Following recent work that highlights the potential of deterministic alternatives to the variational formulation in VABS, we propose a simple deterministic autoencoding framework with more powerful regularizers to accommodate for expressive multi-moda priors. In particular, we derive a novel deterministic regularization scheme from a strong metric on probability distributions.
The proposed approach can readily applied effectively shape the latent space of existing autoencoding frameworks towards multi-modal Gaussian priors. Our experimental evaluations show that the proposed training objective yields comparable sampling quality to those of variational autoencoders and achieves better performance in modelling complex discrete data structures.
10

---

References
[1] L. Baringhaus and N. Henze. A consistent test for multivariate normality based on the empirical characteristic function. Metrika, 35:339-348, 1988.
[2] V. Berger and M. Scbag. Variational Auto-Encoder: not all failures are equal. preprint, February 2020.
[3] P. Casale, A. V Dalca, L. Saglictti, J. Listgarten, and N. Fusi. Gaussian process prior variational autoencoders. In Proceedings of the 32nd International Conference Neural Information Processing Systems, NIPS' 18, page 10390-10401, Red Hook, NY, USA, 2018. Curran Associates Inc.
[4] X. Chen, D. P. Kingma, Salimans, Duan, P. Dhariwal, J. Schulman, I. Sutskever, and P. Abbeel. Variational lossy autoencoder. ArXiv, abs/1611.0273.1 2017.
[5] B. Dai and D. Wipf. Diagnosing and enhancing vae models. ArXiv, 0atas//933.050 003.05789, 2019.
[6] N. Dilokthanakul, P. A. M. Mediano, M. Garnelo, M. C. H. Lee, H. Salimbeni, K. Arulkumaran, and M. Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders. CoRR, abs/1611.00 1.02648, 2016.
[7] Z.. Ding, Y. Xu, W. Xu, Y. Yang, M. Welling, and Z. Tu. Guided variational autoencoder for disentanglement learning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7917-7926, 2020.
[8] K. G. Dizaji, A. Herandi, C. Deng, W. Cai, and H. Huang. Deep clustering via joint convolu- tional autoencoder embedding and relative entropy minimization. In Proceedings of the IEEE international conference on computer vision, pages 5736-5745, 2017.
[9] E. Dupont. Learning disentangled joint continuous and discrete representations. In NeurIPS, 2018.
[10] G. Fasano and A. Franceschini. A multidimensional version of the Salimogorov-Smminoo test. Monthly Notices of the Royal Astronomical Society, 225:155-170, 1987.
[11) K. Ghasedi, X. Wang, C. Deng, and H. Huang. Balanced self-paced learning for generative adversarial clustering network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4391-4400, 2019.
[12] Ghosh, M. S. M. Sajjadi, Vergari, M. J. Black, and B. Schölkopf. From variational to deterministic autoencoders. ArXiv, aws//903, 12436, 2020.
[13] T Gneiting and A. E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102:359 378, 2007.
[14] E. Gosset. A three-dimennsoonal extended Kolmogoroy-Smimov test as useful tool in astronomy. Astronomy and Astrophysics. 188(1):258-264 1987.
[15] P. Goyal, Z. Hu, X. Liang, C. Wang, E. P. Xing. and C. Mellon. Nonparametric variational auto-encoders for hierarchical representation learning. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 5104-5112, 2017.
[16] X. Guo, X. Liu, E. Zhu, and Yin. Deep clustering with convolutional autoencoders. In Neural Information Processing, 2017.
[17] R. Gómez-Bombarelli, D. Duvenaud, J. Hernández-LLobtto, Aguilera-Iparraguirre T. Hirzel, R. Adams. and A. Aspuru-Guzia Guzik. Automatic chemical design using data-driven continuous representation of molecules. ACS Central Science, 4, 2016.
[18] J. R. Hershey, Z. Chen, J. Le Roux, and S. Watanabe. Deep clustering: Discriminative embeddings for segmentation and separation. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 31-35, March 2016.

---

[19] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler. and S. Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. In NIPS, 2017.
[20] J. Irwin, T. Sterling, Michael M. Mysinger. Erin S. Bolstad, and R. Coleman. Zinc: A free tool discover chemistry for biology. Journal of Chemical Information and Modeling. 52:1757 1768, 2012.
[21] Y. Jeong and H. O. Song. Learning discrete and continuous factors of data via alternating disentanglement. In International Conference on Machine Learning (ICML), 2019.
[22] P. Ji, T. Zhang, H. Li, M. Salzmann, and I. Reid. Deep subspace clustering networks. In Advances in Neural Information Processing Systems, pages 24-33, 2017.
[23] W. Jin, R. Barzilay, and T. Jaakkola. Junction tree variational autoencoder for molecular graph generation. In ICML, 2018.
[24] D. P. Kingma and M. Welling, Auto-encoding variational bayes. CoRR, abs/1312.61 14, 2014.
[25] M. J. Kusner, Paige, and M. Hernández-Lobato. Grammar variational autoencoder. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, page 1945-1954, 2017.
[26] Y. LeCun and C. Cortes. MNIST handwritten digit database. 2010.
[27] D. B. Lee, D. Min, S. Lee, and S. Hwang. Meta-GMVAE: Mixture of gaussian VAE for unsupervised meta-learning. In International Conference on Learning Representations, 2021.
[28] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[29] R. Lopes, I. Reid, and P. Hobson. The two-dimensional kolmogorov-smimmg test. XI Inter- national Workshop on Advanced Computing and Analysis Techniques in Physics Research, 2007.
[30] X. Lu, J. Gonzalez, Z. Dai, and N. Lawrence. Structured variationally auto-encoded optimization. In Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3267-3275, 2018.
[31] A. Makhzani, J. Shlens, N. Jaitly, and I. Goodfellow. Adversarial autoencoders. ArXiv, hhs/1511.056444 2015.
[32] K. V. Mardia. Measures of multivariate skewness and kurtosis with applications. Biometrika, 57(3):519-534) 12 1970.
[33] M.Caron, B. Piotr, A. Joulin, and M. Douze. Deep clustering for unsupervised learning of visual features. In European Conference on Computer Vision, 2018.
[34] Y. Miao, I.. Yu, and P. Blunsom. Neural variational inference for text processing. In Proceedings of The 33rd International Conference on Machine Learning. Proceedings of Machine Learning Research, pages 1727-1736, 2016
[35] S. Mukherjee, H. Asnani, F. Lin, and S. Kannan. Clustergan: Latent space clustering in gener- ative adversarial networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4610-4617, 2019.
[36] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning. NIPS, 2011.
[37] A. Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning. In NIPS, 2017.
[38] A. Peacock. Twooddimensional goodness-of-ff fit testing in astronomy. Monthly Notices of the Royal Astronomical Society, 202(3):615-66) 627, 1983.
12

---

[39] V. Prasad, D. Das, and B. Bhowmick. Variational clustering: Leveraging variational autoen- coders for image clustering. CoRR, 0bs/2005.046133 2020.
[40] Y. Pu, Z. Gan, R. Henao, X. Yuan, Li, A. Stevens, and L. Carin. Variational autoencoder for deep learning of images, labels and captions. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett, editors. Advances Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
[41] A. Razavi, A. Oord, and O. Vinyals. Generating diverse high-fidelity images with vq-vae-2. ArXiv, 001119960004444 2019.
[42] F. Tian, B. Gao, Q. Cui, E. Chen, and T.-Y. Liu. Learning deep representations for graph clustering. In Twenty-Eighth AAAI Conference on Artificial Intelligence, 2014.
[43] Tolstikhin, O. Bousquet, S. Gelly, and Schoelkopf. Wasserstein auto-encoders. arXiv preprint arXiv: 171.0557 2017.
[44] J. M. Tomezak and M. Welling. Vae with vampprior. In A/STATS, 2018.
[45] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mmist: novel image dataset for benchmarking machine learning algorithms. ArXiv, abs/1708.00747, 2017.
[46] J. Xie, G.Girshick, and A. Farhadi. Unsupervised deep embedding for clustering analysis. In International conference on machine learning, pages 478-487, 2016.
[47] B. Yang, X. Fu, N. Sidiropoulos, and M. Hong. Towards k-means- meann--frendly spaces: Simulta- neous deep learning and clustering. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 3861-3870. JMLR. org, 2017.
[48] M. Zhang, Jiang, Z. Cui, R. Garnett, and Chen. D-vae: variational autoencoder for directed acyclic graphs. In NeurlPS, 2019.
[49] B. Zong, Q. Song, M. Renqiang M., W. Cheng, C. Lumezanu, D. k. Cho, and H. Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In ICLR, 2018.
13

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A]. You are strongly encouraged include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please refer section 5
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Since the proposed method focus on a fundamental learning problem rather than a specific application believe that there are direct societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Please refer to section 3
(b) Did you include complete proofs of all theoretical results? [Yes] Please refer section
3
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] Please refer to the supplementary material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're
using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "w8W9XgTbgwf", "reviewer": "Reviewer_R2eb", "summary": "The paper proposes a deterministic autoencoder that imposes a GMM on the latent space to learn a complex multimodal latent space. The model uses two regularizers including the KS distance and a constant covariance term to minimize the discrepancy between the marginal CDF of the empirical posterior and the given GMM prior. The authors provide multiple experimental results to show the reconstruction and sampling quality, deep clustering, and the latent model capability to generate discrete and complex structures such as chemical molecules.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \nThe originality of the paper seems very limited to me. Using autoencoders to fit a mixture model to the latent process to obtain the continuous and discrete representation factor has been done previously. Here, the author attempted to extend Ghosh et al., 2020 AE framework by imposing a GMM prior via minimizing the KS distance. The authors used the marginal CDF to approximate the KS distance, which is also a sub-optimal solution, without any guarantee. \n\nQuality: \nThe paper does not have enough theoretical aspects. There are abundant experimental studies, but I think it would beneficial to investigate the impact of hyperparameters ($\\lambda$) of the objective function and cross-correlation parameters in the prior distribution.\n\nClarity: \nThe paper is well organized. Although, I think the authors can provide a better description for figures and table captions.\n\nSignificance: \nThe results show the outperformance of the proposed framework for most of the datasets. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- The authors mentioned that some of the comparable methods suffer from computational complexity and training stability. However, the proposed model is using a sub-optimal optimization to estimate a multi-modal distribution, that may suffer from stability and convergence issues. Also, the proposed objective function requires three hyperparameters. The proposed weight estimation for the loss function seems very expensive and requires pre-trained autoencoder results.    \n\n- There is no sensitivity analysis to show the impact of the hyperparameters on the model’s performance.\n\n- As mentioned by the authors, the proposed method greatly relies on the prior GMM, which does not exist in many domains of application. In particular, it is not clear what should be presumed for a covariance matrix.\n\n- In Figure 1, it is not clear how each subfigure supports the outperformance of the proposed model. \n\n- In Table 2, it is not clear what the differences between 1st, 2nd and 3rd are. \n\n- It seems one equation (defining $\\phi(\\mathbf{z}_j)$) is missing in section 3.1, as mentioned in line 156.\n\n- In Eq.9, is the term in the parentheses only a function of one $m$ or all $m=1:M$?\n\n- In sections 4.2 and A.4, besides reporting the clustering performance, can you show the role of each dimension (as a continuous factor)? Since the covariance matrix is assumed as an identity matrix, are the latent dimensions properly independent from each other?\n\n- I think it would be beneficial if authors compared the performance of the proposed model with a GMM prior with other mixture VAE approaches that learn mixture distributions in a disentangled way, i.e. JointVAE (Dupont, 2018), CascadeVAE (Jeong et al., 2019), especially for the clustering task.\n\n\t\nMinor comments:\n- In the related work section, there are works which are not cited in the reference section of the main text, e.g. Jakub et al.\n- It seems that sometimes the bold and non-bold symbols are used interchangeably, e.g. ${F_j}^{(N)} (\\mathbf{z}_{ij})$ (line 155), and ${F_j}^{(N)} (z)$ (Eq.1), or $\\mu_k$ in the second part of eq.5\n- In Figure 2, there is no FID scores. \n", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "mijbV6etV-D", "reviewer": "Reviewer_9iT6", "summary": "The author(s) motivate and derive a novel deterministic auto-encoder loss that naturally encourages a multi-model latent representation that follows a GMM. The author(s) motivate and derive this loss using the KS test metric and generalize it to multiple dimensions and multiple modes. Their experiments, whereby they ancestrally resample latent codes show an impressive ability to generate realistic samples.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality:\n- To the best of my knowledge, the proposed regularization terms (equations 2, 3, 6, 7) are novel.\n\n### Quality:\n- The new regularization terms are well motivated using the KS test.\n\n### Clarity:\n- Paper is generally well written.\n- Ancestral resampling from prior does not necessarily generate new samples from the training distribution (lines 25-26) if the latent space utilization is multi-modal.\n- VAE paragraph references are inconsistent. Sometimes [x] is used other times NAME, et al. is used. Please be consistent. Anytime a name is used there is also no link to the appendix making reference checking cumbersome.\n- Equation 9 sets $\\lambda_{KS}^{-1}$ twice. I imagine the second one is supposed to be $\\lambda_{CV}^{-1}$\n- My major concern is how sampling FID (Samp.) is performed across the methods. Line 219 has \"models/by\" does the '/' denote 'or' or is suppose to be space? If all methods fit Gaussian to latent codes and sample from that, then this seems fair. But if some methods use GMM priors for sampling whereas others use uni-modal distributions, then this comparison seems unfair.\n- For the VAE methods, do the author(s) sample the decoder distribution or report its mean/mode?\n\n### Significance: \n- Table 1 results are strong, but lack error bars, which is moderately concerning. In particular, I wonder if some of the GMM wins are statistically insignificant.\n\n### Other:\nI love that \"Since the cluster is part of a carbon-neutral framework, these experiments did not contribute to climate change.\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author(s) sufficiently address limitations. Although, I do wonder what would happen under class imbalance. For instance say the data naturally falls into two clusters, a majority and minority. Do the author(s) have reason to believe their methods would perform well in this scenario? I think this question could be important to understand potential negative societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "QC01a9jqJ1", "reviewer": "Reviewer_vNcb", "summary": "This paper proposes to learn GMM priors in autoencoders, to be able to construct a generative model. They compare the generation quality of the proposed approach with VAEs using the FID metric. One interesting contribution seems to be the fact that they are using a loss function other than the GMM likelihood to fit the GMM. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed approach is definitely not entirely novel. I might be missing some aspects of it that is novel, but in general using GMMs or its variants has been used in the literature. \n\nThe results with the proposed approach seems to be better than few alternatives in terms of FID scores, but I would like to note that the datasets on which the authors try their approach is are relatively old. I think it would be worth exploring more challenging datasets which larger images such as celeba-hq, or larger images such as Imagenet like it is used in this paper https://arxiv.org/pdf/1906.00446v1.pdf \nwould be required to further convince the audience of the merit of the approach. \n\nThe paper also contains number of typos, and these should be corrected in an eventual acceptance in a conference. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have a small paragraph to discuss the limitations of their work. I am not entirely convinced that it is a comprehensive discussion on the limitations. \n\nThey do not discuss the societal impact, but I do not think there is an outstanding aspect to this work that makes it relevant for such discussion.   ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "OsFYvqpEnBz", "reviewer": "Reviewer_KeyS", "summary": "The paper builds on the ideas of Ghosh et al 2020 [10] pointing out the equivalence between Gaussian prior VAEs and deterministic AEs with noise inputation and further equivalence with regularized deterministic AEs. Since the original paper [10] needed to estimate the latent distribution ex post with a GMM, the authors here propose to use GMM prior directly and translate this into a deterministic AE with more complex regularization derived from univariate Kolmogorov-Smirnof distance combined with another term for covariance matching.They further propose a heuristic to find a reasonable heyper-parameter setting for the two regularizers. Finally, they show on a battery of experiments (generations over standard image datasets, unsupervised clustering, generating discrete structures) the favourable properties of their method.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a logical development from Ghosh et al 2020 [10]. The move towards deterministic AEs (from stochastic VAEs) for data generation is a relatively recent direction which this paper complements. The paper is well written and easy to follow, it develops the theory and argumentation well, posing the current paper well into the existing state of the art. The idea of GMM prior is not new in the VAE setting, however, its reformulation into the regularization terms in the context of deterministic AEs is (as far as I know) novel. The motivation and the expected benefits follow in a natural way from the well explored VAE case. The empirical evaluation is thorough and on a varied set of problems. The main limitation (the need to fix the prior GMM) is important and mentioned in section 5 towards the end of the paper. It may be worthwhile mentioning it sooner and I would like to understand better how this shall be fixed in practice (how was it fixed in the experiments). Overall, I find the paper interesting, proposing an interesting and useful extension on previous research which, in my vies, can be further picked up and develop for subsequent research.\n\n## Comments / questions\n- what's the difference (if any) between your KS distance and the continuous ranked probability score (Gneiting 2004)?\n- in equation (2) for the $L_{KS}$ you say you do MSE. Do you mean just squared error (SE)? I'm not sure what the mean (M) should be calculated over because your estimate of the CDF already uses all your data samples\n- your $L_{CV}$ compares the sample covariance with the prior covariance - does this mean you need to estimate the sample covariance from every batch? -> effects on training stability + compute? (I didn't find this in your limitations discussion)\n- is simple Frobenious norm the most appropriate for comparing the covariance matrices? (Something derived from the Wishart distribution?)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are discussed in section 5 (see above for the covariance estimation).\n\nSocietal impact - N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: By the scores alone, this paper is just borderline. Reviewers had a number of questions and clarifications, which were responded to well during the author response period, and even included additional results against a VQ-VAE baseline; this led to multiple reviewers raising their scores. Leaving aside issues of clarity (which could be fixed by camera-ready, given the discussion here), the main concern across the reviewers is the degree of novelty and originality, given that on the one hand there are a number of other methods which use Gaussian mixture priors on VAEs, and on the other hand this is also a direct extension of Ghosh et al 2020. The other concern is the somewhat heuristic use of an approximation to the KS statistic in the loss, and a question of how sub-optimal the result is, as well as whether using simply a Gaussian mixture likelihood could not lead to similar results.
Decision: Accept (Poster)