Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to learn GMM priors in autoencoders, to be able to construct a generative model. They compare the generation quality of the proposed approach with VAEs using the FID metric. One interesting contribution seems to be the fact that they are using a loss function other than the GMM likelihood to fit the GMM. 
main_review: The proposed approach is definitely not entirely novel. I might be missing some aspects of it that is novel, but in general using GMMs or its variants has been used in the literature. 

The results with the proposed approach seems to be better than few alternatives in terms of FID scores, but I would like to note that the datasets on which the authors try their approach is are relatively old. I think it would be worth exploring more challenging datasets which larger images such as celeba-hq, or larger images such as Imagenet like it is used in this paper https://arxiv.org/pdf/1906.00446v1.pdf 
would be required to further convince the audience of the merit of the approach. 

The paper also contains number of typos, and these should be corrected in an eventual acceptance in a conference. 
limitations_and_societal_impact: The authors have a small paragraph to discuss the limitations of their work. I am not entirely convinced that it is a comprehensive discussion on the limitations. 

They do not discuss the societal impact, but I do not think there is an outstanding aspect to this work that makes it relevant for such discussion.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The author(s) motivate and derive a novel deterministic auto-encoder loss that naturally encourages a multi-model latent representation that follows a GMM. The author(s) motivate and derive this loss using the KS test metric and generalize it to multiple dimensions and multiple modes. Their experiments, whereby they ancestrally resample latent codes show an impressive ability to generate realistic samples.
main_review: ### Originality:
- To the best of my knowledge, the proposed regularization terms (equations 2, 3, 6, 7) are novel.

### Quality:
- The new regularization terms are well motivated using the KS test.

### Clarity:
- Paper is generally well written.
- Ancestral resampling from prior does not necessarily generate new samples from the training distribution (lines 25-26) if the latent space utilization is multi-modal.
- VAE paragraph references are inconsistent. Sometimes [x] is used other times NAME, et al. is used. Please be consistent. Anytime a name is used there is also no link to the appendix making reference checking cumbersome.
- Equation 9 sets $\lambda_{KS}^{-1}$ twice. I imagine the second one is supposed to be $\lambda_{CV}^{-1}$
- My major concern is how sampling FID (Samp.) is performed across the methods. Line 219 has "models/by" does the '/' denote 'or' or is suppose to be space? If all methods fit Gaussian to latent codes and sample from that, then this seems fair. But if some methods use GMM priors for sampling whereas others use uni-modal distributions, then this comparison seems unfair.
- For the VAE methods, do the author(s) sample the decoder distribution or report its mean/mode?

### Significance: 
- Table 1 results are strong, but lack error bars, which is moderately concerning. In particular, I wonder if some of the GMM wins are statistically insignificant.

### Other:
I love that "Since the cluster is part of a carbon-neutral framework, these experiments did not contribute to climate change."
limitations_and_societal_impact: The author(s) sufficiently address limitations. Although, I do wonder what would happen under class imbalance. For instance say the data naturally falls into two clusters, a majority and minority. Do the author(s) have reason to believe their methods would perform well in this scenario? I think this question could be important to understand potential negative societal impacts.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a deterministic autoencoder that imposes a GMM on the latent space to learn a complex multimodal latent space. The model uses two regularizers including the KS distance and a constant covariance term to minimize the discrepancy between the marginal CDF of the empirical posterior and the given GMM prior. The authors provide multiple experimental results to show the reconstruction and sampling quality, deep clustering, and the latent model capability to generate discrete and complex structures such as chemical molecules.
main_review: Originality: 
The originality of the paper seems very limited to me. Using autoencoders to fit a mixture model to the latent process to obtain the continuous and discrete representation factor has been done previously. Here, the author attempted to extend Ghosh et al., 2020 AE framework by imposing a GMM prior via minimizing the KS distance. The authors used the marginal CDF to approximate the KS distance, which is also a sub-optimal solution, without any guarantee. 

Quality: 
The paper does not have enough theoretical aspects. There are abundant experimental studies, but I think it would beneficial to investigate the impact of hyperparameters ($\lambda$) of the objective function and cross-correlation parameters in the prior distribution.

Clarity: 
The paper is well organized. Although, I think the authors can provide a better description for figures and table captions.

Significance: 
The results show the outperformance of the proposed framework for most of the datasets. 

limitations_and_societal_impact: - The authors mentioned that some of the comparable methods suffer from computational complexity and training stability. However, the proposed model is using a sub-optimal optimization to estimate a multi-modal distribution, that may suffer from stability and convergence issues. Also, the proposed objective function requires three hyperparameters. The proposed weight estimation for the loss function seems very expensive and requires pre-trained autoencoder results.    

- There is no sensitivity analysis to show the impact of the hyperparameters on the model’s performance.

- As mentioned by the authors, the proposed method greatly relies on the prior GMM, which does not exist in many domains of application. In particular, it is not clear what should be presumed for a covariance matrix.

- In Figure 1, it is not clear how each subfigure supports the outperformance of the proposed model. 

- In Table 2, it is not clear what the differences between 1st, 2nd and 3rd are. 

- It seems one equation (defining $\phi(\mathbf{z}_j)$) is missing in section 3.1, as mentioned in line 156.

- In Eq.9, is the term in the parentheses only a function of one $m$ or all $m=1:M$?

- In sections 4.2 and A.4, besides reporting the clustering performance, can you show the role of each dimension (as a continuous factor)? Since the covariance matrix is assumed as an identity matrix, are the latent dimensions properly independent from each other?

- I think it would be beneficial if authors compared the performance of the proposed model with a GMM prior with other mixture VAE approaches that learn mixture distributions in a disentangled way, i.e. JointVAE (Dupont, 2018), CascadeVAE (Jeong et al., 2019), especially for the clustering task.

	
Minor comments:
- In the related work section, there are works which are not cited in the reference section of the main text, e.g. Jakub et al.
- It seems that sometimes the bold and non-bold symbols are used interchangeably, e.g. ${F_j}^{(N)} (\mathbf{z}_{ij})$ (line 155), and ${F_j}^{(N)} (z)$ (Eq.1), or $\mu_k$ in the second part of eq.5
- In Figure 2, there is no FID scores.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper builds on the ideas of Ghosh et al 2020 [10] pointing out the equivalence between Gaussian prior VAEs and deterministic AEs with noise inputation and further equivalence with regularized deterministic AEs. Since the original paper [10] needed to estimate the latent distribution ex post with a GMM, the authors here propose to use GMM prior directly and translate this into a deterministic AE with more complex regularization derived from univariate Kolmogorov-Smirnof distance combined with another term for covariance matching.They further propose a heuristic to find a reasonable heyper-parameter setting for the two regularizers. Finally, they show on a battery of experiments (generations over standard image datasets, unsupervised clustering, generating discrete structures) the favourable properties of their method.
main_review: The paper proposes a logical development from Ghosh et al 2020 [10]. The move towards deterministic AEs (from stochastic VAEs) for data generation is a relatively recent direction which this paper complements. The paper is well written and easy to follow, it develops the theory and argumentation well, posing the current paper well into the existing state of the art. The idea of GMM prior is not new in the VAE setting, however, its reformulation into the regularization terms in the context of deterministic AEs is (as far as I know) novel. The motivation and the expected benefits follow in a natural way from the well explored VAE case. The empirical evaluation is thorough and on a varied set of problems. The main limitation (the need to fix the prior GMM) is important and mentioned in section 5 towards the end of the paper. It may be worthwhile mentioning it sooner and I would like to understand better how this shall be fixed in practice (how was it fixed in the experiments). Overall, I find the paper interesting, proposing an interesting and useful extension on previous research which, in my vies, can be further picked up and develop for subsequent research.

## Comments / questions
- what's the difference (if any) between your KS distance and the continuous ranked probability score (Gneiting 2004)?
- in equation (2) for the $L_{KS}$ you say you do MSE. Do you mean just squared error (SE)? I'm not sure what the mean (M) should be calculated over because your estimate of the CDF already uses all your data samples
- your $L_{CV}$ compares the sample covariance with the prior covariance - does this mean you need to estimate the sample covariance from every batch? -> effects on training stability + compute? (I didn't find this in your limitations discussion)
- is simple Frobenious norm the most appropriate for comparing the covariance matrices? (Something derived from the Wishart distribution?)
limitations_and_societal_impact: Limitations are discussed in section 5 (see above for the covariance estimation).

Societal impact - N/A
review_title: null

Meta-review: By the scores alone, this paper is just borderline. Reviewers had a number of questions and clarifications, which were responded to well during the author response period, and even included additional results against a VQ-VAE baseline; this led to multiple reviewers raising their scores. Leaving aside issues of clarity (which could be fixed by camera-ready, given the discussion here), the main concern across the reviewers is the degree of novelty and originality, given that on the one hand there are a number of other methods which use Gaussian mixture priors on VAEs, and on the other hand this is also a direct extension of Ghosh et al 2020. The other concern is the somewhat heuristic use of an approximation to the KS statistic in the loss, and a question of how sub-optimal the result is, as well as whether using simply a Gaussian mixture likelihood could not lead to similar results.
Decision: Accept (Poster)