Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes to use evolutionary algorithm to learn truncated deep latent variable model. The method get good performance in denoising  task. 

Pros:
Quality: The method seems correct 
On denoising task, the performance of the proposed model is good. 
Significance: Inference of Discrete VAE is an important question to address

Cons & questions:
Clarity: The paper is very hard to read due to many reasons: 
a) It does not use commonly used notations, such as the paper use F to present ELBO. The paper seems to use theta to present the decoded mean and variance where people commonly use theta for the decoder weights etc. Under such unique notations which is not wrong, but the author did not explain them clearly either. Around equation (2), it just says that these are parameter to optimize (which made me think that these are the weights) and later on I found that weights are denoted by W.  and phi is the Bernoulli parameter for z (which was denoted as pi before) and theta is the decoded mean and variances. So it just makes reading very confusing. 
b) It is not self contained. For example equation (7) is just pointed to another paper which I checked another paper for mins and did not find equation (7)  and did not try more as the cited paper is very very long. So it would be good to clarify critical equations. 
I also wonder about Eq.(7)'s correctness as the right hand side looks like the model evidence instead of the lower bound of it. 
Other places, such as Eq.(3) it is formulated in this way in the Truncated VAE paper but not cited. 
Generically, the reading of the paper is not easy and there are many simple things to do just to make it more clear. 

Significance:
a) The author seems try to claim that people in the field never used non-amortized way to train a deep latent Gaussian model. This is very wrong, also the author did not discuss related work in this direction at all. e.g. using MCMC
Learning Deep Latent Gaussian Models with Markov Chain Monte Carlo, http://proceedings.mlr.press/v70/hoffman17a/hoffman17a.pdf 

b)  VAE's contribution are two-fold, one is latent variable model and the second part if the amortized inference. The paper does not seems differ such existing separation. The paper in the end contributed a non-amortized inference method, but the discussion of the paper is very entangled.  

c) As the author pointed out, the proposed method is not scalable  as it is per data point and thus it will have limited application impact. 

d) Experiments did not compare any highly relevant inference baselines. It does not compare any other inference methods in deep latent variable model. 

e) there is only one set of experiments in total and on an traditional denositing task which is very limited. More experimental settings are needed to show the usefulness of the method. 

review_title: Ok theory; Clarity can be improved; application impact need to considered 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes an evolutionary optimization framework for training vartional autoencoders (VAEs) with discrete latents. In contrast to the standard VAE paradigm, the proposed TVAE approach does not require an encoder for amortized inference given the input. The method instead relies on a pool of latent variable samples for each data point to activate the decoder network. The latent variable pools are maintained and iteratively updated to increase the average lower-bound of the marginal log-likelihood of the input data. Experimental results show that non-linear decoders optimized by the TVAE framework outperform their linear counterparts on a denoising task.  Further results demonstrate method's competitiveness on zero-shot denoising, where a TVAE decoder is only trained on the noisy input image to reconstruct a smoothed version of the input image.
 
The paper is well-written and easy to follow. The work proposes an interesting alternative for optimizing VAEs which does not require an encoder network for amortized inference. I however have a number of concerns that are as follows:

The method instead imposes the overhead of maintaining and evolving a collection of latent variables for every data point, which can be both sample inefficient and memory-heavy for sizable problems. Then when or why would one trade amortized inference for the proposed approach?

The paper falls short of comparing the proposed optimization procedure with other alternatives for training standard or discrete VAEs. 

From reading the paper it is not clear how the size of the pool may need to be varied as the nature of the task or the number of latent changes.

The authors use a greedy approach to update the pool of latent samples. Does it not cause the optimization procedure to get stuck in local modes? Could one instead use MCMC type sampling approaches to enable mode jumps? 

Why only denoising experiments? Can the authors use their approach for data synthesis tasks where the latents can be shown to control different attributes of the data generative process?
review_title: A different take on training VAEs

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a new approach to train VAEs with binary latents, using an evolutionary algorithm to optimise a discrete set of variational parameters rather than the usual amortised variational model trained with gradient-based methods. The authors consider the setting of training a VAE with discrete, Bernoulli distributed latents and a continuous, Gaussian distributed output. They use a novel approach to form and train a posterior on the discrete latent space, parameterising the posterior somewhat implicitly via the sets defining a truncated posterior - in which the approximate posterior is proportional to the true posterior, but only within a subset of all points. Defining the subset of points amounts to parameterising the approximate posterior. Optimisation of these parameters amounts to a discrete search problem, which the authors tackle using an evolutionary algorithm.

Overall, I find the paper very well written and straightforward to follow. Indeed, the style of writing tends to pique the interest of the reader without being colloquial. I think the motivation of the paper is well established, given the number of papers that have investigated the training of modern generative models with discrete latents. Importantly, I think the proposed approach is novel and seemingly effective. The use of the truncated posterior along with a relatively simple evolutionary optimisation procedure is quite different from the previously seen methods that generally attempt to maintain a differential proxy to the ELBO.

The proposed method does have limitations, which the authors acknowledge. Namely that the method will not scale to large datasets, since the posterior is not amortised. I think it must also be true that the evolutionary optimisation is not particularly suited to high-dimensional spaces, since it is generally accepted that search suffers in such settings. However, these limitations I do not think are overwhelming. The authors find a niche problem setting in which their method is practical - that of “zero-shot” data denoising. In this setting they demonstrate that their method (with a small VAE) outperforms the state-of-the-art methods. The authors also provide a small set of straightforward and convincing toy experiments to show that their approach satisfies basic properties that we require in a learning system, such as appropriate scaling behaviour and recovery of artificial data generating parameters. Although I think the empirical demonstration may be enough, given the method is relatively novel, it is still limited. The main experimental result of the paper is denoising on a single image. Even performing the denoising on a small set of images would help the paper, since there will undoubtedly be variance in the performance across different images.

One other thing I would say about this paper is that identifying the approach as a VAE almost feels inappropriate. It is already true that the usual VAE is not an autoencoder at all, and so the name is a slight misnomer. But with the authors’ approach here, the posterior is not even parameterised as a function of the data (although there is of course an implicit dependence established through training). Hence identifying the method as related to an autoencoder seems to be misleading (although understandable given that the method is clearly related to the VAE).
review_title: Novel and interesting idea with limited empirical evaluation

final_score: rating: 6: Marginally above acceptance threshold
review_content: **summary** 
the paper proposes a novel approach to training variational autoencoder models, based on non-parametric form of truncated approximate posterior. Posterior is truncated to have support on a small subset of latent space allowing for exact marginalization. The support of approximate posterior in latent space for each data point $x$ is learned via evolutionary algorithm, minimizing ELBO. Method is applied to denoising tasks for images.

**pros**
the idea of using evolutionary algorithm for approximating truncated posterior is new to me, the fact that it works in practice is very interesting. 

**cons**
* The idea is limited to small architectures due to the need to evaluate the decoder for every state in the truncated set as opposed to single evaluation in conventinal approaches. The authors for this reason choose to apply their approach to denoising applications, which I am not familiar with and can't evaluate the advantage of their method.
* The fact that approximate posterior is not amortized over the dataset is also a potential issue: for each data point $q(z|x)$ will need to be updated to match evolution of the decoder of the entire epoch. It seems plausible that applying more evolutionary steps can take care of this but it would be useful to see experiments with varying number of  evolutionary steps.
* authors didn't compare their approach to the recent paper "Efficient Marginalization of Discrete and Structured Latent Variables via Sparsity" https://arxiv.org/abs/2007.01919, it would be good to understand which one works better.


comments:
* The approach is reminiscent of EM approach where samples from the posterior are approximated via Metropolis-Hastings updates. This can be made more efficient by adding temperature annealing. I would be interested to see how the proposed approach compares to this.
* it would be clearer to make x-dependence in Eq (3) explicit rather then just through superscript (n)
* it would be interesting to see how large is gap between the proposed ELBO and true LL in the trained models, to get a sense of good the approximation is.



review_title: Interesting idea of using evolutionary search for training VAEs, analysis can be improved by comparing with existing approaches

Meta-review: The paper presents an evolutionary optimization framework for training discrete VAEs, which is different to the standard way of training VAEs. One of the main criticism of the paper was the choice of experiments, but the authors addressed this point by adding an inpainting benchmark.

Unfortunately, the reviewers' scores are borderline, and one of the reviewers pointed out the lack of scalability (more precisely, linear scalability with the number of observations) and cannot recommend acceptance based on the limited application impact. Given the large number of ICLR submissions, this paper unfortunately does not meet the acceptance bar. That said, I encourage the authors to address this point and resubmit the paper to another (or the same) venue.
Decision: Reject