Direct Evolutionary Optimization of Variational Autoencoders with Binary Latents
Authors: Enrico Guiraud
Jakob Drefs
Jorg Lucke
Venue:  (2021)
Keywords: variational optimization
variational autoencoders
denoising
evolutionary algorithms

Abstract: Discrete latent variables are considered important to model the generation process of real world data, which has motivated research on Variational Autoencoders (VAEs) with discrete latents. However, standard VAE training is not possible in this case, which has motivated different strategies to manipulate discrete distributions in order to train discrete VAEs similarly to conventional ones.
Here we ask if it is also possible to keep the discrete nature of the latents fully intact by applying a direct discrete optimization for the encoding model. The studied approach is consequently strongly diverting from standard VAE training by altogether sidestepping absolute standard VAE mechanisms such as sampling approximation, reparameterization trick and amortization.                                                                              

Discrete optimization is realized in a variational setting using truncated posteriors in conjunction with evolutionary algorithms (using a recently suggested approach). For VAEs with binary latents, we first show how such a discrete variational method (A)~ties into gradient ascent for network weights and (B)~uses the decoder network to select latent states for training.                       
                                                                                                    
More conventional amortized training is, as may be expected, more efficient than direct discrete optimization, and applicable to large neural networks.
However, we here find direct optimization to be efficiently scalable to hundreds of latent variables using smaller networks.
More importantly, we find the effectiveness of direct optimization to be highly competitive in 'zero-shot' learning (where high effectiveness for small networks is required).
In contrast to large supervised neural networks, the here investigated VAEs can denoise a single image without previous training on clean data and/or training on large image datasets.                                                

More generally, the studied approach shows that training of VAEs is indeed possible without sampling-based approximation and reparameterization, which may be interesting for the analysis of VAE training in general. In the regime of few data, direct optimization, furthermore, makes VAEs competitive for denoising where they have previously been outperformed by non-generative approaches.

Under review as a conference paper at ICLR 2021
DIRECT EVOLUTIONARY OPTIMIZATION OF VARIATIONAL AUTOENCODERS WITH BINARY LATENTS
Anonymous authors
Paper under double-blind review
ABSTRACT
Discrete latent variables are considered important to model the generation process of real world data, which has motivated research on Variational Autoencoders (VAEs) with discrete latents. However, standard VAE training is not possible in this case, which has motivated different strategies to manipulate discrete dis- tributions in order to train discrete VAEs similarly to conventional ones. Here we ask if it is also possible to keep the discrete nature of the latents fully intact by applying direct discrete optimization for the encoding model. The studied approach is consequently strongly diverting from standard VAE training by al- together sidestepping absolute standard VAE mechanisms such sampling ap- proximation, reparameterization trick and amortization. Discrete optimization is realized in variational setting using truncated posteriors in conjunction with evo- lutionary algorithms (using a recently suggested approach). For VAEs with binary latents, we first show how such discrete variational method (A) ties into gradient ascent for network weights and (B) uses the decoder network to select latent states for training. More conventional amortized training is, as may be expected, more efficient than direct discrete optimization, and applicable to large neural networks. However, we here find direct optimization to be efficiently scalable to hundreds of latent variables using smaller networks. More importantly, we find the effective- ness of direct optimization to be highly competitive in 'zero-shot' learning (where high effectiveness for small networks is required). In contrast to large supervised neural networks, the here investigated VAEs can, e.g., denoise a single image without previous training More generally, the studied approach shows that training of VAEs is indeed possi- ble without sampling- based approximation and reparameterization, which may be interesting for the analysis of VAE-training in general. In the regime of few data, direct optimization, furthermore, makes VAEs competitive for denoising where they have previously been outperformed by non-generative approaches.
clean data and/or training large image datasets.
INTRODUCTION AND RELATED WORK
Variational autoencoders (Kingma & Welling, 2014; Rezende et al., 2014) are prominent and very actively researched models for unsupervised learning. VAEs, in their many different variations, have successfully been applied to large number of tasks including semi-supervised learning (e.g. Maaløe al., 2016), anomaly detection (e.g. An & Cho, 2015; Kiran al.. 2018), sentence in- terpolation (Bowman et al., 2016), music interpolation (Roberts et al., 2018) and drug response prediction (Rampasek et al., 2017). The success of VAEs rests on series of methods that enable the derivation of scalable training algorithms to optimize their model parameters (discussed further below). A desired feature when applying VAEs a given problem is that their latent variables (i.e., the encoder output variables) correspond to meaningful properties of the data, ideally to those latent causes that have originally generated the data. However, many real-world datasets suggest the use of discrete latents as they often describe the data generation process more naturally. For instance, the presence or absence of objects in images is best described by binary latents (e.g. Jojic & Frey, 2001). Discrete latents also a popular choice in modeling sounds; for instance, describing piano sounds may naturally involve binary latents: keys are pressed or not (c.g. Titsias & 2011; Goodfellow al., 2013; Sheikh al., 2014). The success of standard forms of VAEs has

---

Under review as a conference paper at ICLR 2021
consequently spurred research on novel formulations that feature discrete latents (e.g. Rolfe, 2016; Khoshaman & Amin, 2018; Roy et al., 2018; Sadeghi et al., 2019; Vahdat et al., 2019).
The objective of VAE training is the optimization of generative data model which parameterizes a given data distribution. Typically we seek model parameters of a VAE that maximize the data log-likelihood, L()) = En, log (pe(z(())
where we denote by z(I:N) a set of N observed data points, and where pe (T) denotes the modeled data distribution. Like conventional autoencoders (e.g., Bengio et al., 2007), VAEs use a deep neural network (DNN) to generate (or decode) observ- ables E from latent code Unlike conventional autoencoders, however, the generation of data is not deterministic but it takes the form of a probabilistic generative model.
For VAEs with binary latent variables, as they will be of interest here, we consider the following VAE generative model:
(=) Bern 2, R) =
(T (1 II
pa -V
(I)
where E (0, is binary code and the non-linear function Tū(;; W) is a DNN that outputs the mean of the Gaussian distribution. p= ž) is commonly referred to as decoder. The set of model parameters is = (म,, W, J"). where W incorporates DNN weights and biases. We assume homoscedasticity of the Gaussian distribution, but note that there is no obstacle generalizing the model by inserting a DNN non-linearity that outputs a correlation matrix. Similarly, the algorithm could easily be generalized to different noise distributions should the task at hand call for it. For the purpose of this work, however, we will focus on as elementary as possible VAEs, with the form shown in Eqn. (1).
Given standard or binary-latent VAEs, essentially all learning algorithms seek to approximately maximize the log-likelihood using the following series of methods (we elaborate in the appendix):
(A) Instead of the log-likelihodd, a variational lower-bound (a.k.a. ELBO) is optimized.
(B) VAF posteriors are approximated by encoding model, that is specific distribution (often Gaussian) parameterized by one or more DNNs.
(C) The variational parameters of the encoder are optimized using gradient ascent on the lower bound, where the gradient is evaluated based on sampling and reparameterization trick to obtain sufficiently low-variance and yet efficiently computable estimates.
(D) Using samples from the encoder, the parameters of the decoder are optimized using gradient ascent on the variational lower bound.
Optimization procedures for VAEs with discrete latents follow the same steps (Points A to D). How- ever, discrete or binary latents pose substantial further obstacles learning. mainly due to the fact that backpropagation through discrete variables is generally not possible (Rolfe, 2016; Bengio et al., 2013). In order maintain the general VAE framework for encoder optimization, different groups have therefore suggested different possible solutions: work by Rolfe (2016), for instance, extends VAEs with discrete latents by auxiliary continuous latents such that gradients can still be computed. Work the concrete distribution (Maddison al., 2016) or Gumbel-softmat distribution (Jang et al., 2016) proposes newly defined continuous distributions that contain discrete distributions as limit cases. Work by Lorberbom al. (2019) merges the Gumbel-Max reparameterization with the use of direct loss minimization for gradient estimation, enabling efficient training on structured la- tent spaces. Finally, work by van den Oord al. (2017), and Roy et al. (2018) combines VAEs with a vector quantization (VQ) stage in the latent layer. Latents become discrete through quantization but gradients for learning are adapted from latent values before they are processed by the VQ stage. All methods have the goal of treating discrete distributions such that standard VAE training as de- veloped for continuous latents can still be applied. These techniques interact during training with the standard methods (Points A-D) already in place for VAE optimization. Furthermore, they add further types of design decisions and hyper-parametent for example parameters for annealing from softened discrete distributions to the (hard) original distributions for discrete latents.
For discrete VAEs, it may consequently be a desirable goal to investigate alternative, more direct optimization procedures that do not require softening of discrete distributions or the use of other indirect solutions. Such a direct approach is challenging, however, because once DNNs are used to define the encoding model (Point B) standard tricks to estimate gradients (Point C) seem unavoid- able. A direct optimization procedure, as is investigated here, consequently has to substantially change VAE training. For the data model (1) we will maintain the variational setting and decoding
2

---

Under review as a conference paper at ICLR 2021
model with DNNs non-linearity (Points A and D). However, we will not use encoder model parameterized by DNNs (Point B). Instead, the variational bound will be increased w.r.t. the encoder model by using discrete optimization approach. The procedure does not require gradients to be computed for the encoder such that discrete latents are addressed without the use of reparameteriza- tion trick and sampling approximations.
TRUNCATED VARIATIONAL OPTIMIZATION
Let us consider the variational lower bound of the likelihood. we denote by 8 the variational (m)
distributions with parameters T (x) and by (h(=)), ag" E= 14 (z) lx2) expectation values w.r.t.
to 4 (2), then the lower bound can be written as:
=( (log (pe(#( pe pe(2))) Σn (og (9g (m) (2)) (m)
(2)
The general challenge for the maximization of F(Ø, Q) is the optimization of the encoding model (n) VAEs with discrete latents add this challenge the problem of taking gradients w.r.t. discrete latents. If we seek to avoid derivatives w.r.t. discrete variables, we have to define an alternative en- coding model de (m) but such an encoding has to remain sufficiently efficient. Considering prior work on generative models with discrete latents, variational distributions based on truncated posteriors offer themselves
such alternative (Lücke & Sahani, 2008). Truncated posterior approxima- tions have been shown to be functionally competitive (e.g. Sheikh et al., 2014; Hughes & Sudderth, 2016; Shelton et al., 2017), and they are able to efficiently train also very large scale models with hundreds or thousands latents (e.g. Shelton et al., 2011: Sheikh & Lücke, 2016; Forster Lücke, 2018). However, the important question for training discrete VAEs is if or how truncated variational distributions can be used in gradient-basee for the first time, address this question noting that all previous approaches relied on closed-form (or pseudo-closed form) parameter update equations in an expectation-maxxin maximization learning paradigm.
optimization of neural network parameters. We here,
Optimization of the Decoding Model. In order optimize the parameters W of the decoder DNN ū(ž, W), the gradient of the variational bound (2) w.r.t. W has to be computed. We consequently need, for any VAE, a sufficiently precise and efficient approximation of the expectation value w.r.t.
the encoder d (n) (2). Gradient estimation is of central importance for deep unsupervised learning, and approaches, e.g., for variance reduction of estimators have played an important role and dedicated solely to this purpose (e.g., Williams, 1992). Reparameterization finally emerged as a key method because allowed for sufficiently low-variance estimation of gradients based, e.g., on Gaussian middle-laye units (Kingma & Welling, 2014; Rezende et al., 2014).
For discrete VAES, however, reparameterization requires the introduction of additional manipula- tions of discrete distributions that we here seek to fully avoid.
Instead of using reparameterization or variance reduction, we will compute gradients based on trun- cated posterior as variational distributions. A truncated posterior has the following form:
( (n))
(7 (m)
(e())
10 (n) (7) Exeegt
if EE
[(()
Egegm Pe E') Pe("")
where for all / + the probability (()( (E) equals zero. That is, variational distribution (m) proportional to the true posteriors in subset o (") which acts as its variational parameter.
We compute the gradient of (2) w.r.t. the decoder weights W which results in:
F( = 20 Σ Σ
(n) (2) 4
W (l 22
(4)
(")
The right-hand-side has salient similarities to the standard gradient ascent for VAE decoders. Es- pecially the familiar gradient of the mean squared error (MSE) shows that, e.g., standard automatic differentiation tools can be applied. However, the decisive difference are the weighting factors
de (Z). Considering (3), in order to compute the weighting factors require all H क (m)
(n)
to be
3

---

Under review as a conference paper at ICLR 2021
passed through the decoder DNN. As all states of + (*) coder for the MSE term of (4), the overall computational complexity is not higher than an estimation of the gradient with samples instead of states in +b (*) (we elaborate in Appendix A).
anyway have be passed through the de-
To complete the decoder optimization, update equations for variance o² and prior parameters 7f can be computed in closed-form (compare, e.g., Shelton et al., 2011) and given by:
n.nw
DN
Σ
Σ (Eq(
da
(m)
| (x)
iū W)
gnew =uew= N Σ Σ
2o
(3)
(5)
Zeg())
where N is the number of samples in the training dataset and D is the number of observables.
Optimization of the Encoding Model. After having established that the decoder efficiently and by using standard DNN methods, the important question is if the encoder can be trained efficiently. Encoder optimization is usually based on reformulation of the variational bound (2) given by: -( En (log (pe(7 [=)) A.> En. DRL. ( (e))
be optimized
(6)
Centrally for this work, truncated posteriors allow specific alternative reformulation of the bound that enables efficient optimization. The reformulation recombines the entropy term of the original form (2) with the first expectation value into a single term, and is given by (see Lücke, 2019, for details):
F(Ø, O) =I log Σ (e( (2) p))()))
Thanks to the simplified form of the bound, the variational parameters 4 (n) of the encoding model can now be sought using direct discrete optimization procedures. More concretely, because of the
specific form (7), pairwise comparisons of joint probabilities are sufficient to maximize the lower
bound: if we update the set + ()) for a given & (x) by replacing state gold + (n)
with state
gnow & 4 (n) then F(Ø, θ) increases if and only if:
log (pe(f (n) g"ex) > log (pe ( (p))))))
(8)
To obtain intuition for the pairwise comparison, consider its form when inserting the binary VAE (1) into the left- and right-hand sides. Eliding terms that not depend on ž we obtain:
logPe (z. 2) - || मスž w) - o² En गn 2b.
where 7n = log ((1 - /n//TA) ). The expression assumes an even more familiar form if we restrict ourselves for moment to sparse priors क < i.e., 0. Criterion (8) then becomes:
| (n) [T(znew W2 + 20 Inew
| (r) (l W)ll + 2 goll
(10)
where 12" = -m 2h- Such functions are routinely encountered in sparse coding or compressive sensing (Eldar & Kutyniok, 2012): for each set x (n) we seek those states / that are reconstructing z(0) well while being sparse (5 with few non-zero bits). For VAES, 77(znew W) is DNN and as such much more flexible in matching the distribution of observables / than can be expected from linear mappings. Furthermore, criteria like (10) usually emerge for maximum a-posteriori (MAP) training in sparse coding (Olshausen & Field, 1996). In contrast, we here seek a population of states
Z in & (n) for each data point. It is consequence of the reformulated lower bound (7) that it remains optimal to evaluate joint probabilities (as for MAP) although the constructed population of states 4 (a)
can capture (unlike MAP training) a rich posterior structure.
But how can new states gnew that optimize 4 (") be found efficiently in high-dimensional latent spaces? Random search and search by sampling has recently been explored for elementary gener- ative models (Lücke et al., 2018). Here we will follow another recent suggestion (Guiraud et al., 2018) and make use of search based on evolutionary algorithms (EAs). In this setting we interpret sets 4 (m) as populations of binary genomes 7 and base the fitness function on Eqn. (9).
Concretely, using " (n) as initial parent pool, we apply the following genetic operators sequence: firstly, parent selection picks N, states from the parent pool. In our numerical experiments we used
4

---

Under review as a conference paper at ICLR 2021
A. Parent selection
B. Mutation
C. New population
Figure 1: The optimization process of the variational parameters p(n) using evolution- ary search. A. Some states are selected as parents. B. Each child undergoes mutation. C. Children are merged with the original population and the least fit are discarded.
+
Erssess
6toee
0010
0010
0010
0010
0001
GOLD
001
1010
2.5
0001
0011
0001
proppotooolt parent selection, for which we add an offset (constant w.r.t. E) the fitness values in order to make them strictly non-negative. Each of the children undergoes mutation: or more bits are flipped to further increase offspring diversity, In our experiments we perform random uniform selection of the bits to flip. Crossover could also be employed to increase offspring diversity. We repeat the procedure using the children generated this way as the parent pool, giving birth to multiple generations of candidate states. Finally, update & by substituting individuals with low fitness with candidates with higher fitness. The whole procedure can be seen as an evolutionary algorithm with perfect memory very strong elitism (individuals with higher fitness never drop out of the gene pool). Note that the improvement of the variational lower bound depends on generating many possible different children with high fitness over the course of training.
We point out that the EAs optimize each (") independently, so this technique can be applied to large datasets in conjunction with stochastic or batch gradient descent the model parameters O: does not require to keep the full dataset (or all sets & (²)) in memory at a given time. Fig shows how EAs produce new states that are used to update each set Φ (m) The full training procedure for binary VAEs is summarized in Algorithm 1.
Algorithm Training Truncated Variational Autoencoders
Initialize model parameters θ { 7, o Initialize each + (x) with S distinct latent states repeat
2 Z~ Bern (2rm(2)77) (t) ReLU layer(s)
for all batches dataset do for sample in batch do Thee (x)
Linear output layer i W)
for all generations do
T neo mutation (crossover (selection ()))))))
+ (n) = U dhew
I N~N(3,
end for Truncate (*)
to S fittest elements based on (9)
Figure 2: Graphical representation of the model architecture used in numeri- cal experiments.
end for
Use Adam update W using objective (4)
end for
Use (5) update म, o² until parameters O have sufficienlly converged
3 NUMERICAL EXPERIMENTS
Having defined the training procedure, we numerically investigated its properties. After first verify- ing that the procedure can recover generating parameters using ground-truth data (see Appendix B), we conducted experiments to address the following two standard questions:
(1) How efficient, i.e. how scalable, is the direct discrete optimization of binary VAFs? (2) How effective is the procedure, i.c., how well does it perform for given VAE model?
In all numerical experiments, the training of the DNN parameters based on (4) is performed with mini-batches the Adam optimizer (Kingma & Ba, 2014) and decaying or cyclical learning rate scheduling (Smith, 2017). Xavier/Glorot initialization (Glorot & Bengio, 2010) is used for the DNN weights. while biases are always zero-initialized. Parameters म and x² updated via Equ. (5) and initialized to t (H is the size f)) and 0.01 respectively. flyper-paramete optimization was conducted manually and, for the more complex datasets, also made use of black box Bayesian op-
5

---

Under review as a conference paper at ICLR 2021
Table 1: Denoising perfor- mance PSNR (dB) for the 'house' image under con- trolled conditions (D=8x8, H=64 for all algorithms).
0=15
o=25
0=50
MTMKL
34.29
31.88
28,08
GSC
32.68
31.10
28.02
VAR-BSC
32.25
31.15
28.62
TVAE
34.27 + 022
32.65 = .06
29.61 + .02
timization based on Gaussian Processes (Nogueira, 2019). We will refer the binary VAE trained with the method described above as Truncated Variational Autoencoder (TVAE) as the use of trun- cated posteriors the main distinguishing feature.
Scalability and improvement on linear models. Let us first numerically investigate scalability properties of TVAE especially in comparison with linear models. After verifying parameter recovery for ground-truth data (see Appendix B), we used natural data in the form of image patches as an intermediately large scale and natural dataset. Concretely, we used 100,000 whitened image patches of 16 x 16 pixels extracted from standard image database (van Hateren & van der Schaaf, 1998) and pre-processed as in Guiraud et al. (2018).
The simplest possible VAEs would use linear mappings for the decoder µ((Z W.. For standard Gaussian latents, a linear VAE can recover probabilistic PCA solutions (e.g. Lucas et al., 2019). For Bemoulli latents, we would recover binary sparse coding (Haft et al., 2004; Shelton et al., 2011) solutions. We therefore start training (using H = 300 latents) with a linear VAE. After 100 epochs the weights of the linear mapping
used to initialize the bottom layer of deeper decoder network with three layers of 300, 300 and 16 x 16 = 256 units. The weights of the deeper layers the identity matrix. Furthermore, prior and variance were optimized.
were simply initialized The described setup guarantees a common starting point for linear and non-linear VAEs such that the difference provided by deeper decoder DNNs can be highlighted. Fig. 3 shows the variational bounds during learning of the linear VAE compared to the non-linear VAE for a typical experiment.
The non-linear VAE can be observed to quickly and significantly optimize the lower bound beyond linear VAE. We will later (when we are not interested in comparisons to linear VAEs) simply optimize the weights of non-linear TVAE directly as we did not observed an advantage by first optimizing a linear VAE.
Incer binary VAE TWAE starting from Inear mappirg
Original
Noisy (a 50)
Denoised
5000
-20
-40
-50
-00
25
50
160
125
150
175
208
spoch
Figure 4: TVAE denoising of house image with noise level o 50. The denoised image has PSNR=30.03, the best of the runs of Tab. 2.
Figure 3: ELBO gain of TVAE compared to linear VAE with binary latents (on 16 16 image patches).
Compared to shallow linear models, we observed a similar efficiency and scalability of TVAE. The main additional computational costs are given by passing the latent states through the full decoder DNN instead of just through a linear mapping. The sets of states used could be kept small, at size S c (m) compares to N x M states that would be used for standard VAE training (given M samples are
64, such that N x (1 (m), db (m) !) states had be evaluated for each epoch. This
drawn per data point). Differently standard VAE training the "
(m) have be remembered across
iterations. For very large datasets, the additional O(N x ( (n) ()H) memory demand can be dis- tributed over compute nodes, however. To further investigate scalability,
went up to H=1000 latent variables (while using 100 units in the DNN middle layer). TVAE training time remained in line with the theoretical linear scaling with H while the variational bound further increased.
Effectiveness: Image Denoising, As we have observed, scaling to large latent spaces does not pose problem for the presented approach. It is clear, however, that memory and computational cost

---

Under review as a conference paper at ICLR 2021
Denoising perfor- mance in PSNR (dB) for the 'house' image for different al- gorithms with different opti- mized hypee-parameters. The top category only requires the noisy image. The mid- dle requires additional infor- mation such noise level (KSVD, WNNM, BM3D) or additional noisy images with matched noise level (n2vl). The bottom requires large clean datasets.
Table 2:
0=15
0=
a=
N2V*
32.05
29.20
25.42
MTMKL
34.29
31.88
28.08
GSC
33.78
32.01
28.35
S5C
33.50
32.08
28.35
VAR-BSC
33.50
32.32
28.91
TVAE
34.27 .02
32.65 .06
29.98 + .05
N2V
33.91
32.10
28.94
KSVD
34.32
32.15
27.95
WNNM
35.13
33.22
30.33
BM3D
34.94
32.86
29.37
EPLL
34.17
32.17
29.12
BDGAN
34.57
33.28
30.61
DPDNN
35.40
33.54
31.04
increase with the number of data points. Above, we processed 100, 000 data points which is still feasible for the small DNNs used. However, larger DNNs increase computational load significantly
because N x (| (*) + 2 (*) l)
latent states have to be passed through the decoder. Furthermore, larger DNNs require more data points not overfit which further increases computational load of our N-dependent method. In many applications, there is, however, anyway relatively few data available which makes the application of large DNNs prohibitive. One example the task of "zero- shot' denoising, i.e., denoising of an image when only the image itself is available. Learning without clean data recently became very popular. The task is currently addressed using approaches based on standard ferd-forward DNNs whose training objectives have been altered to allow for training on noisy images (e.g. Lehtinen et al., 2018; Krull et al., 2019). Deep generative models are, on the other hand, more naturally suited for training noisy data as their learning objective directly. Shocher et al. (2018) also argue that smaller DNNs are sufficient for the 'zero-shot' setting. Because of its recent popularity and suitability for approaches with smaller DNNs, we consequently focus on 'zero-shot' denoising. As a very significant additional benefit, the task allows for directly comparing the VAE approach to large range of other approaches that have recently been suggested. Most notably we can compare to non-deep generative models, large needffowwrrd DNNs (Zhu et al., 2019: Dong et al., 2019) and DNNs dedicated learning from noisy data (Lehtinen et al., 2018; Krull et al., 2019).
be used
The one denoising benchmark that offers the broadest possible comparison other methods is probably the 'house" image (Fig. left). The standard benchmark settings for 'house' make use of additive Gaussian white noise with standard deviations a E {15, 25, 50). First, consider the com- parison in Tab. where all models used the same patch size of D 8 x 8 pixels and H 64 latent variables (Appendix for details). Tab. I lists the different approaches dard measure of peak signal-to toiee ratio (PSNR). Values for MTMKL (Titsias & Greailla,,,,, 2011), 1), GSC (Sheikh et al., 2014) and S5C (Sheikh & Lücke, 2016) were taken from the respective original publications (which all established new state-of-the-an results when first published). As be observed, TVAE significantly improves performance for high noise levels. TVAE is able to learn the best data representation for denoising and represents the state-of-the-art in this controlled setting (i.e., fixed D and H). The decoder DNN of TVAE provides the decisive performance advantage: TVAE significantly improves performance compared to the linear Binary Sparse Coding (var-BSC, Henniges et al., 2010; Shelton et al., 201 1), confirming that the high lower bounds of TVAE on nat- ural images translate into improved performance on a conerete benchmark. For 25 and o 50, TVAE also significantly improves on MTMKL, GSC, and S5C. These three approaches are based on spike-and-slal sparse coding model (also compare Goodfellow et al., 2012). Despite the less flexible Bernoulli prior, the decoder DNN of TVAE provides the highest PSNR values for high noise levels.
terms of the stan-
In order to further extend our comparison, in the last experiment we considered the denoising task without controlling for equal conditions. Concretely, we allowed for any approach that performs denoising the benchmark including approaches that trained on large image datasets and/or use different patch sizes (including multi-scale and whole image processing). Note that different approaches may employ very different sets of hyper-parameters that can be optimized for denoising

---

Under review as a conference paper at ICLR 2021
performance: for sparse coding approaches, paper--rrameterss include patch and dictionary sizes; for DNN approaches they include all network and training scheme hyper-parameters. parameters. By allowing for comparison in this less controlled setting, we can include number of recent approaches including large DNNs trained on clean data and training schemes specifically targeted to noisy training data. Tab. 2 shows the denoising performance for the three noise levels we investigated, with results for other algorithms taken from their corresponding original publications unless specified otherwise. For WNNM and EPLL we cite values from Zhang et al. (2017). The results reported for noise2void (n2v, Krull et al., 2019) were produced specifically for this work (see Appendix B).
Note that the best performing approaches in Tab. 2 cannot be trained on noisy data; EPLL (Zoran & Weiss, 2011), BDGAN (Zhu et al., 2019) and DPDNN (Dong et al.. 2019) all make use of clean training data (typically hundreds of thousands of data points or more). For denoising, EPLL also requires the ground-truth noise level of the test image. Ground-truth noise level information is also required by KSVD (Elad& & Aharon, 2006) and WNNM (Gu et al., 2014). As noisy data very frequently occurring, removing the requirement of clean data has been of considerable interest with, e.g., approaches like noise2noise (n2n, Lehtinen et al., 2018) and noise2void being very actively discussed currently. The n2n approach can achieve denoising performance which is almost as high as the performance of given DNN when trained on clean data. would thus outperform all approaches in Tab. 2 except for the bottom three. However, n2n requires different noise realizations of the very same underlying image. noise2void aims to remove this artificial assumption. Considering Tab. 2, PSNR values of TVAE were consistently higher than those of n2v even if n2v was trained on external data with matched- TVAE is 0.2dB lower than BM3D for o 25 and 0.6dB higher for 50, which makes it, for large noise levels, the state-of-thee art on this benchmark in the 'zero-shot' setting (i.e., the setting n2n and n2v aim to address).
noisy training data
level (n2v" in Tab. 2). Performance of
4 DISCUSSION
We investigated a novel way to train VAEs with binary latents. In order to avoid derivatives w.r.t. stochastic discrete latents, we here changed the standard training setup substantially. Updates of the decoder DNN now involve a weighted sum over states (4) and the encoder DNN is replaced by discrete evolutionary optimization. The direct optimization of the encoder replaces methods that are usually considered indispensable for the training of VAEs: sampling approximation and reparameterization trick. Furthermore, the here investigated encoding model does not use joint mapping for all datapoints to latent space, i.e., the approach is not amortized. While amortization can be advantageous information can be shared across datapoints, disadvantages in terms of less tight lower bounds have also been pointed out (e.g. Kim et al., 2018; Cremer et al., 2018). Related to this point, standard VAE training usually involves factored Gaussian approximations of VAE posteriors which can introduce biases (c.g., discussion by Vértes & Sahani, 2018). The investigation of alternatives may therefore, more generally, shed further light on the consequences of specific approximation choices used to define VAE encoders.
The price we pay for not using amortization is efficiency: we optimize variational parameters for each data point which is, of course, more costly. However, direct optimization can scale VAEs to large latent spaces smaller DNNs are used. When the use of large DNNs is anyway prohibitive because of limited data, the here studied approach can play out its effectiveness. For the recently popular task of 'zero-shot" denoising, we observed state-of-the-art results a domain where VAËS have not been reported to be competitive before. The competitive performance is presumably due to the approach not being subject to an amortization gap as well as not being based on factored variational distributions.
Our conclusion is consequently that direct discrete optimization can serve as an alternative for train- ing discrete VAEs. In sense, the approach can be considered brute-force optimization which is slower than conventional amortized training but more effective for scales at which it can be applied. To our knowledge, the approach is also the first training method for VAEs that is not using sampling- based gradient estimates, and the first which makes VAEs competitive for 'zero-shot' denoising.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using reconstruc- tion probability. Special Lecture on IE, 2(1), 2015.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In Advances in neural information processing systems, pp. 153-160, 2007.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation, 2013.
Manuel Blum, Robert W Floyd, Vaughan Pratt, Ronald L Rivest, and Robert E Tarjan. Time bounds for selection. Journal of computer and system sciences, 7(4):448-461, 1973.
Samuel Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. In Proceedings of the Twentieth Conference on Computational Natural Language Learning (CoNLL)., 2016.
Harold Burger, Christian J Schuler, and Stefan Harmeling. Image denoising: Can plain neural networks compete with bm3d? In 2012 IEEE conference on computer vision and pattern recog- nition, pp. 2392-2399. IEEE, 2012.
Subhajit Chaudhury and Hiya Roy. Can fully convolutional networks perform well for general image restoration problems? In International Conference on Machine Vision Applications, pp. 254-257, 2017.
Chris Cremer, Xuechen Li, and David Duvenaud. Inference suboptimality in variational autoen- coders. In International Conference on Machine Learning, pp. 1078-1086, 2018.
W. Dong, P. Wang, W. Yin, G. Shi, F. Wu, and X. Lu. Denoising Prior Driven Deep Neural Network for Image Restoration. TPAMI, 2019.
M. Elad and M.I Aharon. Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries. IEEE Transactions on Image Processing, 15(12)3373--3744 2006.
Yonina C Eldar and Gitta Kutyniok. Compressed sensing: theory and applications. Cambridge university press, 2012.
Dennis Forster and Jörg Lücke. Can clustering scale sublinearly with its clusters? A variational EM acceleration of GMMs and k-means. In AISTATS, pp. 124-132, 2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
I. J. Goodfellow, Courville, and Y. Bengio. Scaling up spike-and -nd- models for unsupervised feature learning. TPAMI, 3508): 2013.
Ian Goodfellow, Aaron C. Courville, and Yoshua Bengio. Large-Scale Feature Learning With Spike- and-Slab Sparse Coding. In /CML, 2012.
Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng. Weighted Nuclear Norm Minimiza- tion with Application to Image Denoising. CVPR, pp. 2862-2869. IEEE, 2014.
E. Guiraud, J. Drefs, and J. Lücke. Evolutionary Expectation Maximization. In GECCO, 2018.
M. Haft, R. Hofman, and V. Tresp. Generative binary codes. Formal Pattern Analysis & Applica- tions, 6:269-84, 2004.
M. Henniges, G. Puertas. J. Bornschein, J. Eggert, and J. Lücke. Binary Sparse Coding. In LVA/CA, pp. 450-57. Springer, 2010.
Michael C Hughes and Erik B Sudderth. Fast Learning of Clusters and Topics via Sparse Posteriors. arXiv preprint arXiv: 1609.0260 2016.

---

Under review as a conference paper at ICLR 2021
Ryuji Imamura, Tatsuki Itasaka, and Masahiro Okuda. Zero-Shot Hyperspectral Image Denoising With Separable Image Prior. In IEEE International Conference on Computer Vision Workshop, 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax, 2016.
N. Jojie and B. Frey. Learning flexible sprites in video layers. In IEEE Conference on Computer Vision and Pattern Recognition, 2001.
Amir H Khoshaman and Mohammad Amin. Gumbolt: extending gumbel trick to boltzmann priors. In Advances Neural Information Processing Systems, pp. 4061-4070, 2018.
Yoon Kim, Sam Wiseman, Andrew Miller, David Sontag, and Alexander Rush. Semi-amortized variational autoencoders. In International Conference on Machine Learning, pp. 2678-2687,
2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint IrXiv::412.69000 2014.
Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. In ICLR, 2014.
B Kiran, Dilip Thomas, and Ranjith Parakkal. An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos. Journal of Imaging. 4(2):36, 2018.
Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. Noise2void-leamiing denoising from single noisy images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- lion, pp. 2129-2137, 2019.
Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2Noise: Learning image restoration without clean data. In ICML, volume 80, pp. 2965-2974, 2018.
Guy Lorberbom, Andreea Gane, Tommi Jaakkola, and Tamir Hazan. Direct optimization through arg max for discrete variational auto-encoder. In Advances in Neural Information Processing Systems, pp. 2203-6214, 2019.
James Lucas, George Tucker, Roger B Grosse, and Mohammad Norouzi. Don't blame the elbo! a linear vae perspective on posterior collapse. In Advances in Neural Information Processing Systems, pp. 9408-9418, 2019.
J. Lücke and M. Sahani. Maximal causes for non-linear component extraction. Journal of Machine Learning Research, 9:1227-67, 2008
Jörg Lücke. Truncated Variational Expectation Maximization. arXiv preprint, arXiv:1610.03111 2019.
Jörg Lücke, Zhenwen Dai, and Georgios Exarchakis. Truncated Variational Sampling for "Black Box" Optimization of Generative Models. In LVA/IC, pp. 467-478, 2018.
Lars Maalge, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. Auxiliary deep gen- erative models. In 33rd International Conference on Machine Learning (ICMI. 2016), 2016.
Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables, 2016.
Xiaojiao Mao, Chunhua Shen, and Yu-Bin Yang. Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections. In NIPS, pp. 2802-2810, 2016.
D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int'l Conf. Computer Vision, volume 2, pp. 416-423, July 2001.
Fernando M. F. Nogueira.
Bayesian
optimization
package.
github. cer..ccooofo hccm//ffnn/ayyeiianaee mi iitionn, 2019.
10

---

Under review as a conference paper at ICLR 2021
B. Olshausen and D. Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 381:607-9, 1996.
Ladislav Rampasek, Daniel Hidru, Petr Smirnov, Benjamin Hlibe-Kains, and Anna Goldenberg. Dr.vae: Drug response variational autoencoder, 2017.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. ICML proceedings, 2014.
Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, and Douglas Eck, hierarchical latent vector model for learning long-term structure in music, 2018.
Jason Tyler Rolfe. Discrete variational autoencoders, 2016.
Aurko Roy, Ashish Vaswani, Arvind Neelakantan, and Niki Parmar. Theory and experiments on vector quantized autoencoders. arXiv preprint arXiv: 1805. 1063, 2018.
Hossein Sadeghi, Evgeny Andriyash, Walter Vinci, Lorenzo Buffoni, and Mohammad H Amin. Pixelvae++: Improved pixelvae with discrete prior. arXiv preprint arXiv: 1908. 09948, 2019.
A.-S. Sheikh and J. Lücke. Select-and-Samppe for Spike-and-Slab Sparse Coding. In NIPS, 2016.
Abdul-Saboor Sheikh, Jacquelyn A. Shelton, and Jörg Lüccee. A Truncated EM Approach for Spike- and-Slab Sparse Coding. Journal of Machine Learning Research, 15:2653-2687, 2014.
J. A. Shelton, J. Bornschein, A.-S. Sheikh, P. Berkes, and J. Lücke. Select and Sample A Model of Efficient Neural Inference and Learning. NIPS, 24:2618-2626 2011.
Jacquelyn A Shelton, Jan Gasthaus, Zhenwen Dai, Jörg Lücke, and Arthur Gretton. GP-Select: Accelerating EM Using Adaptive Subspace Preselection. Neural Computation, 297))221772200, 2017.
Assaf Shocher, Nadav Cohen, and Michal Irani. "Zero-Shot" Super Supor-Resolution using Deep Internal Learning. CVPR, pp. 3118-333 3126.....221 IEEE, 2018.
Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE Winter Confer- ence on Applications of Computer Vision (WACV), pp. (64-472. IEEE, 2017.
Michalis K. Titsias and Miguel Lázaro-Gredille Spike and Slab Variational Inference for Multi-Task and Multiple Kernel Learning. In NIPS, 2011.
Arash Vahdat, Evgeny Andriyash, and William Macready. Learning undirected posteriors by backpropagation through meme updates. arXiv preprint arXiv: 1901.0349 2019.
Aaron den Oord, Oriol Vinyals, al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 6306-6315, 2017.
J. H. van Hateren and A. van der Schaaf. Independent component filters of natural images compared with simple cells primary visual cortex. Proceedings of the Royal Society of London. Series B: Biological Sciences, 265:359-66. 1998.
Eszter Vértes and Maneesh Sahani. Flexible and accurate inference and learning for deep generative models. In NIPS, 2018
Ronald J. Williams.
Simple statistical gradient-following algorithms for connectionist rein- forcement learning. Machine Learning, 8(3):299-566. May 1992. ISSN 1573-0565. doi: 10. N0.1007/BF00992666 URL https: http:::///ooooo//1///0/ 100.///P000996696666
Kai Zhang, Wangmeng Zuo, Yumjin Chen, Deyu Meng, and Lei Zhang. Beyond a Gaussian De- noiser: Residual Learning of Deep CNN for Image Denoising. IEEE Transactions on Image Processing, 26(7):3142-3155 2017.
Kai Zhang, Wangmeng Zuo, and Lei Zhang. FFDNet: Toward a fast and flexible solution for CNN based image denoising. IEEE Transactions Image Processing, 2018

---

Under review as a conference paper at ICLR 2021
Shipeng Zhu, Guili Xu, Yuehua Cheng, Xiaodong Han, and Zhengsheng Wang. BDGAN: Image Blind Denoising Using Generative Adversarial Networks. In Chinese Conference on Pattern Recognition and Computer Vision, pp. 241-252, 2019.
Daniel Zoran and Yair Weiss. From Learning Models of Natural Image Patches Whole Image Restoration, In IEEE International Conference on Computer Vision, pp. 479-486, 2011. 1.
12

---

Under review as a conference paper at ICLR 2021
A
DETAILS OF ENCODER AND DECODER OPTIMIZATION
General Decodingg Model
Standard Decoding Model
Binary Decoding Model
~~ Pe (2)
8' N (~N
1 ~ Bern (2)
+
W.
W2
2~ pe (x
I N (Z:
2 N~N (3. (())
N
N
N
Figure 5: From left right: generic VAE decoding model, continuous- latent VAE model with Gaussian noise and the binary-latent VAE model of Eqn. (1), in plate notation.
See Fig. 5 for a graphical comparison between the decoding models of a vanilla VAE and the binary VAE considered here (I). Fig. 6 graphically illustrates different steps optimize standard VAEs, and additional steps suggested by different contributions in order to optimize discrete VAEs.
For the optimization of the binary VAE (1), consider the original form of the lower bound, Eqn. (2). When taking derivatives of F(Ø, Q) w.r.t. è we can ignore the entropy term'. For the binary VAE model of Eqn. (1) the gradient of the lower bound w.r.t. W then given by:
V W F(O, 0) = Σ V log (pe (n) (2) pe)(())) 9%
Σ V log (pe (f( 2))) (n)
= -E V log (E, ()),
=
36
W Σ
(m) | p0(z, W)ll
20
i)
= 20
4 (3)
V
w - fl W)Il
(11)
(")
where the weighting factors d (z) are by using (3) and (1) given by:
(=(((n) 2) pe
20 (n)
(z)
Exregl (e(F(n) 3") (=
(n)
(ZZ W) 12 TTZ)
exp
2n
(12)
Σ exp
20 (m)
pizz, W) TT
2'e4(1) (m)
for all E क where 7n log (1-mh). Note that the 44 (E) are evaluated the current values of the parameters θ they are therefore treated as constant, e.g., for the gradient w.r.t. W.
(r)
It may be interesting to compare the gradient estimate (11) to the gradient estimate of conventional VAE training. For this consider a standard encoder given by an amortized variational distribution
which we shall denote by =(m) (Z). The distribution ((n) (z) could be a Gaussian whose mean and variance are set by passing data point (*) through encoder DNNs. For discrete VAEs, (()) (z) can be thought of as an analog discrete distribution. If we now take gradients of (6) w.r.t. W and estimate
For our choice of variational distributions. it is not trivial that the entropy term actually can be ignored because the encnding model qa($;; d') in (3) is defined in terms of the decoding model and its parameters. For truncated distributions, however, it can be shown that the entropy
still be ignored (Läcke, 2019).
13

---

Under review as a conference paper at ICLR 2021
(m) using samples from do (2), we obtain the familiar form:
Ww.F(1, Q)
E V ( log (po pe) (3)) ml
= V log ( W), o o21))
M V (7( W)l?, where x (m) (3) (2)
20
m=1
We can slightly rewrite this expression to obtain:
1 -
FwF( R
(13)
20 2
M
If we now compare with the gradient using the truncated approximation 0660
F(
1 M 20
Σ
Is (n))
(z)
Vw
(= م0 W)l
(14)
n ZEe(")
one can discuss analogous roles played by the subsets T (x) (the variational parameters of d
(1))
(#))
and by a standard encoder of The states in a subset & are used to estimate the gradient similar to the samples from a standard encoder 76 (#). The size of + (m)
can consequently be thought of as analog to the number of samples used in a conventional estimation of the gradient. Standard VAE training estimates the gradient by weighting all samples equally (with (1/M)) and the gradient di-
rection is approximated using sufficiently many samples drawn from the current T6 (E). In contrast,
truncated gradient estimation uses the states in + (n) and the gradient is computed using a weighted
(n) (z). These weights are computed by passing the states ă through the summation with weights gb.
decoder network. The gradient is then, notably, not a stochastic estimation but exact: gradient ascent is guaranteed (for small steps) to always monotonically increase the variational lower bound.
Computational Complexity. To add to the discussion of computational complexity of TVAE com- pared to standard VAE training, consider again Eqns. 13 and 14. If as many samples M are used, per data point, as there are states in each + (") then both sums have the same number of summands. The evaluation of the gradients of the mean square error (MSE) is consequently precisely the same for (#) have to be computed for TVAE. However, the weighting factors just represent a small overhead because the evaluation of the decoder DNN for the states in + (n) is computation that can be reused from the updates of $" (n)
both approaches. The additional weighting factors t (E)
The main computational differences are in the updates of 4 (") compared to the update of encoder DNNs for conventional VAEs. Once the parameters θ (W, .) are updated using (14), new states for o" (x) have to be sought based on criterion (9). In practice and for each n, we generate M' new states according to the applied evolutionary procedure. To select the best states have to pass all these M' new states through the decoder DNN to evaluate (9). Furthermore, we have
to pass all M states already in x (n) through the DNN to re-evaluate (9) because the parameters θ have changed. In summary, we do require ONN x (M + M')) passes through the decoder DNN. Selecting the MM best states from the LM+M')) states does not add complexity this can be done in OM + M') for each n (Blum et al., 1973). The EA does add to the computational load but parent selection and mutation only add a constant offset for each of the considered states.
For comparison with standard VAEs, if we use M samples of an encoder g(")) (3), we require O(M x N) passes through the decoder DNN to update the parameters @ according to (13). For the encoder update, requires N M passes through encoder and decoder DNN estimate the gradient w.r.t. the encoder weights (if we draw M samples for each data point from conventional encoder distribution (n)) (z). The additional overhead to actually draw the samples is usually negligible.
Hence, the computational complexity of TVAE training is comparable if M x M' x M. However, conventional VAE training is amortized, i.e., the update of encoder weights uses information from all
14

---

Under review as a conference paper at ICLR 2021
Additional methods used for VAEs with discrete latents
Standard
Alternative
Optimization lower bound instead of
Optimization of lower instead likellhood
Introduce smoothing continuous latents (Rolfe, 2017)
Use suitable posterior approximation
or
Use truncated posterior approximation
Add vector quantization step, commitment loss Ray et al., 2018)
Reparameterize
compact one-term
posterior obtain low-variance estimates gradients of objective w.r.t. posterior parameters
or
Soften discrete distriautions recover them as limit cases through annealing (jang al., 2017, Maddison al., 2017)
Use efficient discrete optimization to maximize objective posterior parameters
Use DNNs as posterior non-linearities
Apply Gumbel-Max reparameterizatin loss minimization (Lorberbom et (2019)
Use stochastic gradient ascent optimize objective w.r.t. posterior parameters
Figure 6: Standard series of methods applied to optimize the encoding model of VABs. Left: meth- ods applied for encoding models of standard VAEs. Middle: additional methods applied to maintain standard procedure of encoding model optimization also for discrete latent variables. Right: al- ternative approach to optimize the VAE encoding model using direct discrete optimization.
data points n. contrast, TVAE training not amortized, i.e., the + (n) are updated per data point. The advantage of amortization is that in practice, weights of a conventional encoder can converge faster or (alternatively) less samples M are required. Considering the observed runtimes, more efficient conventional VAE training can presumably using amortization. Furthermore, the used number of samples M for conventional VAE training is
large parts attributed to faster convergence
()) 64 and c (m) 200 for usually smaller than best working sizes of op (n) (we used, e.g.,
|
(*) results in overhead computations. On the other hand, amortization also has disadvantages (e-g. Kim et al., 2018: Cremer et al., 2018). The competitive performance for denoising may consequently be attributed at least in part to TVAE not being subject to an amortization gap.
denoising, see Tab. 3); and the required storage of Φ
B DETAILS ON THE NUMERICAL EXPERIMENTS
B. VERIFICATION ON GROUND- TROUUD--RUUHH DATA
We first evaluated TVAE training artificial datasets with known ground-truth parameters and log- likelihood, in order to verify the correct functioning of the algorithm and to investigate possible local optima effects. The dataset consisted of 500 4x4 images generated by linear superposition of vertical and horizontal bars, with a small amount of Gaussian noise. The DNN's input and middle layers had units each. The ((n) variational sets consisted of 64 hidden states each. Fig. shows the evolution of the that achieved the highest ELBO value of ten. All parameters were correctly recovered, and the ELBO value was consistent with actual ground -tuth ligeliieldoood
Such simple test, however, also be solved by linear models. In order to demonstrate that TVAEs can solve non- linear problems, taking advantage of the neural network non- non-linearity embedded the generative model, introduced correlations between pairs of bars: the bars combinations shown
15

---

Under review as a conference paper at ICLR 2021
Epach
One-hot
10
25
50
Figure 7: TVAE training on simple bars data: noiseless output of the TVAE's DNN for the possible one-hot input vectors over several training epochs. Generating parameters are in the last row.
log /laee pf (27)
ground trufs
-184.49
-183.15
19.17
18.73
18.89
18.54
17.34
Figure 8: Correlated bars test. The plot shows the ratio between inferred and ground-truth log- likelihoods log pe (z') of datapoints with interesting bar combinations. The inferred values are re- ported below the datapoints themselves.
in the first two datapoints from the left in Fig. 8 were discouraged from appearing together. We employed the same evolutionary scheme and again we selected the run with highest peak ELBO value out of ten. The model correctly learns that certain combinations of bars are much more unlikely than others, and correctly estimates their likelihood.
Fig. 9 offers some more insight into the correlated bars test experiment described. The left section of the figure shows the generative parameters for the dataset used: Wo the 8x8 weight matrix of the top-to-middle layer: this makes it so that the activation of the first latent variable inhibits activation of the second, and activation of the last latent variable inhibits activation of the last. Concretely, this results in dataset where these specific bars combinations weights Wi, visualized as 8 4x4 matrices, generate the actual bars. o² was set to 0.01 and the dataset contained average of two superimposing bars per datapoint (Тy 2/8 for each h).
discouraged from appearing. The
The middle section of the figure shows the ELBO values (averages over all batches for each epoch) training progresses. The cyclic learning rate schedule responsible for the oscillatory behavior.
The right section shows some example datapoints together with samples from the trained TVAE model that reached the highest ELBO value of the runs.
Ground-trath Wa (8x8)
ELBO vs epochs
Example datapoints
Ground-truth Wi (8xc16 showed 8xcee)
Trained TVAE samples
-15
45
Figure 9: From left to right: generative parameters for the correlated bars test; ELBO values over epochs for 10 runs; example datapoints and samples from the generative model.
16

---

Under review as a conference paper at ICLR 2021
B.2 DENOISING
Given a trained TVAE with parameters θ we estimated the value of pixel in a single patch as xa
polzalz) When using pe (Ir JF) E(s Pe Z)pe (3 ()) we obtain:
(Ira) pe (2)
(Ma(())
(15)
The expectation value on the right-hand hatd-hhan-sdde of Eqn. (15) is then approximated based on the encod- ing parameters & (*) using truncated posteriors. Finally, we took a weighted average of the estimates of a pixel value in different patches (see, e.g., Burger et al., 2012) in order to generate the pixel values of the full denoised image.
In Tab. 3 we list the exact byper-parameet used to obtain the PSNR values reported. In parentheses, the parameters for the run on data with noise level 50 and unconstrained aaprrpaarmeters are given, when they differ from the other experiments.
Hyper-pand parameters for the denoising experiments on the house image.
Table 3:
Table 4: Denoising performance of n2v in PSNR (dB) for the 'house' image with AWG noise. For comparison, we additionally list the performance of TVAE (numbers copied from Tab. 2). PSNR values for n2v* are obtained by train- ing only the noisy image (i.c., in the same setting used for MTMKL,, GSC, var-BSC and TVAE in Tab. 2. More training data improves performance for n2v. PSNR values for n2vt show performance if additional training data in the form of noisy images with AWG noise o 25 used. Fur- ther improvements (especially for high noise) are obtained if the n2v network is trained on training data with a noise level that matches the noise of the test set (see n2v!). ). For instance, we used for n2v+ training data with o noise the "house" with = 50. See text for further details.
Neural network units
Input (H)
64 (512)
Middle
64 (512)
Output (D)
64 (144)
Cyelic Learning Rates
Min l.r.
0.0001
Max lr.
0.01 (0.05)
Epochs/cycle
20
Batch size
to de-
Evolutionary parameters
Parents
10 (5)
Children
9 (4)
0=15
o=25
o=50
Generations
4 (1)
n2v*
32.05
29.20
25.42
Size (n)
200 (64)
n2yf
32.93
32.10
20.96
n2v!
33,91
32.10
28.94
TVAE
34,27 = .02
32.65 + .06
29.98 + .05
To evaluate the performance on standard denoising benchmarks, first compared TVAE to related probabilistic sparse coding approaches such as MTMKL, GSC and var-BSC (Tab. 1). MTMKL and GSC use the data model of spike-and-slat sparse coding and for training mean-field field and trun- cated posterior approximations with pre-selection are used, respectively. Compared to MTMKL and GSC, var-BSC uses less complex data model and training scheme also based evolution- ary optimization (Guiraud et al., 2018). The denoising performance observed in the scenario with controlled conditions (Tab. shows that for high noise level (a 50). var-BSC achieves higher PSNR values than MTMKL and GSC although the method uses simpler data model. This observa- tion demonstrates the effectiveness of the evolutionary training method used by var- BSC. However, PSNR values for TVAE are significantly higher due to the higher flexibility in modeling the data distribution provided by the used DNN.
In a second step, Tab. 2 compared the performance of TVAE with respect to different denoising approaches including deterministic sparse coding (KSVD), a mixture model approach (EPLLL, non-local image processing method (WNNM) and state-of-the- art denoising methods based on deep neural networks (BDGAN and DPDNN). These approaches can be distinguished, e.g., by the amount of employed training data and by the requirement for clean data.
TVAE as well as MTMKL, GSC and var-BSC do not require clean images for training. Furthermore, be trained if only the single noisy image is available ('zero-shot" learning; compare, c.g., Shocher et al., 2018; Imamura et al., 2019). Instead, EPLL, BDGAN and DPDNN use clean training data (typically tens hundreds of thousands of data points are collected for training).
all these approaches
17

---

Under review as a conference paper at ICLR 2021
Approaches such noise2noise (n2n Lehtinen et al., 2018) and noise2void (n2v Krull et al., 2019) occupy a middle ground: they can be trained on noisy data but they typically require much larger amounts of data than, e.g., TVAE MTMKL. In the original n2v publication, for instance, 400 (noisy) 180 x 180 BSD (Martin et al., 2001) images were used to create a training dataset (this procedure also involved data augmentation; compare Krull al. 2019). For our comparison with results of Tab. 2, we used the standard, publicly available code for n2v together with the default training set (a = 25) employed in the original n2v publication. We then applied the trained n2v network to denoise the 'house' image with a 25. The resulting PSNR value was 32.10dB which 0. 0.76 lower than the PSNR value for BM3D (32.86dB) The difference is consistent with an on average B.88d lower performance of n2v compared to BM3D the BSD68 test set (see Krull et al., 2019). The same network can also be used to denoise an image with lower or higher noise level. The n2v network trained on = 25 does, for instance, result in PSNR values of 32.93d for the 'house' image with a = 15 and in 20.96d for the 'house' image with o = 50 (see n2nl in Tab. 4). Especially for high noise levels performance can be much improved, however, if the n2v network is trained using images with the same noise level as the test image. In order to do so, we followed the procedure described in the n2v publication while adapting the noise level of = 15 in one case and a = 50 for the other case. Trained on a dataset with matched noise, we then denoised the 'house' image with = 15 in the one, and a 50 in the other case (results listed n2vł in Tab. 4). The PSNR values obtained for 'house' in this meatched-noise-eee compared the scenario with unmatched noise level (e.g., for o 50 the PSNR improvement is approximately 8 dB). The much lower performance for mismatched noise for n2v is in this respect consistent with observations for standard DNN denoising for which training with the ground-truth noise level has been pointed out as important for performance (Chaudhury & Roy, 2017; Zhang et al., 2018).
scenario are much higher
The n2v approach can avoid having to know the exact noise level, e.g., if it is trained on just the single noisy image. In a last experiment, we hence investigated this 'zero-shot' denoising feature of n2v and applied the algorithm to denoise the 'house' image while using the same noisy image for training that we seek to denoise (we took the publicly available code of n2v as an example and manually adjusted hyperparameters as follows: patch" to a value of 0.4, as "Number of training epochs" we used 400 and we set the "Number of parameter update steps per epoch" to 33). The obtained PSNR values are listed n2v" in Tab. 4.
set the "Percentage of pixel to manipulate per
From Tab. 4 it can be observed, that for all considered training settings of n2v and all noise levels, PSNR values of TVAE are consistently higher than those of n2v even if n2v is trained on external data with matched-noise level. Additional parameter tuning may improve performance of n2v* to a certain extent but PSNRs are in general much lower than n2v1. While we followed for n2v" the standard hyperparameter setting of the original paper/code publication of n2v (Krull et al., 2019), we cannot exclude further improvements with parameter fine tuning for the 'house" benchmark. However, we remark that the difference of n2v+ and BM3D for the "house" benchmark is on the very same range as the differences between n2v and BM3D as reported on the BSD data set in the original n2v publication. The stronger performing BM3D is according to denoising performance the preferable comparison and as such included in Tab. 2. In terms of efficiency, the n2v approach is in general (once trained) faster than BM3D as well as TVAE, however.
PSNR values of noise2noise (n2n) are usually very closely aligned with PSNR values achievable by feed-forward DNNs. More concretely, n2n uses, for instance, RED30 network (Mao et al., 2016) which achieves 31.07 dB PSNR on the BSD300 data set if trained on clean data. If directly trained on noisy data, RED30 achieves 31.06 dB (Lehtinen et al., 2018). n2n is thus strongly performing in terms of PSNR. The caveat of n2n compared to n2v is, however, that the noisy data n2n uses is rather artificial. The pairs of images n2n is trained on consist of two different noise realization of the same underlying clean image. For real data, such a setting is only approximately occurring at most, which has motivated the n2v approach.
Like n2v, BDGAN and DPDNN are optimized for specific noise levels (specific standard deviations are used to generate the noisy training examples). EPLL is trained exclusively on clean image patches; for denoising, the algorithm requires the ground-truth noise level of the test image as input parameter. Ground-truth noise level information is also required by KSVD and WNNM.
Like all approaches in the top category of Tab. 2. TVAE does not require ground-truth noise level information, nor clean images, nor large amounts of training data. For the 'zero-shot' setting, TVAE
18

---

Under review as a conference paper at ICLR 2021
is consequently the best performing system on the 'house' benchmark. Such a high performance is notably achieved using a basic DNN and relatively small patch sizes of D = x 8 (for o = 15 50). All feed-forward DNNs for denoising use much
and o = 25) or D 12 x 12 (for larger patches (e.g., n2v use 64 x 64). That a competitive denoising performance can be achieved for small patches, in general, argues in favor for VAE approaches to denoising. Indeed, TVAE even comes close to state-of-the-art approaches (BDGAN and DPDNN) that use very intricate DNN architectures and large amounts of clean training data. We believe that such results underline the potential of the here investigated approach although the novelty of the approach is the focus rather than extensive benchmarking.
On the other hand, an important limitation of TVAE is its computational demand. For our exper- iments on the 'house' image with noise level a = 50 in Tab. we used N = 60025 patches of D = 12 x 12 pixels, which amounts to all possible non-overlapping square patches of that size that can be extracted from the image. For training and denoising we used a TVAE with H 512
latent variables, sizes of [ 1 = 64, and 512 units in the DNN middle layer of the decoder. TVAE training required 49 seconds per training epoch when executing on single NVIDIA Titan Xp GPU and 2.5 GB of GPU memory. We ran for 500 epochs which required between seven and eight hours on the single GPU. We did not observe significant changes in variational bound values or in de- noising performance after 500 epochs in any of the experiments we conducted for Tabs I and 2. Runtime complexity increased linear with the number data points N, with the dimensionality of the data D, with the number of the latents H, and with the size of the DNN used. Runtimes also increased approximately proportional w.r.t. the size of $ (m) Empirically we observed sublinear scaling with +b (n) presumably because of significant overhead computations: for example, increas- ing from 4 (n) 64 to 1 (*) 128 (while keeping all other parameters as above) computational time increases from 49 seconds per training epoch to 75 seconds.
For noise levels 15 and σ 25 in Tab. 2 we used smaller patch sizes (D 8 x 8) and fewer stochastic latents (H 64) but larger &h (n) (i.e., a (m) = 200). In general, the patch size D is increased, more structure has to be captured. This can be done either by increasing the size of the stochastic latents or by using larger DNNs. Both, in turn, requires more training data in order to estimate the increased number of parameters. In the current setup, the sizes of D which are currently feasible are comparably small. The denoising performance based on small patches is, however, notably very high.
For comparison, n2v uses up to D = 64 x 64 and also all other feod-forward DNN approaches use significantly larger patch sizes than TVAE (and the other approaches in category 1). Still, n2v can be trained efficiently on large patches requiring approximately 19 hours on a NVIDIA Tesla K80 GPU for training on approximately 3k noisy images of shape 180x 180 and seconds for the denoising of one 256x256 image. The higher computational demand of TVAE is also the reason why averaging across databases with many images (such as BSD68) or applications to large single images quickly becomes infeasible. As novel approach, TVAE is, however, far from being fully optimized algorithmically compared to large feed-forward approaches. and there is certainly further potential to improve training efficiency.
While denoising is, in general, well suited for deep generative models, performance for standard image denoising is by far not as common as such results for standard DNNs (which may also be related to efficiency aspects). An exception is a recent GAN approach (BDGAN; Zhu et al., 2019). VAEs are often evaluated using binarized MNIST with approximate log-likelihoods for comparison; that benchmark, however, is not consistent with the Gaussian noise model used here and does not allow a direct comparison with beed-forward DNNs which are the ofte-of-theearr.
19


Reviews:
[{"review_id": "WRiOFYszNFf", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**summary** \nthe paper proposes a novel approach to training variational autoencoder models, based on non-parametric form of truncated approximate posterior. Posterior is truncated to have support on a small subset of latent space allowing for exact marginalization. The support of approximate posterior in latent space for each data point $x$ is learned via evolutionary algorithm, minimizing ELBO. Method is applied to denoising tasks for images.\n\n**pros**\nthe idea of using evolutionary algorithm for approximating truncated posterior is new to me, the fact that it works in practice is very interesting. \n\n**cons**\n* The idea is limited to small architectures due to the need to evaluate the decoder for every state in the truncated set as opposed to single evaluation in conventinal approaches. The authors for this reason choose to apply their approach to denoising applications, which I am not familiar with and can't evaluate the advantage of their method.\n* The fact that approximate posterior is not amortized over the dataset is also a potential issue: for each data point $q(z|x)$ will need to be updated to match evolution of the decoder of the entire epoch. It seems plausible that applying more evolutionary steps can take care of this but it would be useful to see experiments with varying number of  evolutionary steps.\n* authors didn't compare their approach to the recent paper \"Efficient Marginalization of Discrete and Structured Latent Variables via Sparsity\" https://arxiv.org/abs/2007.01919, it would be good to understand which one works better.\n\n\ncomments:\n* The approach is reminiscent of EM approach where samples from the posterior are approximated via Metropolis-Hastings updates. This can be made more efficient by adding temperature annealing. I would be interested to see how the proposed approach compares to this.\n* it would be clearer to make x-dependence in Eq (3) explicit rather then just through superscript (n)\n* it would be interesting to see how large is gap between the proposed ELBO and true LL in the trained models, to get a sense of good the approximation is.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PUl8LBFiJj4", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an evolutionary optimization framework for training vartional autoencoders (VAEs) with discrete latents. In contrast to the standard VAE paradigm, the proposed TVAE approach does not require an encoder for amortized inference given the input. The method instead relies on a pool of latent variable samples for each data point to activate the decoder network. The latent variable pools are maintained and iteratively updated to increase the average lower-bound of the marginal log-likelihood of the input data. Experimental results show that non-linear decoders optimized by the TVAE framework outperform their linear counterparts on a denoising task.  Further results demonstrate method's competitiveness on zero-shot denoising, where a TVAE decoder is only trained on the noisy input image to reconstruct a smoothed version of the input image.\n \nThe paper is well-written and easy to follow. The work proposes an interesting alternative for optimizing VAEs which does not require an encoder network for amortized inference. I however have a number of concerns that are as follows:\n\nThe method instead imposes the overhead of maintaining and evolving a collection of latent variables for every data point, which can be both sample inefficient and memory-heavy for sizable problems. Then when or why would one trade amortized inference for the proposed approach?\n\nThe paper falls short of comparing the proposed optimization procedure with other alternatives for training standard or discrete VAEs. \n\nFrom reading the paper it is not clear how the size of the pool may need to be varied as the nature of the task or the number of latent changes.\n\nThe authors use a greedy approach to update the pool of latent samples. Does it not cause the optimization procedure to get stuck in local modes? Could one instead use MCMC type sampling approaches to enable mode jumps? \n\nWhy only denoising experiments? Can the authors use their approach for data synthesis tasks where the latents can be shown to control different attributes of the data generative process?", "extra_scores": {}, "extra_text": {}}, {"review_id": "Jw1sNnXiUBa", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use evolutionary algorithm to learn truncated deep latent variable model. The method get good performance in denoising  task. \n\nPros:\nQuality: The method seems correct \nOn denoising task, the performance of the proposed model is good. \nSignificance: Inference of Discrete VAE is an important question to address\n\nCons & questions:\nClarity: The paper is very hard to read due to many reasons: \na) It does not use commonly used notations, such as the paper use F to present ELBO. The paper seems to use theta to present the decoded mean and variance where people commonly use theta for the decoder weights etc. Under such unique notations which is not wrong, but the author did not explain them clearly either. Around equation (2), it just says that these are parameter to optimize (which made me think that these are the weights) and later on I found that weights are denoted by W.  and phi is the Bernoulli parameter for z (which was denoted as pi before) and theta is the decoded mean and variances. So it just makes reading very confusing. \nb) It is not self contained. For example equation (7) is just pointed to another paper which I checked another paper for mins and did not find equation (7)  and did not try more as the cited paper is very very long. So it would be good to clarify critical equations. \nI also wonder about Eq.(7)'s correctness as the right hand side looks like the model evidence instead of the lower bound of it. \nOther places, such as Eq.(3) it is formulated in this way in the Truncated VAE paper but not cited. \nGenerically, the reading of the paper is not easy and there are many simple things to do just to make it more clear. \n\nSignificance:\na) The author seems try to claim that people in the field never used non-amortized way to train a deep latent Gaussian model. This is very wrong, also the author did not discuss related work in this direction at all. e.g. using MCMC\nLearning Deep Latent Gaussian Models with Markov Chain Monte Carlo, http://proceedings.mlr.press/v70/hoffman17a/hoffman17a.pdf \n\nb)  VAE's contribution are two-fold, one is latent variable model and the second part if the amortized inference. The paper does not seems differ such existing separation. The paper in the end contributed a non-amortized inference method, but the discussion of the paper is very entangled.  \n\nc) As the author pointed out, the proposed method is not scalable  as it is per data point and thus it will have limited application impact. \n\nd) Experiments did not compare any highly relevant inference baselines. It does not compare any other inference methods in deep latent variable model. \n\ne) there is only one set of experiments in total and on an traditional denositing task which is very limited. More experimental settings are needed to show the usefulness of the method. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "F6HG5MP3Xzv", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new approach to train VAEs with binary latents, using an evolutionary algorithm to optimise a discrete set of variational parameters rather than the usual amortised variational model trained with gradient-based methods. The authors consider the setting of training a VAE with discrete, Bernoulli distributed latents and a continuous, Gaussian distributed output. They use a novel approach to form and train a posterior on the discrete latent space, parameterising the posterior somewhat implicitly via the sets defining a truncated posterior - in which the approximate posterior is proportional to the true posterior, but only within a subset of all points. Defining the subset of points amounts to parameterising the approximate posterior. Optimisation of these parameters amounts to a discrete search problem, which the authors tackle using an evolutionary algorithm.\n\nOverall, I find the paper very well written and straightforward to follow. Indeed, the style of writing tends to pique the interest of the reader without being colloquial. I think the motivation of the paper is well established, given the number of papers that have investigated the training of modern generative models with discrete latents. Importantly, I think the proposed approach is novel and seemingly effective. The use of the truncated posterior along with a relatively simple evolutionary optimisation procedure is quite different from the previously seen methods that generally attempt to maintain a differential proxy to the ELBO.\n\nThe proposed method does have limitations, which the authors acknowledge. Namely that the method will not scale to large datasets, since the posterior is not amortised. I think it must also be true that the evolutionary optimisation is not particularly suited to high-dimensional spaces, since it is generally accepted that search suffers in such settings. However, these limitations I do not think are overwhelming. The authors find a niche problem setting in which their method is practical - that of “zero-shot” data denoising. In this setting they demonstrate that their method (with a small VAE) outperforms the state-of-the-art methods. The authors also provide a small set of straightforward and convincing toy experiments to show that their approach satisfies basic properties that we require in a learning system, such as appropriate scaling behaviour and recovery of artificial data generating parameters. Although I think the empirical demonstration may be enough, given the method is relatively novel, it is still limited. The main experimental result of the paper is denoising on a single image. Even performing the denoising on a small set of images would help the paper, since there will undoubtedly be variance in the performance across different images.\n\nOne other thing I would say about this paper is that identifying the approach as a VAE almost feels inappropriate. It is already true that the usual VAE is not an autoencoder at all, and so the name is a slight misnomer. But with the authors’ approach here, the posterior is not even parameterised as a function of the data (although there is of course an implicit dependence established through training). Hence identifying the method as related to an autoencoder seems to be misleading (although understandable given that the method is clearly related to the VAE).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an evolutionary optimization framework for training discrete VAEs, which is different to the standard way of training VAEs. One of the main criticism of the paper was the choice of experiments, but the authors addressed this point by adding an inpainting benchmark.

Unfortunately, the reviewers' scores are borderline, and one of the reviewers pointed out the lack of scalability (more precisely, linear scalability with the number of observations) and cannot recommend acceptance based on the limited application impact. Given the large number of ICLR submissions, this paper unfortunately does not meet the acceptance bar. That said, I encourage the authors to address this point and resubmit the paper to another (or the same) venue.
Decision: Reject