Title: Improving Sampling from Generative Autoencoders with Markov Chains
Authors: Antonia Creswell
Kai Arulkumaran
Anil Anthony Bharath
Venue: ICLR 2017
Keywords: Deep learning
Unsupervised Learning
Theory

Abstract: We focus on generative autoencoders, such as variational or adversarial autoencoders, which jointly learn a generative model alongside an inference model. Generative autoencoders are those which are trained to softly enforce a prior on the latent distribution learned by the inference model. We call the distribution to which the inference model maps observed samples, the learned latent distribution, which may not be consistent with the prior. We formulate a Markov chain Monte Carlo (MCMC) sampling process, equivalent to iteratively decoding and encoding, which allows us to sample from the learned latent distribution. Since, the generative model learns to map from the learned latent distribution, rather than the prior, we may use MCMC to improve the quality of samples drawn from the generative model, especially when the learned latent distribution is far from the prior. Using MCMC sampling, we are able to reveal previously unseen differences between generative autoencoders trained either with or without a denoising criterion.

TL;DR: Iteratively encoding and decoding samples from generative autoencoders recovers samples from the true latent distribution learned by the model

Review discussion:
For the sake of understanding, can you position your paper relative to https://arxiv.org/abs/1410.6460 and https://arxiv.org/abs/1612.00005 ? Note that I'm not discussing precedence. I'd like to understand the contributions of this paper compared to the above two and GSNs.
1) Markov Chain Monte Carlo and Variational Inference: Bridging the Gap - Salimans et al. [1]
In both our approach and the approach of the authors of [1] we want to learn a transition operator p(z_t | z_{t-1}).
Our approach is to train an auto-encoder, made up of an encoder, E:x—>z and decoder, D:z—>x, modelled as neural networks with learnable parameters. To train the auto-encoder we minimise a reconstruction loss and a loss on the distribution of the encodings, w.r.t. a prior distribution p(z). We formulate a homogenous fixed p(z_t | z_{t-1}), i.e. it does not change with time. We then iteratively draw samples from p(z_t | z_{t-1}); ideally we sample until the chain converges to a stationary distribution.
In contrast, Salimans et al. [1] set up the transition operator, p_t (z_t | z_{t-1}) as a parameterised distribution which is sampled in order to calculate an unbiased MCMC estimate for the variational lower bound of p(x). Parameters of p_t are updated using gradient descent methods on the estimate of the lower bound, in order to update p_t. The lower bound is also updated and gradient methods applied again to update p_t.
Differences:
The transition operator proposed by Salimans et al. [1]  is not homogenous, but is updated after each MCMC estimation of the lower bound. In our work the parameters are learned by training the auto-encoder and are fixed before sampling. The parameters are not updated after sampling.
In addition:
i) Salimans et al. [1] use MCMC sampling to approximate a parameterised, variational lower bound on the likelihood of p(x). In ours we draw samples from a Markov chain to approximate q(z), in order to sample from q(x|z).
ii) Salimans et al. [1] learn the parameters of the transition operator by minimising the estimated, parameterised, variational lower bound. In our case the parameters of the transition operator are learned by training an auto-encoder.
iii) Salmons et al. [1] introduce auxiliary variables, which require an inverse model for the MCMC approximation of the lower bound. For practicality, this requires that the transition operator satisfies detailed balance, while our approach requires that the transition operator is ergodic.
2) Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space - Nguyen et al. [2]
i) Nguyen et al. [2] are focused on an MCMC chain that converges to an approximation of p(x). We generate samples from q(x), but can only produce good samples from q(x) by sampling q(z) and then q(x|z). Since we do not know q(z), only that p(z) is close to q(z), in practice we use MCMC sampling to efficiently approach q(z) starting from p(z), in order to generate samples from q(x) which approximates p(x).
ii) Both approaches use an auto-encoder to derive distribution models for the transition operator.
However, while we directly sample from the encoder and decoder to define a transition operator for a latent distribution (which we denote z, and they denote h), Nguyen et al. [2] use an auto-encoder to evaluate the gradient of p(h) w.r.t h to calculate terms in their transition operator.
iii) Their transition operator also includes a term to push samples towards a specific class. This requires a third classification network to be trained using labelled images. We do not do this.
iv) Section 3.2 of Nguyen et al. [2] focuses on sampling a latent representation.
However, instead of iteratively passing samples ONLY forwards through networks (i.e. no backpropagation, no explicit calculation of gradients), Nguyen et al. [2] pass latent samples through a generator and classification network and then backpropagate back to the latent space to get a new latent sample. The new latent sample is passed through the generator to generate a new image.
v) Their motivation for using the latent space was to improve diversity, ours was to prevent having to use training samples to initialise the sampling chain.
vi) Section 3.3 of Nguyen et al. [2] also focus on sampling a latent representation using an auto-encoder. They use the auto-encoder to calculate the gradient of the p(h) w.r.t. h to be used to calculate the transition, unlike our transition operator which directly uses samples from the auto-encoder.
Further, points from iv) apply.
Further, we train a DAE on x, they train the DAE on h - features extracted from a pre-trained classification network.
vii) The architecture used in Sections 3.4 and 3.5 of Nguyen et al. [2] are the most similar to our own as they do not have a separate generator network - the DAE is an encoder network along with the generator.
However, the generator network is trained with additional losses - we use only a reconstruction loss. The DAE is used to approximate the gradient of log p(h) w.r.t. h, unlike our approach which directly samples the DAE to produce latent samples which converge to q(z) and finally sampling q(x|z) to generate more realistic samples.
3) Deep Generative Stochastic Networks Trainable by Backprop - Bengio et al. [3]
i) No prior is placed on P(H), so the generative model P(X|H) may not be sampled, therefore H0 must be produced by H0 ~ P(H|X0) where X0 is a sample from the training data. This may lead to X1 ~ P(X|H0) that are very similar to examples from the training data.
ii) Bengio et al. [3] make a distinction between P(X), the true data distribution, and P_theta(X), the distribution learned by sampling the model, which we call Q(X). In our case we are interested in sampling Q(Z), in order to sample Q(X|Z), the conditional distribution learned by the model. In our case, we know the prior, P(Z), but not the distribution learned by the model, Q(Z).
While GSNs apply sampling to recover P(X), we use sampling to recover Q(Z) in order to sample Q(X|Z) and produce samples from Q(X). Bengio et al. [3] assumes that with sufficient training samples P_theta(X) —> P(X). We only want to find the Q(Z) that the encoder and decoder have learned to map image samples to and from.
iii) Further, because we constrain Z using P(Z), we are able to train an auto-encoder without noise.
The authors argues that the standard ancestral sampling from stochastic autoencoders (such as the Variational Autoencoder and the Adversarial
Autoencoder) imposes the overly-restrictive constraint that the encoder distribution must marginally match the latent variable prior. They propose, as an alternative, a Markov Chain Monte Carlo approach that avoids the need to specify a simple parametric form for the prior.
The paper is not clearly written. Most critically, the notation the authors use is either deeply flawed, or there are simple misunderstanding with respect to the manipulations of probability distributions.
For example, the authors seem to suggest that both distributions Q(Z|X) and Q(X|Z) are parametrized. For this to be true the model must either be trivially simple, or an energy-based model. There is no indication that they are speaking of an energy-based model.
Another example of possible confusion is the statement that the ratio of distributions Q(Z|X)/P(Z) = 1. I believe this is supposed to be a ratio of marginals: Q(Z)/P(X) = 1.
Overall, it seems like there is a confusion of what Q and P represent. The standard notation used in VAEs is to use P to represent the decoder distribution and
Q to represent the encoder distribution. This seems not to be how the authors are using these terms. Nor does it seem like there is a single consistent interpretation.
The empirical results consist entirely of qualitative results (samples and reconstructions) from a single dataset (CelebA). The samples are also not at all up to the quality of the SOTA models. The interpolations shown in Figures 1 and 3 both seems to look like interpolation in pixel space for both the VAE model and the proposed DVAE.
The details of changes have been included in a compiled latex document at the following link: https://www.dropbox.com/s/2pt62lb9ccke5qz/ICLR_review_response.pdf?dl=0
Below is the source code.
\textbf{Reviewer 1:} \textit{This paper attempts to learn a Markov chain to estimate a probability distribution over latent variables Z, such that $P(X | Z)$ can be eased to generate samples from a data distribution.}
\\ \\ \textbf{Reviewer 1:} \textit{1. No quantitative evaluation. The authors do include samples from the generative model, which however are insufficient to judge performance of the model}
\\\textbf{ac2211:}  Since we are interested in using autoencoders to generate new samples rather than reconstruct pre-existing samples it does not make sense to quote reconstruction loss. Work by Theis \cite{theis2016note} suggests that quantitative evaluation of generative models, such as using Parzen windows, is often unreliable.
Also, please note that the primary contribution is in the {\em sampling} of generative models, rather than their training.
\\ \\ \textbf{Reviewer 1:} \textit{2. The description of the model is very unclear. I had to indulge in a lot of charity to interpret what the authors ``must be doing''.
a) What does $Q(Z)$ mean? Does it mean the true posterior $P(Z | X)$?}
\\\textbf{ac2211:}
We agree! We initially used a notation in which $P(\cdot)$ corresponded to distributions underlying the real data and $Q(\cdot)$ to distributions captured by models (i.e. trained networks). In our original notation, we used $Q(Z)$ to represent the distribution that the encoder maps data samples to. Under this original choice of notation, the encoding process was described as drawing samples from $Q(Z|X)$.
\\\textbf{Changes made:}
We have updated our notation to be consistent with that used by Kingma et al. \cite{kingma2013auto} and introduce $\hat{P}(Z)$ in place of $Q(Z)$ which we define as follows (also now included in the revised version of the paper):
\begin{quote}
\[ \int Q_{\phi}(Z|X)P(X)dX=\hat{P}(Z) \]
where $Q_{\phi}(Z|X)$ is the encoding model and $P(X)$ is the underlying distribution corresponding to the training data.
\end{quote}
\textbf{Reviewer 1:} \textit{
b) What is the generative model here? Typically, it's $P(Z)P(X|Z)$.}
\\ \textbf{ac2211:} The generative model is $Q(X|Z)Q(Z)$ (in the original notation). Continuing with the original notation, an autoencoder is trained using a {\em chosen} prior distribution $P(Z)$ over the latent, to learn a conditional distribution $Q(X|Z)$ such that the marginalisation of $Q(Z|X)P(X) = P(Z)$. We argue that marginalisation of $Q(Z|X)P(X)$ does not give $P(Z)$, but rather gives $Q(Z)$, such that in order to generate meaningful data samples, $Q(X|Z)P(Z)$ is not sufficient.
Instead, we need to sample $Q(X|Z)Q(Z)$, but $Q(Z)$ is unknown. We use MCMC sampling to sample from $Q(Z)$. We hope that the new choice of notation makes the process we suggest clearer, and hence the contribution more easily appreciated.
\\ \textbf{Change made:} Using the revised notation, consistent with Kingma et al., we have added the following to the introduction, where ``This approach" refers to the sampling $P(Z)P_{\theta}(X|Z)$ as you suggested. The revised text now reads as follows:\\\\
\begin{quote}
This approach assumes that $\int Q_{\phi}(Z|X)P(X) dX = P(Z)$, suggesting that the encoder maps all data samples from $P(X)$ to a distribution that matches the prior distribution, $P(Z)$.
However, it is not always true that $\int Q_{\phi}(Z|X)P(X)dX = P(Z)$.
Rather they map to a different distribution which we call, $\hat{P}(Z)$:
$$ \int Q_{\phi}(Z|X)P(X)dX = \hat{P}(Z) $$
where it is not necessarily true that $\hat{P}(Z)=P(Z)$ because the prior is only softly enforced. The decoder, on the other hand, is trained to map encoded data samples (i.e. samples from $\int Q_{\phi}(Z|X)P(X) dX$) to samples from $X$ which have the distribution $P(X)$. If the encoder maps observed samples to latent samples with the distribution $\hat{P}(Z)$, rather than the desired prior distribution, $P(Z)$, then:
$$\int P_{\theta}(X|Z)P(Z) dZ \neq P(X)$$
This suggests that samples drawn from the decoder, $P_{\theta}(X|Z)$, conditioned on samples drawn from the prior, $P(Z)$, may not be consistent with the data generating distribution, $P(X)$.
However, by conditioning on $\hat{P}(Z)$:
$$\int P_{\theta}(X|Z)\hat{P}(Z) dZ = P(X)$$
This suggests that to obtain more realistic generations, latent samples should be drawn via $\mathbf{z} \sim \hat{P}(Z)$ rather than $\mathbf{z} \sim P(Z)$, followed by $\mathbf{x} \sim P_{\theta}(X|Z)$. A limited number of latent samples may be drawn from $\hat{P}(Z)$ using the first two steps in Approach 1 - however this has the drawbacks discussed in Approach 1. We introduce an alternative method for sampling from $\hat{P}(Z)$ which does not have the same drawbacks.
\end{quote}
We have also added Fig.1 to illustrate this more clearly.
\\ \\ \textbf{Reviewer 1:} \textit{
VAEs use a variational approximation $Q(Z | X)$ to the true posterior $P(Z | X)$. Are you trying to say that your model can sample from the true posterior $P(Z | X)$?}
\\ \textbf{ac2211:}
We are saying that (using our original notation) we may sample from the distribution to which $Q(Z|X)$ maps samples from $P(X)$, which allows us to draw more realistic looking samples from $Q(X|Z)$
\\ \textbf{Changes made:} Using the new notation, we are saying that we may sample from the distribution $\hat{P}(Z)= \int Q_{\theta}(Z|X)P(X)$, allowing us to draw more realistic looking samples from $P_{\theta}(X|Z)$.
\\ \\ \textbf{Reviewer 1:} \textit{
comments:\\
1. Using additive noise in the input does not seem like a reasonable idea. Any justification of why this is being done? }
\\ \textbf{ac2211:}
There is a long history of denoising \cite{seung1997learning} in the neural network literature as a way of learning more robust representations. Quoting from our original submission: ``denoising autoencoders (DAEs), which are motivated by the idea that learned features should be robust to ``partial destruction of the input" (Vincent et al., 2008)".
As to justification: additive noise itself is a very good approximation to imaging sensor noise, and is also widely used as a source of image corruption e.g. \cite{vincent2010stacked,alain2014regularized}.
Denoising autoencoders have also been used by Bengio et al. \cite{bengio2013generalized} in combination with MCMC.\\
\textbf{Changes made:} We have re-written the contributions section as follows to include several references to the use of denoising autoencoders and acknowledged the use of denoising VAEs from previous work:
\begin{quote}
We reformulate our original MCMC sampling process to incorporate the noising and denoising processes, allowing us to use MCMC sampling on denoising generative autoencoders. We apply this sampling technique to two models. The first is the denoising VAE (DVAE) introduced by Im et al. [-@im2015denoising]. We found that MCMC sampling revealed benefits of the denoising criterion. The second model is a denoising AAE (DAAE), constructed by applying the denoising criterion to the AAE. There were no modification to the cost function. For both the DVAE and the DAAE, the effects of the denoising crtierion were not immediately obvious from the initial samples. Training generative autoencoders with a denoising criterion reduced visual artefacts found both in generations and in interpolations. The effect of the denoising criterion was revealed when sampling the denoising models using MCMC sampling.
\end{quote}
\textbf{Reviewer 1:} \textit{
2. Approaches which learn transition operators are usually very amenable to data augmentation-based semi-supervised learning. I encourage the authors to improve their paper by testing their model on semi-supervised learning benchmarks.}\\
\textbf{ac2211:} We assume that the reviewer is referring to ``Walkback'' \cite{bengio2013generalized}. We are not using Walkback to augment our dataset for training.  We will consider doing this in future work.
The authors propose to sample from VAEs through a Markov chain [z_t ~ q(z|x=x_{t-1}), x_t ~ p(x|z=z_t)]. The paper uses confusing notation, oversells the novelty, ignoring some relevant previous results. The qualitative difference between regular sampling and this Gibbs chain is not very convincing, judging from the figures. It would be a great workshop paper (perhaps more), if the authors fix the notation, fix the discussion to related work, and produce more convincing (perhaps simply upscaled?) figures.
Comments:
- Rezende et al's (2014) original VAE paper already discusses the Markov chain, which is ignored in this paper
- Notation is nonstandard / confusing. At page 1, it’s unclear what the authors mean with “p(x|z) which is approximated as q(x|z)”.
- It’s also not clear what’s meant with q(z). At page 2, q(z) is called the learned distribution, while p(z) can in general also be a learned distribution.
- It’s not true that it’s impossible to draw samples from q(z): one can sample x ~ q(x) from the dataset, then draw z ~ q(z|x).
- It's not explained whether the analysis only applies to continuous observed spaces, or also discrete observed spaces
- Figures 3 and 4 are not very convincing.
The details of changes have been included in a compiled latex document at the following link: https://www.dropbox.com/s/2pt62lb9ccke5qz/ICLR_review_response.pdf?dl=0
Below is the source code.
\textbf{Reviewer 2:} The authors argues that the standard ancestral sampling from stochastic autoencoders (such as the Variational Autoencoder and the Adversarial
Autoencoder) imposes the overly-restrictive constraint that the encoder distribution must marginally match the latent variable prior. They propose, as an alternative, a Markov Chain Monte Carlo approach that avoids the need to specify a simple parametric form for the prior.
\\\\\textbf{Reviewer 2:}
{\it The paper is not clearly written. Most critically, the notation the authors use is either deeply flawed, or there are simple misunderstanding with respect to the manipulations of probability distributions.}
\\ \textbf{Changes made:} We are now using notation that is consistent with Kingma et al.'s \cite{kingma2013auto} VAE paper. See below.
\\\\ \textbf{Reviewer 2:} {\it For example, the authors seem to suggest that both distributions $Q(Z|X)$ and $Q(X|Z)$ are parametrized. For this to be true the model must either be trivially simple, or an energy-based model. There is no indication that they are speaking of an energy-based model.}
\\\textbf{ac2211:}
We agree that this was not the right use of the term ``parameterised''.  Under the old notation, “$Q(Z|X)$ and $Q(X|Z)$ are parametrized” - Distributions $P(Z|X)$ and $P(X|Z)$ are approximated by conditional functions $Q(Z|X)$ and $Q(X|Z)$ by learning parameters of an encoder and decoder, respectively, modelled using CNNs.
\\\textbf{Change made:} We now have the following (using the revised notation):
\begin{quote}
The process of encoding and decoding may be interpreted as sampling the conditional probabilities $Q_{\phi}(Z|X)$ and $P_{\theta}(X|Z)$ respectively. The conditional distributions may be sampled using the encoding and decoding functions $e(X;\phi)$ and $d(Z;\theta)$, where $\phi$ and $\theta$ are learned parameters of the encoding and decoding  functions respectively.
\end{quote}
\textbf{Reviewer 2:} \textit{Another example of possible confusion is the statement that the ratio of distributions $Q(Z|X)/P(Z) = 1$. I believe this is supposed to be a ratio of marginals: $Q(Z)/P(X) = 1$. }
\\\textbf{ac2211:} Under the original notation, $Q(Z|X)/P(Z) = 1$ should be $\frac{\int Q(Z|X)P(X) dX}{P(Z)}=\frac{Q(Z)}{P(Z)}=1$.
\\ \textbf{Change made:} Using notation consistent with Kingma et al. we have replaced this with the following, where we define $\hat{P}(Z)$ to be $\hat{P}(Z)=\int Q_{\phi}(Z|X)P(X) dX$; the revised text now reads:\\
\begin{quote}
... the goal is to learn a conditional distribution $Q_{\phi}(Z|X)$ such that the distribution of the encoded data samples, $\hat{P}(Z)=\int Q_{\phi}(Z|X)P(X) dX$ matches the prior distribution, $P(Z)$.
\end{quote}
\textbf{Reviewer 2:} {\it Overall, it seems like there is a confusion of what Q and P represent. The standard notation used in VAEs is to use P to represent the decoder distribution and
Q to represent the encoder distribution. This seems not to be how the authors are using these terms. Nor does it seem like there is a single consistent interpretation.}
\\ \textbf{ac2211:} Under the original notation, $P(X)$ is the true data distribution. $Q(Z|X)$ and $Q(X|Z)$ are conditional distributions captured by a learned encoder and decoder, respectively. The encoder and decoder are trained to match a prior, $P(Z)$, over the latent distribution s.t. the marginal of $Q(Z|X)P(X)$is $P(Z)$.
However, we argue that the marginal of $Q(Z|X)P(X)$ is not $P(Z)$, but rather $Q(Z)$. This means that $Q(X|Z)P(Z)$ may not generate meaningful samples. We use MCMC sampling to sample from $Q(Z)$ in order to generate meaningful data samples via $Q(X|Z)Q(Z)$.
\\ \textbf{Changes made:} Using notation consistent with Kingma et al's VAE notation. We have now added the following to the introduction:
\begin{quote}
The process of encoding and decoding may be interpreted as sampling the conditional probabilities $Q_{\phi}(Z|X)$ and $P_{\theta}(X|Z)$ respectively. The conditional distributions may be sampled using the encoding and decoding functions $e(X;\phi)$ and $d(Z;\theta)$, where $\phi$ and $\theta$ are learned parameters of the encoding and decoding functions respectively.
\end{quote}
And made it clear that we do not make the assumption that:
\begin{quote}
$\int Q_{\phi}(Z|X)P(X) dX = P(Z)$, suggesting that the encoder maps all data samples from $P(X)$ to a distribution that matches the prior distribution, $P(Z)$.
However, it is not always true that $\int Q_{\phi}(Z|X)P(X)dX = P(Z)$.
Rather they map to a different distribution which we call, $\hat{P}(Z)$:
$$ \int Q_{\phi}(Z|X)P(X)dX = \hat{P}(Z) $$
where it is not necessarily true that $\hat{P}(Z)=P(Z)$ because the prior is only softly enforced. The decoder, on the other hand, is trained to map encoded data samples (i.e. samples from $\int Q_{\phi}(Z|X)P(X) dX$) to samples from $X$ which have the distribution $P(X)$. If the encoder maps observed samples to latent samples with the distribution $\hat{P}(Z)$, rather than the desired prior distribution, $P(Z)$, then:
$$\int P_{\theta}(X|Z)P(Z) dZ \neq P(X)$$
This suggests that samples drawn from the decoder, $P_{\theta}(X|Z)$, conditioned on samples drawn from the prior, $P(Z)$, may not be consistent with the data generating distribution, $P(X)$.
However, by conditioning on $\hat{P}(Z)$:
$$\int P_{\theta}(X|Z)\hat{P}(Z) dZ = P(X)$$
This suggests that to obtain more realistic generations, latent samples should be drawn via $\mathbf{z} \sim \hat{P}(Z)$ rather than $\mathbf{z} \sim P(Z)$, followed by $\mathbf{x} \sim P_{\theta}(X|Z)$. A limited number of latent samples may be drawn from $\hat{P}(Z)$ using the first two steps in Approach 1 - however this has the drawbacks discussed in Approach 1. We introduce an alternative method for sampling from $\hat{P}(Z)$ which does not have the same drawbacks.
\end{quote}
We have also added Fig.1 to illustrate this more clearly.
\\\\\textbf{Reviewer 2:}{\it The empirical results consist entirely of qualitative results (samples and reconstructions) from a single dataset (CelebA). The samples are also not at all up to the quality of the SOTA models. The interpolations shown in Figures 1 and 3 both seems to look like interpolation in pixel space for both the VAE model and the proposed DVAE.}
\\\textbf{ac2211:}
Fig.~1 (from the original paper), showed interpolations using approaches from \textbf{previous work}, i.e. a standard VAE without sampling, {\em not our work}; these are what we aim to improve on.
Note that in Fig.~3. (original paper) the top row of each set of interpolations ie. rows (a),(e),(i),(m) are the original interpolations - which may look like interpolations in pixel space.
In contrast, rows (c),(g),(k),(o) are after 5 iterations of MCMC sampling -- these are clearly not interpolations in pixel space.
For example: the identity of people in row (c) clearly changes, while the images remain sharp.
Due to space restrictions we only showed results on the CelebA dataset, however in the supplementary material we also show results on SVHN.
\\\textbf{Changes made:}
We have removed the original Fig.~1. We have introduced a new figure (Fig.~2), to show both examples using methods from previous work and our method. We show only the results after 5 steps of MCMC sampling (rather than 1,5,and 10 as we had shown before) to make our contributions clearer.
We have updated the caption on the new Fig.~2 to make it clear that this represents ``prior work'':
\begin{quote}
\textbf{Prior work}: Spherically interpolating (White, 2016) between two faces using a VAE (a, c). In (a), the attempt to gradually generate sunglasses results in visual artifacts around the eyes. In (c), the model fails to properly capture the desired change in orientation of the face, resulting in three partial faces in the middle of the interpolation. \textbf{This work}: (b) and (d) are the result of 5 steps of MCMC sampling applied to the latent samples that were used to generate the original interpolations, (a) and (c). In (b), the discolouration around the eyes disappears, with the model settling on either generating or not generating glasses. In (d) the model moves away from multiple faces in the interpolation by producing new faces with appropriate orientations.
\end{quote}
This paper attempts to learn a Markov chain to estimate a probability distribution over latent variables Z, such that P(X | Z) can be eased to generate samples from a data distribution.
The paper in its current form is not acceptable due to the following reasons:
1. No quantitative evaluation. The authors do include samples from the generative model, which however are insufficient to judge performance of the model. See comment 2.
2. The description of the model is very unclear. I had to indulge in a lot of charity to interpret what the authors "must be doing". What does Q(Z) mean? Does it mean the true posterior P(Z | X) ? What is the generative model here? Typically, it's P(Z)P(X|Z). VAEs use a variational approximation Q(Z | X) to the true posterior P(Z | X). Are you trying to say that your model can sample from the true posterior P(Z | X)?
Comments:
1. Using additive noise in the input does not seem like a reasonable idea. Any justification of why this is being done?
2. Approaches which learn transition operators are usually very amenable to data augmentation-based semi-supervised learning. I encourage the authors to improve their paper by testing their model on semi-supervised learning benchmarks.
The details of changes have been included in a compiled latex document at the following link: https://www.dropbox.com/s/2pt62lb9ccke5qz/ICLR_review_response.pdf?dl=0
Below is the source code.
\textbf{Reviewer 3:} \textit{The authors propose to sample from VAEs through a Markov chain $[z_t \sim q(z|x=x_{t-1}), x_t \sim p(x|z=z_t)]$. The paper uses confusing notation, oversells the novelty, ignoring some relevant previous results. The qualitative difference between regular sampling and this Gibbs chain is not very convincing, judging from the figures. It would be a great workshop paper (perhaps more), if the authors fix the notation, fix the discussion to related work, and produce more convincing (perhaps simply upscaled?) figures.}
Comments:\\
\textbf{Reviewer 3:}
{\it Rezende et al's (2014) original VAE paper already discusses the Markov chain, which is ignored in this paper.}
\\\textbf{Changes made:} We have updated our section on related work to include that of Rezende et al., with the following:\\
\begin{quote}
Our work is similar to several approaches proposed by Bengio et al. \cite{bengio2013generalized,bengio2014deep} and Rezende et al. \cite{rezende2014stochastic}. Both Bengio et al. and Rezende et al. define a transition operator in terms of $X_t$ and $X_{t-1}$. Bengio et al. generate samples with an initial $X_0$ drawn from the observed data, while Rezende et al. reconstruct samples from an $X_0$ which is a corrupted version of a data sample. In contrasts Bengio et al. and Rezende et al., in this work we define the transition operator in terms of $Z_{t+1}$ and $Z_t$, initialise samples with a $Z_0$ that is drawn from a prior distribution we can directly sample from, and then sample $X_1$ conditioned on $Z_0$.
Although the initial samples may be poor, we are likely to generate a novel $X_1$ on the first step of MCMC sampling, which would not be achieved using Bengio et al.'s or Rezende et al.'s approach. We are able draw initial $Z_0$ from a prior because we constrain $\hat{P}(Z)$ to be close to a prior distribution $P(Z)$; in Bengio et al. \cite{bengio2013generalized,bengio2014deep} a latent space is either not explicitly modeled or it is not constrained.
Further, Rezende et al. explicitly assume that the distribution if $\mathbf{z}$ samples drawn from $Q_{\phi}(Z|X)$ matches the prior, P(Z). We assume the opposite, that samples drawn from $Q_{\phi}(Z|X)$ have a distribution $\hat{P}(Z)$ that does not match the the prior, P(Z) and so propose an alternative method for sampling $\hat{P}(Z)$ in order to improve the quality of generated image samples. Our motivation is also different to Rezende et al. since we use sampling to generate improved, novel samples;
while they use sampling to reconstruction corrupted samples. \end{quote}
\textbf{Reviewer 3:} {\it Notation is nonstandard / confusing. At page 1, it’s unclear what the authors mean with “$p(x|z)$ which is approximated as $q(x|z)$”.}
\\\textbf{ac2211:}
Under the old notation, the true conditional distribution $P(X|Z)$ is approximated by a learned distribution $Q(X|Z)$ which is modelled using a CNN. We introduced new notation to cope with the idea that the VAE is not ideal.\\\\
In an ideal VAE: The marginal of $P(Z|X)P(X)=P(Z)$, and the marginal of $P(X|Z)P(Z)=P(X)$, where $P(Z|X)$ and $P(X|Z)$ are ideal encoder, decoder functions and $P(Z)$ is a chosen prior distribution and $P(X)$ is the data distribution.\\\\
Realistically: The marginal of $Q(Z|X)P(X)=Q(Z)$, and the marginal of $Q(X|Z)P(Z)=Q(X)$, rather than $P(X)$, the true distribution underlying the training data. Where $Q(Z|X)$ and $Q(X|Z)$ are learned encoder and decoder functions and $Q(Z)$ is the prior of the latent distribution for the learned model.
\\\textbf{Change made:} We are now using notation consistent with Kingma et al. \cite{kingma2013auto}; we make the following changes:
\begin{quote}
The process of encoding and decoding may be interpreted as sampling the conditional probabilities $Q_{\phi}(Z|X)$ and $P_{\theta}(X|Z)$ respectively. The conditional distributions may be sampled using the encoding and decoding functions $e(X;\phi)$ and $d(Z;\theta)$, where $\phi$ and $\theta$ are learned parameters of the encoding and decoding  functions respectively.
\end{quote}
\textbf{Reviewer 3:} {\it It’s also not clear what’s meant with q(z). At page 2, q(z) is called the learned distribution, while p(z) can in general also be a learned distribution.}
\\\textbf{ac2211:} $Q(Z)$ is $\int Q(Z|X)P(X) dX$ where $P(X)$ is the data generating distribution and $Q(Z|X)$ is the encoder.
While $P(Z)$ is a known prior distribution that $Q(Z)$ is trained to match.
\\\textbf{Changes made:} We now use the notation of Kingma et al. \cite{kingma2013auto} and add the following to the paper, where ``This approach'' refers to drawing samples from the decoder using $z \sim P(Z)$ and $x \sim P_{\theta}(X|Z)$:
\begin{quote}
... $P(Z)$ is the prior distribution enforced during training and $P_{\theta}(X|Z)$ is the decoder trained to map samples drawn from $Q_{\phi}(Z|X)$ to samples consistent with $P(X)$. This approach assumes that $\int Q_{\phi}(Z|X)P(X) dX = P(Z)$, suggesting that the encoder maps all data samples from $P(X)$ to a distribution that matches the prior distribution, $P(Z)$.
However, it is not always true that $\int Q_{\phi}(Z|X)P(X)dX = P(Z)$.
Rather they map to a different distribution which we call, $\hat{P}(Z)$:
$$ \int Q_{\phi}(Z|X)P(X)dX = \hat{P}(Z) $$
where it is not necessarily true that $\hat{P}(Z)=P(Z)$ because the prior is only softly enforced. The decoder, on the other hand, is trained to map encoded data samples (i.e. samples from $\int Q_{\phi}(Z|X)P(X) dX$) to samples from $X$ which have the distribution $P(X)$. If the encoder maps observed samples to latent samples with the distribution $\hat{P}(Z)$, rather than the desired prior distribution, $P(Z)$, then:
$$\int P_{\theta}(X|Z)P(Z) dZ \neq P(X)$$
This suggests that samples drawn from the decoder, $P_{\theta}(X|Z)$, conditioned on samples drawn from the prior, $P(Z)$, may not be consistent with the data generating distribution, $P(X)$.
However, by conditioning on $\hat{P}(Z)$:
$$\int P_{\theta}(X|Z)\hat{P}(Z) dZ = P(X)$$
This suggests that to obtain more realistic generations, latent samples should be drawn via $\mathbf{z} \sim \hat{P}(Z)$ rather than $\mathbf{z} \sim P(Z)$, followed by $\mathbf{x} \sim P_{\theta}(X|Z)$. A limited number of latent samples may be drawn from $\hat{P}(Z)$ using the first two steps in Approach 1 - however this has the drawbacks discussed in Approach 1. We introduce an alternative method for sampling from $\hat{P}(Z)$ which does not have the same drawbacks.
\end{quote}
\textbf{Reviewer 3:} {\it It’s not true that it’s impossible to draw samples from $q(z)$: one can sample $x \sim q(x)$ from the dataset, then draw $z \sim q(z|x)$.}
\\\textbf{ac2211:} Under the old notation, $Q(X)$ is not the data generating distribution, $P(X)$ is. You are suggesting that we sample $Q(Z|X)P(X)$.  This would only allow us to sample $z \sim Q(Z|X)$, that is, the portion of latent space for which we had training examples for. This would not allow us to more generally sample $Q(Z)$.
Further, generating samples, by decoding encoded versions of training data samples, would give generations similar to the training samples -- rather than novel generations. This is why we insist on being able to sample from $Q(Z)$.
\\\textbf{Change made:} We have added the following to the paper:
\begin{quote}
...there are two traditional approaches for sampling generative autoencoders:
There are two traditional approaches for sampling generative autoencoders:\\
\underline{Approach 1} \cite{bengio2014deep}:
$$\mathbf{x}_0 \sim P(X) \hspace{5mm} \mathbf{z}_0 \sim Q_{\phi}(Z|X=\mathbf{x}_0) \hspace{5mm} \mathbf{x}_1 \sim P_{\theta}(X|Z=\mathbf{z}_0)$$
where $P(X)$ is the data generating distribution.
However, this approach is likely to generate samples similar to those in the training data, rather than generating novel samples that are consistent with the training data.
\end{quote}
\textbf{Reviewer 3:} {\it It's not explained whether the analysis only applies to continuous observed spaces, or also discrete observed spaces.}
\\ \textbf{ac2211:} So far, we have only considered the continuous space. We may keep discrete analysis for future work.
\\\\\textbf{Reviewer 3:} {\it  Figures 3 and 4 are not very convincing.}
\\ \textbf{ac2211:}
The caption under the original Fig.~3 may be unclear. Note that in the original Fig.~3, the top row of each set of interpolations ie. rows (a),(e),(i) and (m) are the original interpolations - which may look like interpolations in pixel space. Rows (c),(g),(k) and o are after 5 iterations of MCMC sampling. These are clearly not interpolations in pixel space, for example: the identity of people in rows (c) and (d) clearly change, and do not have the same artifacts found in rows (a),(e),(i) and (m).
The images in the original Fig.~4 may have been too small to see the effects of sampling.
\\ \textbf{Changes made:}
In Fig.~2 (new paper), we now only show the original interpolations (prior work, without MCMC sampling) and with $5$ iterations of sampling to make the figures clearer.
We have also made the samples in Fig.~4 larger so that the effect of sampling can be more clearly discerned.
We have updated our paper to include changes that reflect comments made by the reviewers. A PDF document detailing our changes  may be found at: https://www.dropbox.com/s/2pt62lb9ccke5qz/ICLR_review_response.pdf?dl=0.
Many of the changes made to our paper were related to the notation used, and the changes are best seen in Latex format rather than plain text.

Meta-review:
role: AC
data: comment: While you cite Rezende et al. 2014 when referring to VAEs, you claim the main contribution of your work is the generation of posterior samples from a Markov Chain. However, Rezende et al. 2014 presented a very similar idea. Let me quote them directly:

We do not integrate over the missing
values, but use a procedure that simulates a Markov
chain that we show converges to the true marginal distribution
of missing given observed pixels.

-- Section 5.4

Image completion can be approximatively achieved by
a simple iterative procedure which consists of (i) initializing
the non-observed pixels with random values;
(ii) sampling from the recognition distribution given
the resulting image; (iii) reconstruct the image given
the sample from the recognition model; (iv) iterate the
procedure.

-- Appendix F

How is this procedure different from your main contribution? I mean, they motivate with missing data imputation and start in a different way, but the main loop seems the same. Even if there is a significant difference between the procedures, I believe this should be addressed in the paper. I also suggest taking a closer look at appendix F, they provide some proofs which seem relevant to your work.

role: Author
data: value: comment: Thank you for your reference to this detail in Rezende et al. (2014). Indeed, the main loop is similar, and we will make note of this in the next revision. Furthermore, we could benefit from their proofs in Appendix F. Our work traces the use of MCMC in autoencoders to Bengio et al. (2013), who use this to treat denoising autoencoders as generative models.

We believe that our main contribution can be seen as rejecting the common assumption that, quoting Rezende et al. (2014) directly, "the recognition model is sufficiently close to the true posterior". This assumption is required for their data imputation, starting with a corrupted X. With our view that the model distribution, denoted Q, may not be sufficiently close to the prior, P, imputation makes less sense. Our view of these models places emphasis on how to draw latent samples to generate novel x. In particular, we advocate z ~ Q(Z), rather than z ~ P(Z). Therefore their Markov transition operator is T(X_{t+1}|X_t), whereas ours is T(Z_{t+1}|Z_t); our notation uses x and z to represent observed samples and latent samples, whereas Renzende et al. (2014) use v and xi. This difference arises from our differing motivations - under our interpretation we can use MCMC to improve upon novel samples from learned models, whereas Rezende et al. (2014) use it for data imputation from corrupted data.
role: PC
data: comment: This approach taken in this paper is topical, especially since the importance of sampling and generating diverse samples is increasingly discussed in work on generative models. There were several concerns from reviewers, in three areas particularly: connection and comparison to related work; lack of clarity and understanding of the paper; experiments that are not sufficiently convincing. These have been addressed to some extent by the authors, discussing in more detail the related work, especially in connection to Rezende et al., and GSN of Bengio et al., and with improved figures. But these points are still of concern especially in terms of assessing sample diversity in relation to much of the recent work on richer variational posterior methods and other techniques. For these reasons, the paper is not yet ready for acceptance at this years conference.

Decision: Reject