Title: THE VARIATIONAL WALKBACK ALGORITHM

Abstract: A recognized obstacle to training undirected graphical models with latent variables such as Boltzmann machines is that the maximum likelihood training procedure requires sampling from Monte-Carlo Markov chains which may not mix well, in the inner loop of training, for each example. We first propose the idea that it is sufficient to locally carve the energy function everywhere so that its gradient points in the "right" direction (i.e., towards generating the data). Following on previous work on contrastive divergence, denoising autoencoders, generative stochastic networks and unsupervised learning using non-equilibrium dynamics, we propose a variational bound on the marginal log-likelihood of the data which corresponds to a new learning procedure that first walks away from data points by following the model transition operator and then trains that operator to walk backwards for each of these steps, back towards the training example. The tightness of the variational bound relies on gradually increasing temperature as we walk away from the data, at each step providing a gradient on the parameters to maximize the probability that the transition operator returns to its previous state. Interestingly, this algorithm admits a variant where there is no explicit energy function, i.e., the parameters are used to directly define the transition operator. This also eliminates the explicit need for symmetric weights which previous Boltzmann machine or Hopfield net models require, and which makes these models less biologically plausible.

1. INTRODUCTION:

Although earlier research focused on generating data through Monte Carlo Markov chains (MCMCs), e.g. with various Boltzmann machines (Salakhutdinov & Hinton, 2009) , most of the recent effort in designing deep generative models is based on single-step generation, e.g., with variational auto-encoders (VAEs) (Kingma & Welling, 2013; Rezende et al., 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2014) . However, generating a sample by going through a series of stochastic transformations that gradually improve the generated sample (or its latent representation) to make it more plausible could hold some advantages. A generative process can be seen as a mapping from simple noise variates (e.g., uniform, Gaussian) to samples from a very complicated distribution (maybe concentrated near a low-dimensional manifold) approximating the one which we are trying to learn from. If the data distribution is complex (e.g., the corresponding manifold is highly convoluted and non-linear), the generative process may involve a highly non-linear transformation which could be difficult to learn and optimize. Such highly non-linear transformations are probably best represented (and learned) by composing a large number of slightly non-linear transformations, either with a fixed-depth deep network, or with a variable depth recurrent computation, which is what the repeated application of a transition operator corresponds to.

1.1. MOTIVATIONS:

The main motivation for the paper are the following.

• The main difference between feedforward generation and recurrent generation is twofold:(1) in the recurrent case, the same parameters are used for each step of the transition operator, and (2) by providing an interpretation of each of these steps as the application of a transition operator, we can design training procedures which do not require backpropagating through all the steps of the unfolded computation (from the raw noise samples to the generated output). This is a potential that clearly deserves to be explored further and motivates the learning framework introduced here.

• Another motivation for the Variational Walkback is the idea that we only need to carve the energy function in the right direction at each point in the space of the random variables of interest, which may sideskip the need to actually sample from the stationary distribution of a Markov chain in order to obtain the gradients of the training objective. The intuition is that if the model's transition operator wants to move away from the data and into an area without data, this is a clue that the energy gradient is pointing in the wrong direction at that place. Consider a chain of samples following the model's transition operator (or variants of it at different temperatures), starting at a data point. If the chain moves us away from data points, then we can use the previous state in the chain as a target for the operator when that operator is applied to the next next state, i.e., we want to teach the operator to walk back towards the data. This intuition was already exploited by Bengio et al. (2013c) but without a firm mathematical grounding. In Variational Walkback this is rigorously justified by a variational bound.

• Yet another motivation for the particular approach presented here is that it innovates in the rarely explored direction of parametrizing directly the generative model via a transition operator, rather than via an explicit probability function or energy function. This idea has already been discussed in the context Generative Stochastic Networks (GSNs) (Bengio et al., 2013b) , a generalization of denoising auto-encoders (DAEs) (Vincent et al., 2008) which interprets the auto-encoder as estimating the gradient of an energy function (Alain & Bengio, 2014) or as a transition operator (Bengio et al., 2013c ). An advantage of being able to parametrize directly the generator is seen with GANs and DAEs: we directly parametrize and learn the function which will be used to perform the task of interest (e.g. generating answers to some questions). Instead, the traditional approach is to parametrize a probability function or energy function (e.g., with a Boltzmann machine) and then then use another procedure (the MCMC method of your choice) to sample from it and do inference. Another important reason for exploring algorithms for directly learning a transition operator is that they put less constraint on the form of the transition operator, compared with a transition operator derived from an energy function. More specifically, neural net implementations of transition operators derived from an MCMC typically require the presence of symmetric weights (due to the symmetry of the second derivative of the energy with respect to a pair of units in the neural network), as discussed by Bengio et al. (2015) . When we consider a biologically plausible implementation of these learning algorithms, the weight symmetry constraint (W ij = W ji ) is not reasonable as a hard constraint. Instead, if the transition operator (rather than the energy function) is the object being parametrized and learned, then there is no such hard constraint.

1.2. GENERAL THEORY:

We introduce a novel variational bound which is an alternative to and improves upon the traditional reconstruction error as a training objective for DAEs and GSNs. Similar variational bounds have been used for VAEs as well as for the non-equilibrium thermodynamics generative models (Sohl-Dickstein et al., 2015) . A distribution P over a chain of samples is defined, which corresponds to iteratively applying transition operators with shared parameters, starting from a pure noise initial state. We would like this process to produce training examples. An inverting flow Q is defined starting from a training example (the "walk-away" trajectory), and following the transition operator of the model, i.e., estimating the posterior distribution of the generative chain produced by P , given that it were landing at a training example. If the model does not match the data distribution, that chain Q will tend to walk away from the training samples, and we want to inhibit that by training P to "walk back". Instead of using a completely different parametrization for the variational approximation of the posterior (the Q distribution), like in VAEs and non-equilibrium dynamics, we propose to exploit the decomposition of P as a series of stochastic transformations in order to parametrize Q with the same parameters as P , with the step-wise estimated posterior matching the correct one (from P ) for all but the last step of the walk-away trajectory. To make the approximation in the last step of the chain of walk-away steps better (and thus the variational bound tighter) we introduce the idea of gradually increasing temperature at each step of the walk-away Q chain of transitions (or gradually reducing temperature, at each step of the corresponding walkback trajectory under P ). This also has the advantage that the training procedure will more easily converge to and eliminate spurious modes (those modes of the model where there is no nearby training data). This is because the walk-away Q chain will be making large steps towards the dominant and most attractive modes when the temperature becomes large enough. Unless those modes are near data points, the walkback algorithm will thus "seek and destroy" these modes, these spurious modes.

We present a series of experimental results on several datasets illustrating the soundness of the proposed approach on the MNIST, CIFAR-10 and CelebA datasets. 

EQUATION

where Z θ is the partition function

EQUATION

Let p D be the training distribution, from which a sample D is typically drawn to obtain the training set. The maximum likelihood parameter gradient is

EQUATION

which is zero when training has converged, with expected energy gradients in the positive phase (under p D (v)p θ (h|v)) matching those under the negative phase (under p θ (s)). Note that in the (common) case of a log-linear model, the energy gradient (with respect to parameters) corresponds to the sufficient statistics of the model. Training thus consists in matching the shape of two distributions, as captured by the sufficient statistics: the positive phase distribution (influenced by the data, via the visible) and the negative phase distribution (where the model is free-running and generating configurations by itself).

2.2. MIXING-FREE TRAINING FRAMEWORK FOR UNDIRECTED GRAPHICAL MODELS:

The basic idea of the proposed mixing-free training framework for undirected graphical models is the following. Instead of trying to match the whole positive phase and negative phase distributions (each of which require a difficult sampling operation, generally with an MCMC that may take very long time to mix between well separated modes), we propose to only match the shape of the energy function locally, around well-chosen points s t . Another way to think about this is that instead of trying to directly maximize the likelihood of p θ which requires expensive inference (ideally an MCMC) in the inner loop of training (for each example v ∼ p D ), we would like to learn a transition operator p T (s t+1 |s t ) such that following it at temperature T = 1 would gradually move the state s t towards the data generating distribution.

For this purpose, we propose to use a walkback strategy similar to the one introduced by Bengio et al. (2013c) , illustrated in Algorithm 1. The idea is to start from a configuration of s which is compatible with the observed data x, let the state evolve according to our transition operator, and then punish it for these moves, making it more likely to make backwards transitions on this trajectory. If learning was completed, the only moves that would remain are those between highly probable configurations under the data generating distribution. The other ones would be "punished", like a child walking away from its designated task and forced to walk back (towards the data)1 . Following the model's inclination in order to generate this random trajectory is more efficient than simply adding noise (like in the denoising auto-encoder (Vincent et al., 2008) or the non-equilibrium dynamics (Sohl-Dickstein et al., 2015) algorithms) because it makes the learning procedure focus its computation on state configurations corresponding to spurious modes to be eliminated. To make sure these spurious modes are approached efficiently, the proposed algorithm also includes the idea of gradually increasing temperature (i.e., the amount of noise) along this walk-away trajectory. At high temperature, the transition operator mixes very easily and quickly reaches the areas corresponding to large spurious modes.

Interestingly, all this comes out naturally of the variational bound presented below, rather than as something imposed in addition to the training objective.

Algorithm 

3. VARIATIONAL LOWER BOUND ON THE LOG-LIKELIHOOD:

Let us first consider a way in which our model could approximately generate samples according to our model and the associated transition operator p T (s|s ). That process would start by sampling a state s K inside a volume that contains all the data, e.g., with a broad Gaussian p * (s K ) whose variances are set according to the training data. Then we would sample s K-1 from p Tmax (s|s = s K ), where T max is a high enough temperature so that the noise dominates the signal and is strong enough to move the state across the whole domain of the data on the visible portion of the state. If σ 2 data is the maximum variance of the data (corresponding to the visible dimensions of the state) and σ 2 is the amount noise injected by the transition operator on the visible units at temperature 1, then we could pick

T max = σ 2 data σ 2 (4)

to achieve that goal. From that point on we are going to continue sampling the "previous" state s t according to p T (s|s = s t+1 ) while gradually cooling the temperature, e.g. by dividing it by 2 after each step. In that case we would need

EQUATION

steps to reach a temperature of 1. Finally, we would look at the visible portion of s 0 to obtain the sampled x. In practice, we would expect that a slower annealing schedule would yield samples more in agreement with the stationary distribution of p 1 (s|s ), but we explored this aggressive annealing schedule in order to obtain faster training.

The marginal probability of v = x at the end of the above K-step process is thus

p(x) = s K 1 p T0 (s 0 = x|s 1 ) K t=2 p Tt (s t-1 |s t ) p * (s K )ds K 1 (6)

where T t is an annealing schedule with T 0 = 1 and T K = T max and p * is the "starting distribution", such as the Gaussian of variance σ 2 data . We can rewrite this as follows by taking the log and multiplying and dividing by an arbitrary distribution q(s 1 , . . . , s K ) decomposed into conditionals q Tt (s t |s t-1 ):

log p(x) = log s K 1 q T0 (x)q T1 (s 1 |s 0 (x, )) K t=2 q Tt (s t |s t-1 ) p T0 (s 0 = x|s 1 ) K t=2 p Tt (s t-1 |s t ) p * (s K ) q T0 (x)q T1 (s 1 |s 0 = x) K t=2 q Tt (s t |s t-1 ) ds K 1 (7)

where we understand that s 0 = x. Now we can apply Jensen's inequality as usual to obtain the variational bound

EQUATION

)

This bound is valid for any q but will be tight when q(s K , s K-1 , . . . , s 1 |s 0 ) = p(s K , s K-1 , . . . , s 1 |s 0 ), and otherwise can be used to obtain a variational training objective. Note that both q and p can be decomposed as a product of one-step conditionals. Here, we can make most of the q Tt transition probabilities match their corresponding p Tt transition probabilities exactly, i.e., for 1 ≤ t < K we use q Tt (s|s ) = p Tt (s|s ).

The only approximations will be on both ends of the sequence:

• Sampling exactly from the model's p(v = x) is typically not feasible exactly (it involves the usual posterior inference, e.g., as used in VAEs) but as explained below we will exploit properties of the algorithm to approximate this efficiently. We call the chosen approximation q 1 (v). • At the last step, the optimal q T K (s K |s K-1 ) is not simply the model's transition operator at temperature T K , because this conditional also involves the marginal "starting distribution" p * (s K ). However, because we have picked T K large enough to make samples from q Tmax (s K |s K-1 ) dominated by noise of the same variance as that of p * , we expect the approximation to be good too.

3.1. ESTIMATING THE LOG-LIKELIHOOD USING IMPORTANCE SAMPLING:

In practice we cannot compute L exactly (nor its gradient), but we can easily obtain an unbiased estimator of L (or of its gradient) by sampling s K 1 from the q distributions, i.e., approximate the L integral by a single Monte-Carlo sample. This is what is done by the training procedure outlined in Algorithm 1, which thus performs stochastic gradient ascent on the variational bound L, and this will tend to also push up the log-likelihood log p(x) of training examples x. Note that such variational bounds have been used successfully in many learning algorithms in the past (Kingma & Welling, 2013; Lamb et al., 2016) .

We derive an estimate of the negative log-likelihood by the following procedure. For each training example x, we sample a large number of diffusion paths. We then use the following formulation to estimate the negative log-likelihood.

log p(x) = log E x∼p D ,q T 0 (x)q T 1 (s1|s0(x,))( K t=2 q T t (st|st-1))   p T0 (s 0 = x|s 1 ) K t=2 p Tt (s t-1 |s t ) p * (s K ) q T0 (x)q T1 (s 1 |s 0 = x) K t=2 q Tt (s t |s t-1 )   (10)

4. TRANSITION OPERATORS FOR VARIATIONAL WALKBACK:

Up to now we have not specified what the form of the transition operators should be. Two main variants are possible here. Either we directly parametrize the transition operator, like with denoising auto-encoders or generative stochastic networks, or we obtain our transition operator implicitly from some energy function, for example by applying some form of Gibbs sampling or Langevin MCMC to derive a transition operator associated with the energy function.

An advantage of the direct parametrization is that it eliminates the constraint to have symmetric weights, which is interesting from the point of view of biological plausibility of such algorithms. An advantage of the energy-based parametrization is that at the end of the day we get an energy function which could be used to compute the unnormalized joint probability of visible and latent variables. However, note that in both cases we can easily get an estimator of the log-likelihood by simply using our lower bound L, possibly improved by doing more expensive inference for p T K (s K |s K-1 ).

4.1. PARAMETRIC TRANSITION OPERATOR:

In our experiments we considered Bernoulli and isotropic Gaussian transition operators for binary and real-valued data respectively.

When we sample from the transition operator we do not attempt to pass gradients through the sampling operation. Accordingly, backpropagation is performed locally on each step of the walk-back, and there is no flow of gradient between multiple walk-back steps.

Additionally, we use a "conservative" transition operator that averages its input image together with the sample from the learned distribution (or takes a weighted average with a fixed α weighting) for the transition operator. Just after parameter initialization, the distribution learned by the transition operator's output is essentially random, so it is very difficult for the network to learn to reconstruct the value at the previous step.

Bernoulli Transition Operator:

EQUATION

F ρ , F µ , F σ are functions (in our case neural networks) which take the previous x value from the walkback chain and return estimates of the value of µ and σ respectively. T is the temperature which is dependent on the walkback step t. x t-1 is the previous value in the walkback chain.

5. RELATED WORK:



Contrastive Divergence:

This algorithm is clearly related to the contrastive divergence algorithm with k = T steps (CDk). The CD-k algorithm approximates the log-likelihood gradient by trying to match the sufficient statistics with the data clamped to the sufficient statistics after k steps of the transition operator. The parameter update is the difference of these sufficient statistics, which also corresponds to pushing down the energy of the data-clamped configuration while pushing up the energy of the random variables after k steps of the transition operator.

Two important differences are that, because the temperature is increasing in the variational walkback procedure, 1. the energy gradients ∂E(s) ∂s do not cancel each other telescopically along the chain from s 0 to s T , 2. as t increases we move more and more randomly rather than following the energy of the model, allowing to hunt more effectively the areas near spurious modes.

A third difference is that the learning procedure is expressed in terms of the transition operator rather than directly in terms of the energy function. This allows one to thus train a transition operator directly, rather than indirectly via an energy function.

Generative Stochastic Networks:

The Generative Stochastic Networks (GSN) algorithm proposed by Bengio et al. (2013b) learns a transition operator by iteratively injecting noise and minimizing the reconstruction error after a number of transition operator steps starting at a data point, and back-propagating through all these steps. One thing in common is the idea of using the walkback intuition instead of isotropic noise in order to converge more efficiently. A major difference is that the algorithm proposed for GSNs involves the minimization of overall reconstruction error (from the input data point x to the sampled reconstruction many steps later). This will tend to blur the learned distribution. Instead, the variational walk-back algorithm minimizes reconstruction error one step at a time along the walk-away trajectory.

In addition, the variational walkback GSNs require back-propagating through all the iterated steps, like the DRAW algorithm (Gregor et al., 2015) . Instead the variational walk-back algorithm only requires back-propagating through a single step at a time of the transition operator. This should make it easier to train because we avoid having to optimize a highly non-linear transformation obtained by the composition of many transition operator steps.

Non-Equilibrium Thermodynamics:

There are two main differences between the Variational Walkback algorithm and the Non-Equilibrium Thermodynamics:

1. Instead of isotropic noise to move away from the data manifold, we propose to use the model's own transition operator, with the idea that it will "seek and destroy" the spurious modes much more efficiently than random moves. 2. Instead of injecting a fixed amount of noise per time step, we increase the noise as it moves away from the data manifold, and anneal the noise when we are close to the data manifold. This way, we can quickly reach the noise prior without loosing the details of the data. Our model takes significantly fewer steps to walk away and back to the manifold, as compared to the 1000 steps used for Non-Equilibrium Thermodynamics.

Annealed Importance Sampling (AIS):

Annealed Importance Sampling is a sampling procedure. Like variational walkback, it uses an annealing schedule corresponding to a range of temperature from infinity to 1. It is used to estimate a partition function. Unlike Annealed Importance Sampling, variational walkback is meant to provide a good variational lower bound for training a transition operator.

Reverse Annealed Importance Sampling Estimator (RAISE):

RAISE is a reverse AIS, as it starts from a data point and then increases the temperature. In this way it is similar to the Q-chain in variational walkback. The advantage of RAISE over AIS is that it yields an estimator of the log-likelihood that tends to be pessimistic rather than optimistic, which makes it better as an evaluation criteria.

Like AIS, RAISE estimates the log-likelihood using a form of importance sampling, based on a product (over the chain) of the ratios of consecutive probabilities (not conditional probabilities from the model). Variational walkback does not work with estimates of the model's unconditional probability, and instead works directly with a conditional probability defined by the transition operator. It is for this reason that variational walkback does not need to have an explicit energy function).

6. EXPERIMENTS:

We evaluated the variational walkback on three datasets: MNIST, CIFAR (Krizhevsky & Hinton, 2009) , and CelebA (Liu et al., 2015) . The MNIST and CIFAR datasets were used as is, but the aligned and cropped version of the CelebA dataset was scaled from 218 x 178 pixels to 78 x 64 pixels and center-cropped at 64 x 64 pixels (Liu et al., 2015) . For all of our experiments we used the Adam optimizer (Kingma & Ba, 2014) and the Theano framework (Al-Rfou et al., 2016) . The training procedure and architecture are detailed in appendix A. We reported samples on CIFAR, MNIST, CelebA and inpainting results on MNIST. Our inpainting results on MNIST are competitive with generative stochastic networks and show somewhat higher consistency between the given part of the image and the generated portion (Bengio et al., 2013c) . However, we note that our samples on CIFAR and CelebA show the same "blurring effect" that has been observed with autoencoder-based generative models trained to minimize reconstruction loss (Lamb et al., 2016) .

7. CONCLUSION AND FUTURE WORK:

We have introduced a new form of walk-back and a new algorithm for learning transition operators or undirected graphical models. Our algorithm learns a transition operator by allowing the model to walk-away from the data towards the noise prior and then teaching it to actually to have its transitions trained to go backwards each of these walk-away steps, i.e., towards the data manifold. Variational walk-back increases the temperature along the chain as it is moving further away from the data manifold, and inversely, anneals the temperature at generation time, as it gets closer to the estimated manifold. This allows the training procedure to quickly find and remove dominant spurious modes. Learning a transition operator also allows our model to learn only a conditional distribution at each step. This is much easier to learn, since it only needs to capture a few modes per step. The model also only locally carves the energy function, which means that it does not have to learn the entire joint probability distribution, but rather steps towards the right direction, making sure that everywhere it puts probability mass as well as around the data, the energy gradient is pointing towards the data.

Our experimental results have confirmed that the model can walk towards the data manifold in a few steps, even when the modes are sharp.

Future work should extend this algorithm and experiments in order to incorporate latent variables. The state would now include both the visible x and some latent h. Essentially the same procedure can be run, except for the need to initialize the chain with a state s = ( x, h) where h would ideally be an estimate of the posterior distribution of h given the observed data point x. Another interesting direction to expand this work is to replace the log-likelihood objective at each step by a GANlike objective, thus avoiding the need to inject noise independently on each of the pixels, during one application of the transition operator, and allowing the latent variable sampling to inject all the required high-level decisions associated with the transition. Based on the earlier results from Bengio et al. (2013a) , sampling in the latent space rather than in the pixel space should allow for better generative models and even better mixing between modes. Bengio et al. (2013b) A ARCHITECTURE DETAILS

The architecture that was used for the CelebA and CIFAR dataset was similar to the architecture used by Lamb et al. (2016) , with a convolutional encoder followed by two fully connected hidden layers, followed by a decoder with strided convolutions (Radford et al., 2015) . Batch norm was applied in all layers except for the last layer. For all layers except for the last we used the tanh activation function. Surprisingly, we were unable to obtain good results using the RELU or Leaky RELU activation .

On the binarized MNIST dataset we used a transition operator with Bernoulli outputs. A feedforward neural network was used to estimate the parameters (per-pixel probabilities) for the Bernoulli outputs. This neural network consisted of a single hidden layer with 4096 hidden units and the tanh activation function.

B WALKBACK PROCEDURE DETAILS:

The variational walkback algorithm has three unique hyperparameters. One is the number of walkback steps performed during training. Another is the number of walkback steps performed when sampling from the model. Still another is the temperature schedule used during training, reconstruction, or sampling.

The most conservative hyperparameter setting would involve using a large number of walkback steps during training and slowly increasing the temperature. However, this could make training slow, and if too few steps are used, the end of the walkback chain will not match the noise prior, leading to low quality samples.

A dynamic approach to setting the number of walkback steps and temperature schedule may be possible, but in this work we set these hyperparameters empirically. We found that during training using a temperature schedule of T = T 0 √ 2 t produced good results, where T 0 = 1.0 is the initial temperature and t is the step index. During sampling, we found good results using the reverse schedule: T = √ 2 N √ 2 t , where t is the step index and N is the total number of sampling steps. For MNIST, we achieved our results using 8 training steps of walkback. For CIFAR, we used 15 training steps and 20 sampling steps. For CelebA, we used 30 training steps and 35 sampling steps. In general, we found that we could achieve higher quality results by using more steps during sampling then we used during training. We found that more difficult datasets, like CIFAR and CelebA, required longer walkback chains. Finally, our model is able to achieve results competitive with Non-Equilibrium Thermodynamics (Sohl-Dickstein et al., 2015) , despite that method requiring chains with far more steps (1000 steps for MNIST).

C ALTERNATIVE FORMULATION OF VARIATIONAL BOUND:

The marginal probability of v = x at the end of the above K-step process is thus

p(x) = s K 1 K t=1 p Tt (s t-1 |s t ) p * (s K )ds K 1 (14)

where T t is an annealing schedule with T 0 = 1 and T K = T max and p * is the "starting distribution", such as the Gaussian of variance σ 2 data . We can rewrite this as follows by taking the log and multiplying and dividing by an arbitrary distribution q(s 1 , . . . , s K ) decomposed into conditionals q Tt (s t |s t-1 ): q(s 0 , s 1 , ..., s k ) = K t=1 q Tt (s t |s t-1 ) q(s K )

giving us:

:

This analogy with a child was first used in talks by Geoff Hinton when discussing constrastive divergence (personal communication)

Figures and Tables:

Figure 1: Samples on MNIST using a Bernoulli likelihood in the transition operator, 8 walkback steps during training, and 13 walkback steps during sampling. On right. Diffusion process for sampling MNIST digits starting from bernoiulli noise. This shows how the variational walkback iteratively generates images starting from a noise prior. For intermediate steps we display samples and for the final step (right) we display the transition operator's mean.

Figure 2: Variational Walkback Inpainting MNIST the left half of digits conditioned on the right half. The goal is to fill in the left half of an MNIST digit given an observed right half of an image (drawn from validation set).

Figure 3: Original Images from CelebA (left), Variational Walkback Reconstructions (middle) and Samples (right).

Figure 4: Variational Walkback Samples on CIFAR10 (left and right).

denote the vector of visible units and h denote the vector of hidden random variables, with the full state of the model being s = (v, h). Let p θ denote the model distribution, with joint energy function E θ and parameter vector θ:

Table content: <table><tr><td>2 MIXING-FREE TRAINING FRAMEWORK BASED ON THE WALKBACK IDEA</td></tr><tr><td>2.1 MAXIMUM LIKELIHOOD TRAINING OF UNDIRECTED GRAPHICAL MODELS</td></tr><tr><td>Let v</td></tr></table>

1 VariationalWalkback(θ) Train a generative model associated with a transition operator p T (s|s ) at temperature T (temperature 1 for sampling from the actual model). This transition operator injects noise of variance T σ 2 at each step, where σ 2 is the noise level at temperature 1.

Table content: <table><tr><td colspan="2">Require: Precomputed σ 2 data , the overall variance (or squared diameter) of the data. repeat</td></tr><tr><td>T max ←</td><td>σ 2 data σ 2</td></tr></table>

Peer review (iclr2017):
This paper proposes a new kind of generative model based on an annealing process, where the transition probabilities are learned directly to maximize a variational lower bound on the log-likelihood. Overall, the idea is clever and appealing, but I think the paper needs more quantitative validation and better discussion of the relationship with prior work. In terms of prior work, AIS and RAISE are both closely related algorithms, and share much of the mathematical structure with the proposed method. For this reason, it’s not sufficient to mention them in passing in the related work section; those methods and their relationship to variational walkback need to be discussed in detail. If I understand correctly, the proposed method is essentially an extension of RAISE where the transition probabilities are learned rather than fixed based on an existing MRF. I think this is an interesting and worthwhile extension, but the relationship to existing work needs to be clarified. The analysis of Appendix D seems incorrect. It derives a formula for the ratios of prior and posterior probabilities, but this formula only holds under the assumption of constant temperature (in which case the ratio is very large). When the temperature is varied, the analysis of Neal (2001) applies, and the answer is different. One of the main selling points of the method is that it optimizes a variational lower bound on the log-likelihood; even more accurate estimates can be obtained using importance sampling. It ought to be easy to report log-likelihood estimates for this method, so I wonder why such estimates aren’t reported. There are lots of prior results to compare against on MNIST. (In addition, a natural baseline would be RAISE, so that one can check if the ability to learn the transitions actually helps.) I think the basic idea here is a sound one, so I would be willing to raise my score if the above issues are addressed in a revised version. Minor comments: “A recognized obstacle to training undirected graphical models… is that ML training requires sampling from MCMC chains in the inner loop of training, for each example.” This seems like an unfair characterization, since the standard algorithm is PCD, which usually takes only a single step per mini-batch. Some of the methods discussed in the related work are missing citations. The method is justified in terms of “carving the energy function in the right direction at each point”, but I’m not sure this is actually what’s happening. Isn’t the point of the method that it can optimize a lower bound on the log-likelihood, and therefore learn a globally correct allocation of probability mass?