Learning Deep Latent Variable Models via Amortized Langevin Dynamics
Authors: Shohei Taniguchi
Yusuke Iwasawa
Yutaka Matsuo
Venue:  (2021)
Keywords: Langevin dynamics
amortized inference
deep generative model

Abstract: How can we perform posterior inference for deep latent variable models in an efficient and flexible manner? Markov chain Monte Carlo (MCMC) methods, such as Langevin dynamics, provide sample approximations of such posteriors with an asymptotic convergence guarantee. However, it is difficult to apply these methods to large-scale datasets owing to their slow convergence and datapoint-wise iterations. In this study, we propose amortized Langevin dynamics, wherein datapoint-wise MCMC iterations are replaced with updates of an inference model that maps observations into latent variables. The amortization enables scalable inference from large-scale datasets. Developing a latent variable model and an inference model with neural networks, yields Langevin autoencoders (LAEs), a novel Langevin-based framework for deep generative models. Moreover, if we define a latent prior distribution with an unnormalized energy function for more flexible generative modeling, LAEs are extended to a more general framework, which we refer to as contrastive Langevin autoencoders (CLAEs). We experimentally show that LAEs and CLAEs can generate sharp image samples. Moreover, we report their performance of unsupervised anomaly detection.

Under review as a conference paper at ICLR 2021
LEARNING DEEP LATENT VARIABLE MODELS VIA AMORTIZED LANGEVIN DYNAMICS
Anonymous authors Paper under double-blind review
ABSTRACT
How we perform posterior inference for deep latent variable models in efficient and flexible manner? Markov chain Monte Carlo (MCMC) methods, such Langevin dynamics, provide sample approximations of such posteriors with an asymptotic convergence guarantee. However, it is difficult to apply these methods to large-scale datasets owing their slow convergence and datapoint- wise iterations. In this study, we propose amortized Langevin dynamics, wherein datapoint- MCMC iterations are replaced with updates of an inference model that maps observations into latent variables. The amortization enables scalable in- ference from large-scale datasets. Developing a latent variable model and an infer- ence model with neural networks, yields Langevin autoencoders (LAEs), novel Langevin-baseee framework for deep generative models. Moreover, if we define latent prior distribution with an unnormalized energy function for more flexible generative modeling, LAES are extended to more general framework, which we refer to contrastive Langevin autoencoders (CLAEs). We experimentally show that LAES and CLAEs can generate sharp image samples. Moreover, we report their performance of unsupervised anomaly detection
INTRODUCTION
Latent variable models are widely used for generative modeling (Bishop) 1998 Kingma Welling 2013), principal component analysis (Wold et al. 1987). and factor analysis HHrrann 1976). To learn a latent variable model, it is essential to estimate the latent variables, z, from the observations, Bayesian inference is probabilistic approach for estimation, wherein the estimate is represented as a posterior distribution, i.c., p (z x) = p (z) p (x z) /p (x). A major challenge while using the Bayesian approach is that the posterior distribution is typically intractable. Markov chain Monte S Carlo (MCMC) methods such as Langevin dynamics (LD) provide sample approximations for pos- terior distribution with an asymptotic convergence guarantee. However, MCMC methods converge slowly. Thus, it is inefficient to perform time-consuming MCMC iterations for each latent variable, particularly for large-scale datasets. Furthermore, when we obtain new observations that we would like to perform inference for, we would need to re-run the sampling procedure for them.
In the context of variational inference, a method to amortize the cost of datapoint optimization known as amortized variational inference (AVI) (Kingma & Welling, 2013) Rezende et al., 2014) was recently proposed. In this method, the optimization of datapoint tiisee wise parameters of variational distributions is replaced with the optimization of an inference model that predicts the variational parameters from observations. This amortization enables posterior inference to be performed effi- ciently on large-scale datasets. In addition, inference for new observations can be efficiently per- formed using the optimized inference model. AVI is widely used for the training of deep generative models, and such models are known variational autoencoders (VAEs). However, methods based on variational inference have less approximation power, because distributions with tractable densi- ties used for approximations. Although there have been attempts to improve their flexibility (e.g.. normalizing flows (Rezende & Mohamed] 2015) Kingma et al. 2016 Van Den Berg et al. 2018 Huang et al., 2018). ). such methods typically have constraints terms of the model architectures (e.g., invertibility in normalizing flows).
An implementation is available at: https//bit.ly/2Shmqq3

---

Under review as a conference paper at ICLR 2021
pi Tart
2
0
e
1
Inference Model fax
UPDATE
(1)
(2)
(A)
(B1)
(B2)
Figure 1: (A) Directed graphical model under consideration. (B1) In traditional Langevin dynamics, the samples are directly updated in the latent space. (B2) Our amortized Langevin dynamics replace the update of latent samples with the update of an inference model fa|x that map the observations into the latent variables z.
Therefore, we propose amortization method for LD, amortized Langevin dynamics (ALD). In ALD, datapoint-wise MCMC iterations are replaced with updates of an inference model that maps observations into latent variables. This amortization enables simultaneous sampling from posteriors over massive datasets. In particular, when a minibatch training is used for the inference model, the computational cost is constant with data size. Moreover, when inference is performed for new test data, the trained inference model can be used as initialization of MCMC to improve the mixing, because is expected that the properly trained inference model can map data into the high-density area of the posteriors. We experimentally show that the ALD can accurately perform sampling from posteriors without datapoint-wise iterations. Furthermore, we demonstrate its applicability to the training of deep generative models. Neural networks are used for both generative and inference models to yield Langevin autoencoders (LAEs). LAEs can be easily extended for more flexible generative modeling, in which the latent prior distribution, (z), is also intractable and defined with unnormalized energy function, by combining them with contrastive divergence learning (Hin- ton 2002 Perpennn & Hinton 2005). We refer to this extension of LAES as contrastive Langevin autoencoders (CLAEs). We experimentally show that our LAEs and CLAEs can generate sharper images than existing explicit generative models, such as VAEs. Morcover, we report their performance of unsupervised anomaly detection.
2
PRELIMINARIES
2.1 PROBLEM DEFINITION
Consider probabilistic model with observations x, continuous latent variables z, and model pa- rameters 0, as described by the probabilistic graphical model shown in Figure [[). Although the posterior distribution over the latent variable is proportional to the product of the prior and like- lihood: (z x) p
(z) p (x z) pp/p(xx this is intractable owing to the normalizing constant
p (x) ʃ p p2) (x dz. This study aims to approximate the posterior p (z x) for all n ob-
servations (1)) (n)) efficiently by obtaining samples from
2.2 LANGEVIN DYNAMICS
Langevin dynamics (LD) NNal 2011 is sampling algorithm based on the following Langevin cquation:
dz = V2U (z, dt + /23-1
(1)
where U is a potential function that is Lipschitz continuous and satisfies an appropriate growth condition, B is an inverse temperature parameter, and B is Brownian motion. This stochastic dif- ferential equation has exp -B (x, x)) exp (-BU (x, x²)) dz' as its equilibrium distribution.
We set B and define the potential as follows to obtain the target posterior p (z x) its equi- librium:
U (ar, 2) =- log (z) log p (xc 2).
2

---

Under review as a conference paper at ICLR 2021
Algorithm Amortized Langevin dynamics (training time)
Initialize parameters
2(1) ((n) +-
Initialize sample sets for all datapoints
repeat
Φ 6 Φ¹ N (0': To Ei= V (,((0,2)) f2/x (z!1)) x))) 2 z
70)) + 2 U (Jo (2(11) ,...,,,........., ((N)
,...),,)) {So (xl(1)
D Add samples
until convergence of parameters return _ (n)
Algorithm 2 Amortized Langevin dynamics (test time)
+ falx (x; x²) Z Ø
b Initialize a sample using a trained inference model Initialize sample set
repeat
z' -
(W. 2n,
Update the sample using traditional L.D >> Add samples
0 - 2 { until convergence of parameters return Z
We can obtain samples from the posterior by simulating Eq. ! using the Eulcr-Maruyama method (Kloeden & Platen 2013) as follows:
z N (2': VV (z. 2), 2m
(3)
where 7) is the step size for the discretization. When the step size is sufficiently small, the samples asymptotically move the target posterior by repeating this sampling iteration. LD can applied to any posterior inference problems for continuous latent variables provided the potential energy differentiable on the latent space. However, to obtain samples the posterior p (z x) for all observations x (=),, we should perform an iteration on Eq. per datapoint as shown in Figure BB)). It inefficient particularly if the dataset is large. In the next section, we demonstrate a method that addresses the inefficiency by amortization.
AMORTIZED LANGEVIN DYNAMICS
In traditional LD, we perform MCMC iterations for each latent variable per datapoint. This is inefficient particularly if managing massive datasets. As an alternative to performing the simulation of latent dynamics directly, we define an inference model, Szix- which differentiable mapping from observations into latent variables, and consider the dynamics of its parameter & as follows:
dΦ
V
faxx (1x 2
Because function fzx outputs latent variables, the stochastic dynamics on the parameter space in- duces other dynamics the latent space and is represented the total gradient of fz|x:
dim
(2(()
Jhr dōk
dim (z
U (x10,am (2xx dt
DΦk
aok
dim
(1(
θ U 1,60),11s (al0):0)) di 22
dim 0z(1)
Σ
dB. (5)
aob
Jok
Oo
k=1
1=1,
3

---

Under review as a conference paper at ICLR 2021
MSE
ESS (1 std.)
MSCE
00
LD
0.00145
570.24 (25.20 25)
0.00136
ALD
0.00233
634.27 (40.94)
0.00151
Quantitative comparison of the sample quality be- tween traditional LD and our ALD. The mean squared error Figure 2: Groud truth posteriors (left) (MSE) between the true mean and the sample average, the and their samples by ALD (right) bi- effective sample size (ESS), and the Monte Carlo standard vaiate Gaussian examples.
Table 1:
error (MSCE) are provided as evaluation metrics.
SUALD
Sample values
Samaad values -
100000 values
Tith
Truth Imoan std!
2000 20200 170007700000000
2500 5000 7503 3100012500110 10000 175000000000 Number of MCMC steps
2000 30000
Number MCMC steps
Number of MCMC steps
Figure 3: Evolution of sample values across MCMC iterations for traditional LD and our SGALD in univariate Gaussian examples. The black lines denote the ground truth posteriors (the solid lines show the mean values, and the dashed lines show the standard deviation).
The first term of Eq. (5) approximates -V2G0 2(0) dt in Eq. and the remaining terms introduce random walk behavior to the dynamics in the Brownian term of Eq. (1) For the simulation of Eq. we use the Fuler-Maruyama method, as in traditional LD:
To 2 U 2(00 faix (zl)) 2n
(6)
where 77 is the step size. Through the iterations, the posterior sampling is implicitly performed by collecting outputs of the inference model for all datapoints in the training set as described in Algorithm When we perform inference for new test data, the trained inference model can be used as initialization of a MCMC method (e.g., traditional LD) as shown in Algorithm| 2 because is expected that the trained inference model can map data into the high-density area of the posteriors.
For minibatch training, we can substitute the minibatch statistics of m datapoints for the derivative for all n data in Eq. (6):
Σ VaU (0) 200 S2ix (2010;;0))
71
(0,
x (0) 20 = S21x (z:12;
-
(7)
M
In this case, we refer to the algorithm as stochastic gradient amortized Langevin dynamics (SGALD). SGALD enables to sample from posteriors of a massive dataset with constant com- putational cost. By contrast, performing traditional LD requires a linearly increasing cost with data size. For minibatch training of LD, adaptive preconditioning known be effective improve convergence, which is referred to as preconditioned stochastic gradient Langevin dynamics (pS- GLD) (i et al. 2015). This preconditioning technique is also applicable to our SGALD, and employ it throughout our experiments.
Figure shows a simple example of sampling from posterior distribution, where its prior and likelihood are defined using conjugate bivariate Gaussian distributions (see Appendix [F for more details). ALD produces samples that match well the shape of the target distributions. The mean squared error (MSE) between the true mean and the sample average, the effective sample size (ESS), and the Monte Carlo standard error (MSCE) are provided for quantitative comparison, as shown in Table I It can be observed that the sample quality of ALD is competitive to standard LD, even though ALD does not perform direct update of samples in the latent space. Figure 3 shows the evolution of obtained sample values by traditional LDD and SGALD for posteriors defined by
4

---

Under review as a conference paper at ICLR 2021
simple univariate conjugate Gaussian (see Appendix
F.2 for more experimental details). SGALD's samples converges much faster than traditional LD.
The advantage of our ALD over amortized variational in- ference (AVI) is the ficxibility of posterior approxima- example where the likelihood p (x z)
tion. Fs
defined using a neural network, therefore the poste- rior p (z x) is highly multimodal. AVI methods typ- ically approximate posteriors using variational distribu- tions, which have tractable density function (e.g.. Gaus- sian distributions). Hence, their approximation power is limited by the choice of variational distribution family, and they often fail to approximate such complex posteri- ors. On the other hand, ALD can capture well such poste- riors by obtaining samples. The results in other examples are summarized in Appendix
GT
AVI
ALD
Figure Visualizations of a ground truth posterior distribution (left), vari- ational distribution by AVI (center) and sample apprroximation by ALD (right) the neural likelihood example.
LANGEVIN AUTOENCODERS
Suppose we consider sampling the model parameter 0 in addition to the local latent variables from the joint posterior p (z, 0 x): then, we can ingenuously extend ALD to full Bayesian approach by combining it with standard Langevin dynamics. Herein, the prior of the model parameter p (0) added to the potential U, and is sampled using standard LD or its minibatch version, stochastic gradient Langevin dynamics (SGLD) (Welling & Tch 2011) follows:
U (X 2, log (0) Σ log (1) 0) log P an
z ((1) 100000)
(8)
0 - 0~0'~N no VoU
faix (X: Ø)) 2n²I).
(9)
where Ho is a step size. If we omit the Gaussian noise injection in Eq. 9 it corresponds to gradient descent for maximum posteriori (MAP) estimation of 0; if we additionally use flat prior for p (0), yields the maximum likelihood estimation (MLE). In this study, we assume flat prior for p (0) and omit the notation for simplicity.
Typically, the latent prior (z 0) and the likelihood p (x 2, 0) are defined as diagonal Gaussians:
((= 0) - (z: fzz, diag (o?)), ( z, 0) N (z; fxx fxiz (z: o
(10)
where Me, 2xx and of of mean and variance parameters of Gaussian distributions respectively. fx|x (z: 0) is mapping from the latent space to the observation space. The parameters of the latent prior fs and o can be included to as learnable model parameters, or be fixed to manually decided values (e.g., 2z = 0, 02 1). For the observation variance og. many existing works treat it as hyperparameter. However, its tuning is difficult, and often requires heuristic techniques for proper training (Fu et al. 2019). Instead, here, we apply a different approach, in which the variance parameter is marginalized out, and the likelihood can be calculated only with the mean parameter μ4 (see Appendix B further details). Furthermore, when the original data are quantized into discrete representation (e.g.. 8-bit RGB images). it is not desirable to use continuous distributions, such as Gaussians, as likelihood functions. Thus, we should map quantized data into the continuous space in advance. This process is often referred to dequantization (Salimans et al. 2017 Ho et al. 2019). Our ALD is also applicable as a dequantization method by formulating it as the posterior interence problem. Further detailed explanations provided in Appendix
We can choose arbitrary differentiable functions for the generative model fxix and the inference model fx x- If neural networks chosen for both, achieve Langevin autoencoder (LAE), new deep generative model within the auto-encoding scheme. The algorithm of LAAFs is summarized in Al Algorithm in the appendix.
5 CONTRASTIVE LANGEVIN AUTOENCODERS
Currently, we have dealt with the case where the latent prior distribution p (z (0) is tractable. To en- able more flexible modeling, here, consider that energy-based model (EBM) (Du & Mordatch]
5

---

Under review as a conference paper at ICLR 2021
2019
Pang et al. 2020: Han et al. 2020) is used for the latent prior follows.
exp (-fz (z:
p (= 0)
(11)
Z (0)
where fx (z; 0) is an energy function that maps the latent variable into a scalar value, and Z is exp f= (2; 0)) dz. In this case, the derivative of the normalizing constant, i.e., 7 (0) exp potential energy VoU (x, Z, 0) intractable owing to the normalizing constant. However, we can obtain the unbiased estimator the derivative by obtaining samples from the prior 0).
VoU (x,
Σ Vola (2":0) + Volog Z (0) Ve logp x
z x(0
(12)
1=1
Σ Volz (x:0 0) p Volz (z; Ve logp
(x)
(13)
See Appendix for the derivation. This algorithm used for the training of EBM known con-
trastive divergence learning (Hinton 2002 Perpira- & Hinton 2005). To obtain samples
from the latent prior, we can use standard LD as follows:
z
(z; 21(1
(14)
where 7h is a step size. However, we found that our amortized Langevin algorithm works well even for the case of sampling from unconditional prior distribution. In the unconditional case, prepare a sampler function fzju (u; ab) that maps its input u. into the latent variable z. Here, the input vector u is fixed, because the prior distribution does posterior inference case except the model parameters 0. To run multiple MCMC chains in parallel, we prepare fixed inputs u." (1)
have conditional variables in the
u (A), and update the function fzju as follows.
+ + ap/ N
2/ np E
Vasfa (1)
f2ju (z))); : : 2
(15)
where Ty is step size. Typically, the fixed input vector chosen from samples of a standard Gaussian distribution (i.e., (2) ,..., (k) / (u; O, I)). Figure 5 shows an example of sampling from a mixture of eight Gaussians using ALD. We can observe that ALD properly captures the multimodality of the true density and works well also the unconditional case. For minibatch training, we can substitute the gradient for all & chains with the stochastic gradient of m minibatch chains:
Figure 5: mixture of eight Gaussians (left) and its samples by ALD (right).
k Vo ( = faju (u(((())))))
Vwlz ((()) = = (w)
:0). ())
(16)
771
The advantage of using amortization in the unconditional case is that we can run massive chains in parallel with a constant computational cost using minibatch training. Here, we assume that the number chains equal the number of datapoints for simplicity, i.e.,
In summary, the encoder faix, the decoder fxiz. and the latent energy function fz are trained by minimizing the following loss function L. whereas the latent sampler fzju are trained by maximizing it, while stochastic noise of Brownian motion is injected in their update to avoid shrinking to MAP estimates (or MLE).
Σ fa (aax (2((); :0) ()) fa (fzju (w)) 1=1
C (0.000)
- log
(i) (0)) = falx (z(()) .)
(17)
Furthermore, when the energy function fa and the sampler falu are parameterized using neural networks, refer to the whole model contrastive Langevin autoencoders (CLAEss). At the

---

Under review as a conference paper at ICLR 2021
convergence of the CLAE's training, the inference model faix and the model parameter o match to the true posterior p (z x, 0) and p (0) X), respectively. Typically, when the number of datapoints gets infinity (i.e., n + oo), the generative model p (x 0) = I (= () p (x z, 0) dz converges to the data distribution Pdaata (x). Moreover, when the sampler function's 's outputs corresponds to the marginal latent distribution (m) [p (z æ, 0)], the first and second terms on the right hand side
in Eq. I7 are canceled out; therefore the energy function fz and the sampler function fxju also converge to equilibrium.
6 RELATED WORKS
Amortized inference is well-investigated in the context of variational inference, and it often referred to as amortized variational inference (AVI) (Rezende & Mohamed 2013 [hhu et al.) (018)) The basic idea of AVI is to replace the optimization of the datapoint- wiie variational parameters with the optimization of shared parameters across all datapoints by introducing an inference model that predicts latent variables from observations. Currently, the AVI is commonly used in fields, such as the training of generative models (Kingma & Welling, 2013), semi-supervised learning (Kingma et al. 2014), anomaly detection (An & Chol [2015), machine translation (Zhang et al. 2016), and neural rendering (Eslami et al., 2018: Kumar et al., 2018). However, in the MCMC literature, there are few works on such amortization. (Han et al., (016)) uses traditional LD to obtain samples from posteriors for the training of deep latent variable models. Such Langevin- based algorithms for deep latent variable models are known as alternating pack-propagation (ABP) and are widely applied in several fields (Xie al. 2019 Zhang et al. 2020 Xing et al. 2018 Zhu et al., 2019). However, ABP requires tatapoint-wise Langevin iterations, causing slow convergence. Moreover, when we perform inference for new data in test time, ABP requires to re-run MCMC iterations from randomly initialized samples. Although (Li et al. 2017 Hoffman 2017) propose amortization methods for MCMC, they only amortize the cost of initialization in MCMC by using an inference model. Therefore, they do not completely remove datapoint-wiss MCMC iterations.
Autoencoders (AEs) (Hinton & Salakhutdinov 2006) are a special case of LAEs, wherein the Gaus- sian noise injection to the update of the inference model (encoder) is omitted in Eq. (6). and flat prior is used for p (z 0). When a different distribution is used as a latent prior, it is known as sparse autoencoders (SAEs) Ng al.). In these cases, the latent dynamics in Eq. 55) are dominated by gra- dient V VoU; thereafter, the latent variables converge to MLE or MAP estimates, arg max, P (z x), or other stationary points. That is, ABS (and SAEs) can be regarded as MLE (and MAP) algorithms for both the parameter 6 and the latent variables z. Conversely, LAEs can be considered as a special case of (S)AEs, in which whole models are trained with SGLD instead of stochastic optimization methods like stochastic gradient decent (SGD).
Variational Autoencoders (VAFs) are based on AVI, wherein an inference model (encoder) is defined as variational distribution (z Φ) using a neural network. Its parameter Φ
is optimized by maximizing the evidence lower bound (ELBO) Eq(elars)) log
exp( ((m)))(m))))
Exg(ilass)) [U (x, =)] (q). There is a contrast between VAES and LAES relative to when stochas- tic noise is used. In VAFS, noise is used to sample from the variational distribution in the calculation of potential U, i.e., in forward calculation. However, in LAFs, noise is used for calculating gradient VaU, i.e., in backward calculation. This contrast characterizes their two different approaches to ap- proximate posteriors: the pptimization-basee approach of VAES and the sampling-based approach of LAES. The advantage of LAES over VAES is that LAES can flexibly approximate complex posteriors by obtaining samples, whereas VAES' approximation ability is limited by the choice of variational distribution q (z x: Φ) because requires a tractable density. Although there are several consider- ations in the improvement of the approximation flexibility, these methods typically have constraints normalizing flows (Rezende & Mohamed, 2015 Kingma et al. 2016 Van Den Berg et al.) 2018 Huang et al. 2018 Titsias & Ruiz, 2019), or they incur more computational costs (c.g., MCMC sampling for the reverse conditional distribution in unbiased implicit variational inference Titsiass& & Ruiz 2019)).
in terms of model architectures (e.g., invertibility and ease of Jacobian calculation
Ensrgy-based Models' training is challenging, and many researchers have been studying methodol- ogy for its stable and practical training. A major challenge is that it requires MCMC sampling from EBMs, which is difficult to perform in high dimensional data space. Our CLAEs avoid this difficulty by defining the energy function in latent space rather than data space. A similar approach is taken by
7

---

Under review as a conference paper at ICLR 2021
Table 2: Quantitative results of the image generation for BMNIST, MNIST, SVHN, CIFAR-10, 10, and CelebA. We report the reconstruction eπror (RE) for test sets and the Fréchet Inception Distance (FID) as evaluation metries. The RE is calculated as the binary cross entropy for BMNIST and the mean squared error for the others between the original image and the reconstructed image. Our LAE and CLAE are compared with three baseline methods, VAE, ABP and DLGM.
BMNIST
MNIST
SVHN
CIFAR-10 10
CelebA
RE
RE
RE
FID
RE
FID
RE
FID
VAE
108.3
0.01653
0.01019
167.6
0.02587
249.0
0.02129
123.6
ABP
185.9
0.01884
0.02144
292.6
0.03078
304.6
0.02416
172.4
DLGM
106.9
0.01634
0.01025
136.3
0.02563
262.8
0.02124
108.2
LAE CLAE
108.4
0.01860
0.009903
134.1
0.02654
241.6
0.02303
111.1
40.54
0.00294
0.00352
91.11
0.01408
223.9
0.00919
86.4
Pang et al. 2020). but they do not use amortization for the sampling of the latent prior and posterior as in CLAES. On the other hand, (Han et al. 2020) proposes to learn VAEs and EBMs in latent space, but their energy function is defined for the joint distribution of the observation and the latent variable rather than the latent prior. For more direct approach, in which EBMs are directly defined
in observation space, (Du & Mordatch] 2019) uses spectral normalization (Miyato et al. 2018 for
the energy function smoothen its density, and stabilize its training. (Nijkamp al. 2019 shows short-run MCMC is effective for the training of EBMs.
Generative adversarial networks (GANs) (Goodfellow al. 2014) can be regarded as special case of CLAES by interpreting their discriminator and generator as energy function and sampler function, respectively (see Appendix E for further details).
IMAGE GENERATION
To demonstrate the applicability of our framework to the generative model training, we perform an experiment image generation tasks using binarized MNIST (BMNIST), MNIST, SVHN, CI- FAR10, and CelebA datasets. As baselines, we use VAES and the ABP (Han et al. 2016), which is an algorithm to train deep latent variable models using LD without amortization. We also provide the performance deep latent Gaussian models (DLGMs)) (Hoffman 2017). in which VAE-like en- coders are used initialize MCMC for posterior inference, an alternative approach of amortiza- tion. For quantitative evaluation, we report the reconstruction error (RE) as an alternative of marginal likelihood p [ 0). which cannot be calculated for LAEs and CLAEs. Because the RE cannot be a measure of sample quality, we provide the Fréchet Inception Distance (FID) (Heusel et al. 2017) for SVHN, CIFAR- 10 and CelebA. The results are summarized in Table 2 We also provide the per- formance on denoising by trained VAEs, LAFS and CLAFS in Table 5 in the appendix. CLAAS
consistently outperform the others, and LAES also provide competitive results to the baselines. In addition, LAEs' training is faster than ABP due to amortization as shown in Figure 6 ABP cannot update the inference for datapoints that not included in minibatch, whereas LAEs can through the update of their inference model (encoder). This amortization enables scalable inference for large scale datasets, and accelerates the training of generative models. Qualitatively, images generated by LAEs and CLAES are sharper than those of VAES and ABPs shown Figure 7 Other examples are summarized in the appendix.
350
ABP
-1033
LAE
2033
-253
-3000
Epochs
Figure 6: Learning curves of ABP and
LAE on SVHN, The error bars denote the standard deviations with three seeds.
8 ANOMALY DETECTION
In addition to image generation, the potential energy in Eq. supervised anomaly detection, because it can be a measure of the probability density. For CLAEs, the potential energy itself cannot be calculated, because it includes the logarithm of the normalizing
be useful for performing un-
8

---

Under review as a conference paper at ICLR 2021
MNIST
CelebA
VAE
5
7
0
7
6
6
ABP
6
9
5
3
8
DLGM
9
1
6
J
S
LAE
3
6
6
8
7
?
CLAE
9
9
7
2
3
6
Figure 7: Generated samples of MNIST and CelebA by trained five models. The images are gener- ated by the decoder fxiz (z), and the latent variable z is sampled from the sampler function fzju for CLAEs, and the Gaussian prior for the others.
Table 3: Evaluation of the area under the precision-recall curve (AUPRC) anomaly detection task for MNIST. Our LAEs and CLAEE are compared with autoencoders (AEs) and VAËs. The top row shows which digit class is dealt as normal examples.
2
6
8
AE 0.974
0.996
0.762
0.845
0.863
0.706
0.837
0.952
0.765
0.872
VAE 0.971
0.995
0.768
0.837
0.844
0.699
0.845
0.927
0.754
0.874
LAE CLAE
0.915
0.997
0.749
0.875
0.853
0.844
0.914
0.936
0.779
0.913
0.976
0.862
0.788
0.752
0.915
0.777
0.899
0.790
0.798
0.865
constant of their latent prior log Z (0). However, it can be ignored because it is constant with values of the observation æ and the latent z. Therefore, use the pseudo potential energy, U, measure as follows.
Ú (ax, 2) log (ar 2: - fa (2:
(18)
We test the efficacy of our LAEs and CLAEs for anomaly detection using MNIST. We assume that each digit class is normal and treat the remaining nine digits anomaly examples. We use AEs and VAEs as baselines, and provide the area under precision-recall curve (AUPRC) as the metric for comparing the models. We use the RE as measure of anomaly for AEs and the negative ELBO for VAEs. From Table 3 can be observed that our LAEs and CLAEs outperforms AEs and VAEs.
9 CONCLUSION
We proposed amortized Langevin dynamics (ALD), which is efficient MCMC method for latent variable models. The ALD amortizes the cost of datapoint-wist iteration by using inference models. By experiments, we demonstrated that the ALD can accurately approximate posteriors. Using ALD, we derived a novel scheme of deep generative models called Langevin autoencoders (LAEs). LAES are extended more general setting, where the latent prior is defined with an unnormalized energy function, and we refer to it as contrastive Langevin autoencoders (CLAEs). We demonstrated that our LAES and CLAES can generate sharp images, and they can be used for unsupervised anomaly de- tection. Furthermore, we investigated the relationship between our framework and existing models, and showed that traditional autoencoders (AEs) and generative adversarial networks (GANs) can be regarded as special cases of our LAEs and CLAEs. For future research on ALD, theories providing a solid proof of convergence, deriving eetropolis-Hastinee rejection step, and deriving algorithms based on more sophisticated Hamiltonian Monte Carlo approaches should be investigated.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using reconstruc- tion probability. Special Lecture on IE, 2(1), 2015.
Martin Arjovsky, Soumith Chintala, and Léon Bottou.
Wasserstein gan.
arXiv preprint
urXiv:1701.07877 2017.
Christopher M Bishop. Latent variable models. In Learning in graphical models, pp. 371-403 Springer, 1998.
Miguel A Carreira- Carreina and Geoffrey E Hinton. On contrastive divergence learning. In Aistats, volume 10, pp. 33-40. Citeseer, 2005.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-hased models. arXiv preprint arXiv: /903: 0...9.8.889, 2019.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, al. Neural scene representation and rendering. Science, 16046394)11044112100 2018.
Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao. Asli Celikyilmaz, and Lawrence Carin. Cyclical annealing schedule: A simple approach to mitigating vanishing. arXiv preprint 1003.1000 2019.
lan Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Tian Han, Yang Lu, Song-Chur Chun Zhu, and Ying Nian Wu. Alternating back tookkkrrppggtionnno for generator network. arXiv preprint 1606.08571 2016.
Tian Han, Erik Nijkamp, Linqi Zhou, Bo Pang, Song-Chur Chg--Chunnuu Zhu, and Ying Nian Wu. Joint training of variational auto-encoder and latent energy-based model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2020.
Harry H Harman. Modern factor analysis. University of Chicago press, 1976.
Hao He, Hao Wang, Guang- -ee Lee, and Yonglong Tian. Probgan: Towards probabilistic gan with theoretical guarantees. In ICLR (Poster), 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by a two time-scale update nule converge local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8) 2002.
Geoffrey E Hinton and Ruslan Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 31 313(5788):504-5077 2006.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. arXiv preprint arXiv:1902.00275 2019.
Matthew D Hoffman.
Learning deep latent gaussian models with markov chain monte carlo. In International conference on machine learning, pp. 1510-1519, 2017.
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. arXiv preprint arXiv: arXiv:1804.0nm 1804.00779, 2018.
Rie Johnson and Tong Zhang. Composite functional gradient learning of generative adversarial models. arXiv preprint #rXiv:/801003300 2018.
10

---

Under review as a conference paper at ICLR 2021
Diederik P Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-ennoding variational bayes.
arXiv preprint
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pp- 3581-3589, 2014.
Durk P Kingma, Tim Salimans, Rafal Jozefowiez, Xi Chen, llya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. In Advances in neural information processing systems, pp. 4743 1453-475,,20 2016.
Peter E Kloeden and Eckhard Platen. Numerical solution of stochastic differential equations, vol- ume 23. Springer Science & Business Media, 2013.
Ananya Kumar, SM Eslami, Danilo Rezende, Marta Garnelo, Fabio Viola, Edward Lockhart, and Murray Shanahan. Consistent generative query networks. arXiv preprint arXiv:1807.022 2018.
Chunyuan Li, Changyou Chen, David Carlson, and Lawrence Carin. Preconditioned stochastic gradient langevin dynamics for deep neural networks. arXiv preprint arXiv:1512.07666 2015.
Yingzhen Li, Richard E Turner, and Qiang Liu. Approximate inference with amortised meme. arXiv preprint (eX.iv:102.088300 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05933 2018.
Radford M Neal. Meme using hamiltonian dynamics. Handbook of Markov Chain Monte Carlo, pp. 113, 2011.
Andrew Ng et al. Sparse autoencoder.
Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non- persistent short-run meme toward energy-based model. In Advances in Neural Information Pro- cessing Systems, pp. 5232-5242, 2019.
Bo Pang, Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy-based prior model. arXiv preprint arXiv:2006.0822 08205, 2020.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint urXiv:1505.05771 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4000 2014.
Yunus Saatci and Andrew G Wilson. Bayesian gan. In Advances in neural information processing systems, pp. 3622-3631, 2017.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelenn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05511 2017.
Rui Shu, Hung H Bui, Shengjia Zhao, Mykel J Kochenderfer, and Stefano Ermon. Amortized inference regularization. Advances in Neural Information Processing Systems, pp. 4393-4402, 2018.
Michalis K Titsias and Francisco Ruiz. Unbiased implicit variational inference. In The 22nd Inter- national Conference on Artificial Intelligence and Statistics, pp. 167-176, 2019.
Benigno Uria, lain Murray, and Hugo Larochelle. Rnade: The real-valued neural autoregressive density-estimator. In Advances in Neural Information Processing Systems, pp. 2175-2183, 2013.
Rianne Van Den Berg, Leonard Hasenclever, Jakub Tomezak, and Max Welling. Sylvester nor- malizing flows for variational inference. In 34th Conference Uncertainry in Artificial In- telligence 2018, UAI 2018, pp. 393-402. Association For Uncertainty in Artificial Intelligence (AUAI), 2018.
11

---

Under review as a conference paper at ICLR 2021
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681-688, 2011.
Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37-52, 1987.
Jianwen Xie, Ruiqi Gao, Zilong Zheng, Song-Chin Chun Zhu, and Ying Nian Wu. Learning dynamic generator model by alternating back-propagation through time. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 33, pp. 5498-5507, 2019.
Xianglei Xing, Ruiqi Gao, Tian Han, Song-Chun Chun Zhu, and Ying Nian Wu. Deformable gen- erator network: Unsupervised disentanglement of appearance and geometry. arXiv preprint arXiv:1806.06295 2018.
Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, and Min Zhang. Variational neural machine 2016.
translation. arXiv preprint
Jing Zhang, Jianwen Xie, and Nick Barnes. Learning noise-aware encoder-decoder from noisy labels by alternating back-propagation for saliency detection. arXiv preprint 1iix:20077.12211, 2020.
Yizhe Zhu, Jianwen Xie, Bingchen Liu, and Ahmed Elgammal. Learning Feature-to-featur transla- tor by alternating back pack-propagation for generative zero-shot learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9844-9854, 2019.
A NOTATION
Numbers and Arrays
A scalar (integer or real)
a
vector
I
Identity matrix with dimensionality implied by context
diag(a)
A square, diagonal matrix with diagonal entries given by a
a
scalar random variable
a
A vector-valued random variable
Sets and Graphs
A
A set
R
The set of real numbers The set containing and
{0, 1}
{0, [0. n}
The set of all integers between 0 and n The real interval including and b The real interval excluding but including b
la, b]
(a, b]
Indexing
as or a[1]
Flement of vector a, with indexing starting at Element of the random vector a
or ali]
Calculus
12

---

Under review as a conference paper at ICLR 2021
dy da
Derivative of y with respect to æ
Oy
Partial derivative of y with respect
dz
Vay
Gradient of y with respect to
f fx))dx
Definite integral over the entire domain of
Probability and Information Theory
P(a)
probability distribution over a discrete variable
p(a)
probability distribution over a continuous variable, or over variable whose type has not been specified
a~
Random variable distribution
Exxp[[<x) or (x)())
Expectation of [(r with respect to P(x)
H(x) or H(P)
Shannon entropy of the random variable x that has distri- bution P
Dkl(PIIQ)
Kullback-Leiller divergence of P and Q
N(
Gaussian distribution over with mean μ and covariance
Σ
u(
Uniform distribution over with lower range and upper
range b
Gam(1;a,
Gamma distribution over with shape and rate range ,
Functions
A BB
The function with domain and range B
f(x;0) 0)
function of æ parametrized by 0. (Sometimes we write (a) and omit the argument lighten notation)
logæ
Natural logarithm
(x))
Logistic sigmoid,
+ exp( (xx)
r(=)
Gamma function,
8- f--1e-'ds
[x]
Integer part of x, i.c.,
[x] = max(n E N n <xx x}
condition
is 1 if the condition is true, otherwise
B MARGINALIZING OUT OBSERVATION VARIANCE
When use diagonal Gaussian distribution for the likelihood function (i.e., p p z, 0) N' (z; 2x = fx|x ( I)). we have to decide the parameter of observation variance o A simple and popular way is to manually choose the parameter in advance (e.g., of = 1). However, difficult to choose proper value, and desirable that the variance calibrated for each datapoint. To address it, we use alternative approach, in which the variance is marginalized out and the likelihood can be calculated only with the mean parameter. First, we define the precision pa- rameter, which is the reciprocal of variance, i.e., Xx 1/02. The precision is defined per datapoint, and shared across all dimension of the observation. When we define the prior distribution the pre- cision using uninformative flat prior (e.g., p (xx) Gam (Ax; 0)). the marginal distribution
13

---

Under review as a conference paper at ICLR 2021
Algorithm& 3 Langevin Autoencoders
0, 0.0+ < Initialize parameters
repeat
t 0' N'00~N 1o (x, Z fax (X: -)0)) 2n
t Φ' ~ N (d': no a (x, Z - (x: Φ) 0) 27
until convergence of parameters return 0, Φ
Algorithm 4 Contrastive Langevin Autoencoders
0, Φ.00 vb < Initialize parameters repeat
0' ~ ( 0 no Vo (0, Φ wb) 2n I)
Φ - N Φ no of (0, Φ. 25) 2m
+ 3 N (nb;; + no w C (0, Φ 1p) 20
until convergence of parameters return 0 Φ.
will be simply the integral of the Gaussian likelihood over Axi
f
N (2: fxl1 (z:0), I) ddx
(19)
Ax
Ax(x; 1x 1xx(t))
exp
ddx
(20)
2T
2
/
4/2
Ax Σ1
(2)
Mxx [x]l
exp
d)x
(21)
2T
4+2 3
d d
2x 2 4/2
Σ (2t - mxld))
r
(22)
2
where d is the dimensionality of æ and I' is gamma function. This marginalized distribution is an improper distribution, whose integral over æ diverges and does not correspond to 1. We refer to the distribution the marginalized Gaussian distribution. Marginalized Gaussians can be widely used for mean parameter estimation of a Gaussian distribution, especially when its variance is unknown.
C LANGEVIN DEQUANTIZATION
Many image datasets, such as MNIST and CIFAR10, are recordings continuous signals quantized into discrete representations. For example, standard 8-bit RGB images of the resolution of H x W represented {0, 1,..., 255 ) 3x H HXW If we naively train continuous density model such discrete data, the model will converge to degenerate solution that places all probability mass on discrete datapoints (Uria et al., 2013). A common solution to this problem is first convert the discrete data distribution into a continuous distribution via process called dequantization, and then model the resulting continuous distribution using the continuous density model. Here, we denote the random variable of original discretized data as x E Nd (d = 3 x H X W), dequantized continuous variable as X E Rdd and the continuous density model as p (x). The simplest way of dequantization is to add uniform noise to the discrete data, and deal with it as a continuous distribution, which we refer to as uniform dequantization:
U [ x, x + 1):
(23)
However, this approach introduces flat step-wise regions into the data distribution, and it is unnatural and difficult to fit parametric continuous distributions. Moreover, the range of the dequantized data
14

---

Under review as a conference paper at ICLR 2021
is still bounded (i.e., x (0, 256)"), therefore it is not desirable to fit the continuous density model defined for unbounded range, e.g., Gaussian distributions.
To investigate more sophisticated approach, first consider the quantization process, the inverse process of dequantizatinn, in which the continuous data IR is discretized into {0, 1,...,
.....,.. 255)".
This process is represented as a conditional distribution P (æ) æ). For example, it is defined as follows:
P [x: ②) = = --=11==1200 (256-
(24)
In this definition, the continuous data is first compressed into (0, 256) using the logistic sigmoid, then discretized into {0, ......... 255J with its integer part. Although the quantization process could be formulated with another definition, we here discuss based on this formulation. When we have density model of continuous data p (8), the dequantization process can be formulated
posterior inference problem of [&| x) p (X) P (x X). Although this posterior is typically intractable, we can obtain samples from using our ALD algorithm in the same way as the posterior sampling of latent variable models. When we construct the inference model fx|x {0, 1..., 255] - IR as
follows, the likelihood will be constant, i.e., P (x a fx/ (x: E)) = 1.
x + o (=:
fx/x (z; {) = o
(25)
256
where 9 (x;; E) is a mapping from discretized data into real d-space. Therefore, the potential energy corresponds to the negative log likelihood.
U (a. a = fx (x: 2)) = log (a = fx|x (x;; E)) - log (x fxix (x; E)).
(26)
The parameter of inference model fxix is updated using our ALD algorithm as in Eq. (6)
g - g' ~ N E':E
VeU
fxx (zl)) 2nc
(27)
where ng is step size.
When we use this Langevin dequantization for latent variable models like LAEs, the potential energy is rewritten as follows, and all parameters {0, Φ, } are trained in an end-to-end fashion:
U x, = fxlx (X Z = =f// (x:) ))
Σ logp (2()) fa|x (z)) :))
(ogp(()
(28)
logy (((0) fxx (ml);; 1=20== faxx (2)(0
Figure shows the comparison of SVHN data distributions of the top left corner pixel before and after dequantization. Before dequantization, the data are concentrated on discrete points. After dequantization, it can be observed that the distribution gets continuous enough to fit continuous density models. Furthermore, Figure 9 shows the comparison of generated MNIST samples by LAFS to see the effect of Langevin dequantization. The dequantization seems to affect the sharpness of generated samples.
D DERIVATION OF EQ. 113
Vo log Z (z)
Z (8) (o)
(29)
Vo exp (-fa 0)) dz
(30)
Z (()
exp
(- f& (z; 0))
VoJz (z; 0) dz 7 7
(31)
=- - p(( 0) Vola (2; 0) dz
(32)
Exxp((20) VVola (2:
(33)
15

---

Under review as a conference paper at ICLR 2021
500
400
400
300
300
200
100
100
Before dequantization
After dequantization
Figure 8: Histograms of data before and after Langevin dequantization. Dequantized data is passed through a sigmoid function to keep the support range same.
5
2
9
4
9
w/o dequantization
w/ dequantization
Figure 9: Generated samples of LAEs with or without dequantization.
E
ADDITIONAL RELATED WORK
Generative Adversarial Networks (GANs) (Goodfellow al.] 2014) are similar CLAEs in that both are trained by minimax game between two functions (i.e., the energy function and the sampler function in CLAEs; the discriminator and the generator in GANs). However, there are some differences between them. First, the minimax game is performed in the latent space in CLAEs, while it is performed in the observation space GANs. In other words, the latent variable is identical to the observation (i.e., p (z: =) I==z) in GAN Second, the loss function is slightly different. In GANs, the loss function as follows:
Σ log D (0:0) + logg (1-D (u (1))()(()))) 0))
CGAN (0, 2/)
(34)
where G denotes the generator that maps its inputs u into the observation space, and D denotes the discriminator that maps from the observation space into (0, 1), and (") ~ N (w; 0, I). The discrim- inator is trained to minimize this loss function, whereas the generator is trained maximize it. The main difference Eq. (17) is the second term. When substitute it with log D (G (u(*); vb) 0), becomes more similar to Eq. (17). This modification is known as the - llog trick, and often used
to stabilize the training of GAN's generator (Goodfellow al. 2014 Johnson & Zhang 2018 In this formulation, the counter parts of the energy function and the sampler function are log D (- and G (-; v respectively. However, there is still a difference that the range of log (:; 0) is bounded within (0, co), whereas the range of CLAE's s energy function is unbounded.
Another difference between CLAES and GANs that the input vector of the sampler function is fixed through the training in CLAES, whereas the input of the generator changes per iteration by sampling from A A (u; 0, I] in GANs. Furthermore, CLAES are trained using noise injected gradient, whereas GANs are trained with standard stochastic optimization method like SGD. In the training of CLAES, as the number of inputs of the sampler function increases, the noise magnitude Eq. (15) will relatively decreases. Therefore, in the infinite case (i.e., A oo), Eq. (15) corresponds to standard (stochastic) gradient descent. Thus, GANs can be interpreted as the infinite case of CLAEs with regard to their training of generators. In the infinite case, the generator (sampler) may converge to a solution where it always generates maximum density points rather than samples from the distribution defined by the energy function. This nature cause mode collapsing, which is known as a major challenge of GAN's training. In GANs, the discriminator is also trained with
Note that the latent variable z is different from the input of GAN's generators. Here, the input of the GAN's generators is denoted as for the analogy with CLAFs.
16

---

Under review as a conference paper at ICLR 2021
standard stochastic optimization, which corresponds the MLE case where noise injection in Eq. (9) is omitted. Although there are some investigations to apply Bayesian approach to GANs (Saatii & Wilson 2017 He et al. 2019), their discriminators are not defined energy functions.
In summary, GANs with - log D trick can be considered as a special case CLAEs, where the latent variable is identical to the observation (i.e., p (xx 2) ====); ; the energy function and the sampler function are respectively defined as - log D (-) and G (:: +)); the number of inputs of the sampler function tends to infinity; and the model parameter is point-estimated with MLE.
Wasserstein GANs (WGANs) (Arjovsky et al. 2017) also has a loss function similar to CLAE's:
LWGAN (0, = Σ x 10
() D (
(35)
where D denotes the discriminator of WGANs that maps from the observation space into the real space R. In this case, the counter part of the energy function is -D (a:; f), although D has constraint of 1-Lipschitz continuity, which the energy function of CLAE does not has.
F EXPERIMENTAL SETTINGS
Unless we explicitly state otherwise, we use tanh activation instead ReLU for all experiments, because it is desirable that the whole model is differentiable at all points when performing sampling algorithms based on Langevin dynamics, which require the differentiability of the potential energy. In fact, found that our ALD experimentally performs better when using tanh rather than ReLU.
F.1 CONJUGATE BIVARIATE GAUSSIAN EXAMPLE
In the experiment of conjugate bivariate Gaussian example in Section 3 we initially generate three synthetic data (1), (2) (3)), where each (+)) is sampled from a bivariate Gaussian distribution as follows:
p (z) = N (z: pzz Ez),
:
( 2) = N (x; Ex).
0
1 0
0.7
0.6
In this experiment, Pr
Σn
and Ex
We can
0
0 1
0.7 0.8
calculate the exact posterior follows:
(z x))
N (zi Mz| Ex(x)
with Uxix = Σ,ix [Σ -.'z) Σxix = +
In this experiment, obtain 10,000 samples using ALD. We four fully-connected layers of 128 units with tanh activation for the inference model and set the step size Vis to 0.003.
F.2 CONJUGATE UNIVARIATE. GAUSSIAN EXAMPLE
In the experiment of conjugate univariate Gaussian example Section 3 we initially generate 100
synthetic data (1) (2) (100), where each (n) is sampled from a univariate Gaussian distri-
bution as follows:
P (22) N (z: 02) ( 2) NN (z;2,o))
In this experiment, set M2 0, 1, 0.01. In this case, can caleulate the exact posterior as follows:
p x==1)==\ =N N
In this experiment, obtain 20,000 samples using SGALD. We use four fully-connected layers of 128 units with tanh activation for the inference model and set the step size ns to 0.001. We set the batch size 10.
17

---

Under review as a conference paper at ICLR 2021
AVI (diag.
AVI (full)
ALD (hist.)
ALD (sample)
GT
Figure 10: Neural likelihood experiments.
F.3 NEURAL LIKELIHOOD EXAMPLE
We perform an experiment with a complex posterior, wherein the likelihood is defined with ran- domly initialized neural network fo- Particularly, we parameterize fo by four fully-connected layers of 128 units with ReLU activation and two dimensional outputs like p (x 2) = N (fe (z) I... We initialize the weight and bias parameters with A (0, 0.2I) and N (0, 0.1I), respectively. In ad- dition, we set the observation variance d₂ to 0.25. We used the same neural network architecture for inference model f+- Other settings are same as the previous conjugate Gaussian experiment.
The results are shown in Figure TO The left three columns show the density visualizations of the ground truth or approximation posteriors of AVI methods; the right two columns show the visual- izations of 2D histograms and samples obtained using ALD. For AVI method, we use two different models. One uses diagonal Gaussians, i.c., A (μ (æ; Φ) diag (o² (æ; Φ))). for the variational dis- tribution, and the oher uses Gaussians with full covariance N (μ (x; Φ), (æx; Φ)). From the den- sity visualization of GT, the true posterior is multimodal and skewed; this leads to the failure of the ALD
Gaussian AVI methods notwithstanding considering covariance. In contrast, the samples accurately capture such complex distribution, because ALD does not need to assume any tractable distributions for approximating the true posteriors. The samples ALD capture well the multimodal and skewed posterior, while Gaussian AVI methods fail it even when considering covariance.
F.4 IMAGE GENERATION
In the experiment of image generation, we resize the original image into 32 x 32 for all datasets. For MNIST, we pad original image with zeros to make the size 32 32. We use a diagonal Gaus- sian A (Hz,, diag (o?2) as a latent prior for VAEs, ABP, DLGM and LAEs, and treat the param- eter Plg o as learnable parameters. We use a diagonal Gaussian for the approximate posterior of VAEs. The architecture of neural networks summarized in Table 4 Conv x s x p x de- notes a convolutional layer with & x k kernel, x stride, p padding, and output channels. TonTToeeosee x x X denotes a transposed convolutional layer with x kernel, s x s stride, p x p padding, and output channels. Upsample denotes nearest neighbor upsampling with scale factor of 2. Lineard is a fully connected layer of output dimension d. We apply tanh activation after each convolution, transposed convolution linear layer except the last one. dx. d2 and d, the
18

---

Under review as a conference paper at ICLR 2021
Table 4: Neural network architectures
Encoder
Decoder
x E 1
E R
Conv3x1x0x64 Conx3x1x0x64
+ Conv Ton/Transposee4xxx0x2266
Conv3x1x0x64 1onv4x2x0x128
+ Upsample Conv3x1x2x128 Conv3x1x2x128
10
Conv1x2x0:256
Upsample Conv3x1x2x61 Comv3x1x22661
xxnx/x1xxxdout
Conv3x1x2x61 + Conv3x1x2xd2
Energy
Sampler
E RR
Rdu
+ Linear4096 + Linear4096
+ Linear4096 + Linear4096
+ Linearl
- Lineard,
Table 5: Comparison of denoising performance by VAEs, LAEs and CLAEs. The models eval- uated by the mean squared error between original (unnoisy) data and data reconstructed by the models from noisy data. Noisy data created by adding Gaussian noise sampled from / (0, 0.12) to original data.
MNIST
SVHN
CIFAR-10
CelebA
VAE
0.01714
0.01041
0.02587
0.02131
LAE
0.02243
0.01010
0.02703
0.02435
CLAE
0.007778
0.003676
0.01440
0.009690
dimensionality of x, z and respectively. dout is equal to d2 for LAEs and CLAEs, and 2dg for VAEs. For all datasets, we set the minibatch size m and the step size (me, 7p. Tw) to 1000, 0.01/n respectively, where n is size of training set. We set dz = 32, d, throughout the experiments. We use the same setting for the experiment of anomaly detection.
19

---

Under review as a conference paper at ICLR 2021
4
3
0
J
3
/
6
O
5
3
2
5
9
7
0
9
7
7
6
2
3
9
9
3
7
3
3
3
3
5
9
2
6
7
6
4
6
9
3
5
9
9
7
2
7
3
<
3
V
8
/
5
9
9
9
5
3
3
3
4
7
7
6
0
2
9
3
3
2
9
8
a
d
J
4
3
<
5
5
3
9
8
5
7
2
0
3
9
3
3
7
O
3
7
2
5
8
8
3
6
9
3
3
3
2
7
2
3
3
6
5
3
9
3
VAE
ABP
5
3
2
3
3
9
5
5
L
6
6
P
2
O
3
6
9
6
J
<
2
3
3
5
0
8
2
2
8
K
<
8
7
4
9
6
9
'/3
7
2
4
6
4
3
/
2
<
2
>
3
J
2
3
2
9
3
5
2
6
6
/
2
5
6
4
3
9
3
6
6
8
7
S
3
7
7
3
3
9
Y
2
J
4
3
2
7
9
3
4
6
5
4
)
7
3
6
5
2
Y
5
C
6
5
C.
<
O
8
2
5
3
4
5
7
DLGM
LAE
1
7
0
7
9
S
2
6
5
7
5
7
3
3
6
6
9
9
5
O
7
2
0
8
1
2
6
7
2
2
5
7
7
O
3
5
7
5
6
/
2
4
0
9
2
4
a
3
7
2
2
4
7
8
6
4
L
7
3
2
CLAE
Figure 11: MNIST samples
20

---

Under review as a conference paper at ICLR 2021
VAE
ABP
DLGM
LAE
3
6
6
6
2
8
6
5
2
7
8
5 2
6
5
6
3
6
7
6
5
6
6 8
CLAE Figure 12: SVHN samples
21

---

Under review as a conference paper at ICLR 2021
VAE
ABP
DLGM
LAE
CLAE
Figure 13: CIFAR-10 samples
22

---

Under review as a conference paper at ICLR 2021
VAE
ABP
DLGM
LAE
CLAE
Figure 14: CelebA samples
23


Reviews:
[{"review_id": "VHJ82EqatF3", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summarize what the paper claims to contribute\nThe papers introduce an amortisation for inference by Langevin dynamics (LD). Rather than making each particle track the posterior for a given data point as in normal LD, this new method couples the posterior samples of multiple data points by a dynamic recognition model; the parameters of the model are updated following the Langevin dynamics for a collection of data points. Authors also extend its use to two related models of data: variational autoencoder (LAE) with normalised and unnormalised prior p(z) (CLAE), producing increasingly better fits to data distributions.\n\n# Strong points:\n\n+ The idea to amortise inference for a collection of posterior distributions is not new, but I have not seen before its extension to posteriors induced by sampling dynamics, so this paper makes an interesting proposal. In particular, introducing dynamics at the recognition parameter level is very interesting. \n\n+ The authors did not stop at applying to the common Gaussian generative models as in VAE, but extended it to more complex energy-based prior distributions.\n\n+ The *method itself* is introduced clearly with well-written descriptions. The toy experimental results are helpful and demonstrate the power of the amortised Langevin.\n\n# Weak points:\n\n- My main concern is the experiment which is somewhat lacking on both the design and the quality of the results (detailed below)\n\n- The example for dequantization inference is confusing, and the problem might be technically ill-defined.\n\n- The description of CLAE is ok, but the authors try to link it to adversarial training which is a very different training objective. I also do not find the discussions clear enough to make it the contrast worthwhile\n\n# Recommendation: \nI am slightly tending to reject as it stands because of the experiments and some unclear discussions, but I am happy to increase the score if the authors provide a better argument for their design and clarify on their discussions.\n\n# Comments on experiments.\n- Evaluations based on reconstruction error is always prone to the trivial solution of learning an identity mapping. Could the authors try to perform denoising instead? \n\n- VAEs with continuous observations, especially when trained on MNIST, is known to be hard. How about comparing it to a binary/Bernoulli likelihood evaluated on binarised MNIST? If it outperforms VAE, then this provides much stronger validation. I do not see from the 10-13 that the samples from LAE is much better than traditional VAE. \n\n- The comparison between CLAE and other models is unfair: CLAE has an energy-based prior which may be more flexible (is this the case? Can the authors clarify on the choice of prior energy function? Sorry if I have missed the description.)\n\n- There are only FID and for CIFAR-10 and CelebA, but not for MNIST or SVHN. Can the authors report this figure using features extracted by a relevant neural network, e.g. one trained on MNIST classification for FID on MNIST?\n\n- The learning curve in Figure 5 is the unnormalised likelihood: it's unclear how its stability or convergence implies about learning speed of the normalised likelihood objective. \n\n- Also, how does CLAE look on Figure 5? \n\n- A critical question is whether the leaned dynamics is indeed more efficient than normal Langevin. The authors claim this but did not show results on the comparison. The effective sample in Table 1 provides some clue, but a convergence plot would be much stronger to support this claim. \n\n# Questions:\n\n? I find that updating all the recognition model parameters for each new data point a bit counter-intuitive. What if the initial samples for a set of x are very far away from the true distribution? The authors compare this method with the \"amortised\" MCMC in which the initial proposal is drawn from a recognition model, could this be combined with the dynamic-parameter recognition model to yield better results. Basically, if all parameters are dynamic, I do not see how the initial proposals can adapt to the generative model and characteristics of the data distribution. \n\n? Figure 4: are the samples from the conditional or unconditional Langevin? \n\n? What's the space of fixed random positions u? Also, could authors clarify the dimensionality of variables of z and u in Table 4?\n\n? I haven't been able to understand this claim:\n\"In VAEs, noise is used to sample from the variational distribution in the calculation of potential U, i.e., in forward calculation. However, in LAEs, noise is used for calculating gradient ∇φU, i.e., in backward calculation.\"\nThe reparametrised samples in VAE are indeed used for the backward calculation. The forward pass simply evaluates the objective and retains dependence on recognition parameters φ.\n\n? Figure 7: The After dequantization figure is better plotted after passing through a sigmoid? the huge difference in support range makes it hard to compare. \n\n? I am very puzzled with the content in appendix C and would like to authors to help with understanding. In particular, the sentence preceding (25) doesn't seem logical. I see that any \\hat{x} from (25) is mapped to the x, meaning that \"the likelihood is a constant\", which is OK. However, the first term on the RHS of (26) is, in fact, a (log) conditional distribution of \\hat{x} given x, but there is no distribution over the data x itself. Does this mean that the model is just a conditional distribution, and does not learn the data distribution p(x) at all? Also, what is the distribution for this first term? A delta or Gaussian?\n\n? Could the authors clarify the following sentence in appendix D:\n\"In other words, the latent variable is identical to the observation (i.e., p (x | z) = 1_{x=z}) in GANs\"\nI do not see the connection of this method to GAN, I'm happy to start again by better understanding this part. \nMore generally, I do not believe making the comparison is worthwhile if at all correct, because the different training objectives differ a lot. Also, the solution of GAN is obtained at an equilibrium established by the two players, but the authors do not show such results for this new method. Could something similar be established, i.e. the optimal recognition and data model is established at the minimax solution? \n\n# Detailed comments and suggestions (these points are here to help, and not necessarily part of your decision assessment)\n\nThe discussion on EBM in the appendix seems more relevant to the GAN discussion. If space permits, it should be moved in to the main text. Also, CLAE applies to a model with energy-based prior. How hard is it to extend to a fully latent variable EBM? \n\nSome typos:\n* Eqns (12) and (13), commas (,) before \\theta should be semicolons (;)\n* Second lines below (24) \"Altough\"-> Although\n* Line above (28) \"rewrited\"-> rewritten\n*  Third line below (28) \"enough continous\" -> continuous enough\n* Third line from bottom of page 15. \"caan\" -> can\n\n===== update =====\n\nI am very grateful for the patient and detailed response. Due to limited time, I wasn't able to quickly follow up on the discussion.  I think the current quality of the paper is improved, so I increase the score slightly. However, I still struggle to follow some of the statements even after reading the response, it could be my comprehension or something to do with style/writing.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "UYSw4Qe5u3i", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the author presented an advanced autoencoder framework LAE. Instead of element-wise MCMC, LAE collected samples from the posterior using the amortized Langevin dynamics of a potential energy distribution. In CLAE, an extended version of LAE, the author used an intractable energy function as the prior, and collected samples using its Langevin function. The author claims that LAE and CLAE are more efficient in large scale data and have better performance compared with traditional autoencoders and variational autoencoders.\n\n[Strengths]\n1. The proposed model replaces the posterior in VAE with a potential energy distribution, enabling fast sampling with its Langevin function.\n2. The model is flexible with intractable prior distributions with unnormalized energy function, which can enhance the approximation power of the model.\n3. The extended generative model CLAE with intractable prior indeed achieves has higher performance.\n\n[Critiques and weaknesses]\n1. Figures are poorly organized in the manuscript and captions in some of the figures are missing, making the figures difficult to understand without reading the context.\n2. Although the author claimed that LAE is able to collect samples efficiently from the posterior, the runtime comparison in large scale data was not shown in the result section.\n\n[Typos]\n1. In the caption of figure 1:  f_{x|z} should be f_{z|x}\n2. In the last paragraph of section 3, the first ‘figure 3’ should be ‘figure 2’\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TH7g5nIDXcY", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed a method for amortizing latent variable sampling that is applicable to a variety of problems. The demonstrated examples show that generative models using their proposed method do better than ones of existing benchmark models. The paper is relatively easy to follow.\n\nSome comments follow.\n\n1. The notation for the $z^{(i)}$ in the description of Algorithm 1 seems to be inconsistent (need to have bold $z^{(i)}$?). \n\n2. A key idea in this paper is to find parameters of a parametrized mapping between data $x$ and latent variables $z$ using Langevin-like steps. The updates to the parameters and to the latent variables are done alternately. After convergence of the parameters, you can map the observed data into latent variables using the learned mapping. \n\n3. At this point, I don't understand or it is unclear to me how this helps to draw posterior samples of the latent variables themselves. Perhaps after learning an inference model on one data set you can use same parameters $\\phi$ on another data set, initialize a MCMC in the latent space at the output of $f_{z|x}$, then you don't need to run the chain long as the learned function allows to achieve a \"warm start\" of the MCMC run?\n\n4. What if you don't use noisy Langevin training for determining $\\phi$ of the amortization model as you have, and just perform standard optimization on $\\phi$? What is it about Gaussian noise in training the amortization model that makes the learned parameters better?\n\nI think this is an interesting direction to explore, particularly if you can somehow make a connection with an exact method for sampling the latent variables $z$ (as you mention in the conclusion). It isn't clear how one would go about this. \n\nI would certainly be willing revise my review if the above points are satisfactorily clarified.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper had three borderline reviews. While the idea of posterior sampling of a neural network is potentially useful and Langevin dynamics are a way to attempt to address that, the reviewers did not appear convinced by the experiments and what the MCMC sampling was doing wasn't really front and center there.
Decision: Reject