Langevin Autoencoders for Learning Deep Latent Variable Models
Authors: Shohei Taniguchi
Yusuke Iwasawa
Wataru Kumagai
Yutaka Matsuo
Venue: ICLR 2022 Submitted (2022)
Keywords: Langevin dynamics
amortized inference
latent variable model
deep generative model

Abstract: Markov chain Monte Carlo (MCMC), such as Langevin dynamics, is valid for approximating intractable distributions. However, its usage is limited in the context of deep latent variable models since it is not scalable to data size owing to its datapoint-wise iterations and slow convergence. This paper proposes the amortized Langevin dynamics (ALD), wherein datapoint-wise MCMC iterations are entirely replaced with updates of an inference model that maps observations into latent variables. Since it no longer depends on datapoint-wise iterations, ALD enables scalable inference from large-scale datasets. Despite its efficiency, it retains the excellent property of MCMC; we prove that ALD has the target posterior as a stationary distribution with a mild assumption. Furthermore, ALD can be extended to sampling from an unconditional distribution such as an energy-based model, enabling more flexible generative modeling by applying it to the prior distribution of the latent variable. Based on ALD, we construct a new deep latent variable model named the Langevin autoencoder (LAE). LAE uses ALD for autoencoder-like posterior inference and sampling from the latent space EBM. Using toy datasets, we empirically validate that ALD can properly obtain samples from target distributions in both conditional and unconditional cases, and ALD converges significantly faster than traditional LD. We also evaluate LAE on the image generation task using three datasets (SVHN, CIFAR-10, and CelebA-HQ). Not only can LAE be trained faster than non-amortized MCMC methods, but LAE can also generate better samples in terms of the Fréchet Inception Distance (FID) compared to AVI-based methods, such as the variational autoencoder.

Under review as a conference paper at ICLR 2022
LANGEVIN AUTOENCODERS FOR LEARNING DEEP LATENT VARIABLE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Markov chain Monte Carlo (MCMC), such as Langevin dynamics, is valid for approximating intractable distributions. However, its usage is limited in the con- text of deep latent variable models since it is not scalable to data size owing to its datapoint-wise iterations and slow convergence. This paper proposes the amor- tized Langevin dynamics (ALD), wherein datapoint MCMC iterations are entirely replaced with updates of an inference model that maps observations into latent variables. Since it no longer depends on datapoint-wise iterations, ALD enables scalable inference from large-scale datasets. Despite its efficiency, it re- tains the excellent property of MCMC; we prove that ALD has the target posterior as stationary distribution under some assumptions. Furthermore, ALD can be extended to sampling from an unconditional distribution such model, enabling more ficxible generative modeling by applying to the prior distribution of the latent variable. Based ALD, construct tent variable model named the Langevin autoencoder (LAF). LAE uses AL. for autoencoder-like posterior inference and sampling from the latent space EBM. Using toy datasets, we empirically validate that ALD can properly obtain samples from target distributions in both conditional and unconditional cases, and ALD converges significantly faster than traditional LD. We also evaluate LAE on the image generation task using three datasets (SVHN, CIFAR-10, and CelebA-HQ). Not only can LAE be trained faster than amottizete MCMC methods, but LAE can also generate better samples terms of the Fréchet Inception Distance (FID) compared to AVI-based methods, such as the variational autoencoder
energy-based
deep la-
INTRODUCTION
Variational inference (VI) and Markov chain Monte Carlo (MCMC) are two practical tools to ap- proximate intractable distributions. Recently, VI has been dominantly used in deep latent variable models (DLVMs) to approximate the posterior distribution over the latent variable z given the ob- servation i.e., p (z x). At the core of the success of VI is the invention of amortized variational inference (AVI) Kingma et al., 2014 An & Cho 2015 Su et al. 2018; Eslami et al. 2018 Kumar et al. 2018). 3 which replaces optimization of datapoint wise variational parameters with inference model that predicts latent variables from observations. The advantage of AVI over traditional VI is that minibatch training can be used for its optimization, which enables efficient posterior inference on large-scale datasets. In addition, we can also leverage the optimized inference model to per- form inference for new data in test time. However, the approximation power of AVI (or VI itself) is limited because it relies on distributions with tractable densities for approximations. Although there have been attempts to improve their flexibility (e.g., normalizing flows (Rezende Mohamed, 2015 Kingma et al.) 2016 Van Den Berg et al.) 2018 [Huang et al. (018)). such methods typically have architectural constraints (e.g., invertibility in normalizing flows).
Compared to VI, MCMC can approximate complex distributions because it does not rely on any tractable distributions. Instead, MCMC repeats sampling from the target distribution and uses ob- tained samples to approximate the posterior distribution. Langevin dynamics is typical example of MCMC for sampling from a continuous distribution. However, despite its high approximation abil- ity, MCMC has received relatively little attention in learning DLVMs. It is because MCMC methods
The implementation is available htttttt: wytpittt.yy/Swwww000

---

Under review as a conference paper at ICLR 2022
piz |x1l))
plc |x(0))
piz |x(0))
plz (x(a
Inference Model fx
x
data points
Update
Posterior density
x
Datapoint-wise MCMC
Update fax
Implicit (simultaneous) MCMC over latents
(A)
(B-1) Traditional LD
(B-2) Proposed Method (ALD)
Figure 1: (A) Directed graphical model under consideration. (B-1) In traditional Langevin dynam- ics, the samples are directly updated in the latent space. (B-2) Our amortized Langevin dynamics replace the update of latent samples with an inference model fa|x that maps the observation æ into the latent variable
take long time to converge, making it difficult to be used in the training of DLVMs. When learning DLVMs with MCMC, we need to run MCMC iterations for sampling from each posterior per data point, i.e., p (z (x))) (i = I....,n). where n is the number of training data, as shown in Figure (B-1). It is problematic, mainly when training with large-scale (n > 10K) dataset, because it is time-consuming to run massive MCMC iterations for all data points. Furthermore, we need to re-run the sampling procedure when obtain new observations in test time.
As in VI, there have been some attempts to introduce the concept of amortized inference to MCMC. For example, [Hofmman| (017)) initializes MCMC sampling using inference model that predicts latent variables from observations. However, as they use inference models only for the initialization of MCMC, these methods still rely on datapoint-wise sampling iterations. Not only is time- consuming, but implementations of such partially amortized methods also tend to be complicated compared to the simplicity of AVI. To make MCMC more suitable for the inference of DLVMs, a more straightforward and sophisticated framework of amortization is needed.
This paper proposes the amortized Langevin dynamics (ALD), which replace datapoint-wist MCMC iterations with updates of an inference model that maps observations into latent variables (Figure (B-2)). Since latent variables depend on the inference model, the updates of the inference model can be regarded as implicit updates of latent variables, which enables us to perform posterior inference without datapoint-wis. MCMC iterations. Notably, our ALD treats outputs of the inference model themselves as samples from the target distribution, whereas existing amortization methods use the outputs only as initialization of MCMC. Therefore, ALD can be implemented straightforwardly like AVI. Moreover, despite the simplicity, can theoretically guarantee that ALD has the true posterior as stationary distribution under some assumptions, which is a critical requirement for valid MCMC algorithms.
Although we have introduced ALD as a posterior sampling algorithm, the application of ALD is not limited sampling from posterior distributions. Recent studies have demonstrated that applying an energy-based model (EBM) into the prior distribution over the latent variable enables more flexible generative modeling for DLVMs. When we train an EBM, EBM by running costly MCMC iterations. By extending our ALD to sampling from unconditional distributions, we can apply ALD into sampling from such EBMs. When sampling from an EBM with ALD,, we prepare function that maps fixed inputs into latent variables, and updates of EBM samples are replaced with updates of the function's 's parameters. In the same way with the posterior case, the updates of the sampler function can be regarded as implicit updates of samples from EBMs.
have to obtain samples from the
Using our ALD for sampling from both the posterior and the EBM over the latent variable, we derive novel framework of learning DLVMs, which we refer to as the Langevin autoencoder (LAE). In- terestingly, the learning algorithm of LAE naturally takes the combined form of an autoencoder like architecture and adversarial training. Our experiments show that ALD can properly obtain samples from target distributions using toy datasets. Subsequently, we perform numerical experiments of the image generation task using the SVHN CIFAR-10, 10, and CelebA-HQ datasets. Not only can LAF
2

---

Under review as a conference paper at ICLR 2022
be trained faster than non-amortized MCMC methods, but LAE also generate better samples in terms of the Fréchet Inception Distance compared to AVI-based methods, such as VAE.
PRELIMINARIES
2.1 PROBLEM DEFINITION
Consider probabilistic model with the dz-dimensional observation x, the dz-dimensional continu- ous latent variable z, and the model parameter è as descrihed by the probabilistic graphical model shown in Figure (A). Although the posterior distribution over the latent variable is proportional to the prior and the likelihood: (z x) = p p z) (x), this is intractable due to the normal- izing constant p (x) f (z) (x z) dz. This study aims to approximate the posterior p (z x) for n observations xr (2),, (m) efficiently by obtaining samples from it.
2.2 LANGEVIN DYNAMICS
Langevin dynamics (LD) Neal 2011 sampling algorithm based on the following Langevin equation:
dz = V (x, 2) dt + 2
(1)
where a Lipschitz continuous potential function that satisfies an appropriate growth condition, an inverse temperature parameter, and B is a Brownian motion. This stochastic differential equation has p (z x)) & exp (-BU (æ, z)) as its equilibrium distribution. We set / = and define the potential as follows to obtain the target posterior p (z x)) as its equilibrium:
U (ax, 2) = log (z) log p (x
=))
(2)
We can obtain samples from the posterior by simulating Eq. (Kloeden & Platen 2013) as follows:
using the Euler-Maruyame method
(2'22222-1 2),
(3)
where 7j a step size for discretization. When the step size sufficiently small, the samples asymp- totically move to the target posterior by repeating this sampling iteration. LD can be applied to any posterior inference problems for continuous latent variables, provided the potential energy is differentiable the latent space. However, to obtain the posterior samples for all observations ee!),.... (n), we should perform iterations of Eq. (3) per data point, as shown in Figure (B-1). 1). It inefficient, mainly if the dataset is large. addition, we need re-run the time-consuming itera- tions for new observations in test time. In the next section, we demonstrate method that addresses the inefficiency by amortization.
AMORTIZED LANGEVIN DYNAMICS
3.1
GENERAL IDEA
As alternative to the direct simulation of latent dynamics, we define inference model faix which maps the observation into the latent variable. Formally, the dynamics of its parameter $
dΦ
V (z()), faix
(z))); ffxx
dt V2dB.
Because the function fa/x outputs latent variables, the stochastic dynamics on the parameter space induce dynamics on the latent space. The main idea of our amortized Langevin dynamics (ALD) is to regard the transition on this induced dynamics as sampling procedure for the posterior distribu- tions, as shown in Figure (B-2).
We can use the Euler- Maruuama method to simulate Eq.
like traditional LD:
T N
Φ
V [x)))
(2()) 1n
(5)
3

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Amortized Langevin dynamics
+ Initialize parameters
g())
((n) f Ø
Initialize sample sets for all data points
repeat
① D
N D - VaU (x(0) x0)) falx (xl ②)) 2
z (faxx (zl)); $)} ....., Z {aaxx (x(*); })}
D Add samples
until convergence of parameters return _ (())
Through the iterations, the posterior sampling is implicitly performed by collecting outputs of the inference model for each data point as described in Algorithm Note that ( denotes set of samples of the posterior for the i-th data (i.e., p (z ()))) obtained using ALD. When we perform inference for new test data, the trained inference model can be used to initialize an MCMC method (e.g., traditional LD) because it is expected that the trained inference model can map data into the high-density area of the posteriors.
By this amortization, we replace the direct update of latent variables (z(1), ..... (n)) with the update of the global parameter 4. A significant advantage of amortization is that the cost of MCMC can be reduced by using minibatch training. For minibatch training, we substitute the minibatch statistics of m data points for the derivative for all data.
Σ VoU (2 J2 (n: 1b)
Y1
V S2ix (xx((())))
(20(1))
We refer the minibatch version of ALD as stochastic gradient amortized Langevin dynamics (SGALD). SGALD enables us to sample from posteriors of massive dataset efficiently. Moreover, in the context of stochastic gradient L.D (SGI.D), it is known that adaptive preconditioning effec- tively improves convergence compared to the naive SGLD (Li al. [ This preconditioning technique is also applicable to our SGALD, and we employ it throughout our experiments.
3.2 THEORETICAL ANALYSIS
To justify our ALD posterior sampling algorithm, we provide theoretical analysis of the stationary distribution of our ALD algorithm. Here, X and Z denote matrices with (0 and 2(0) in rows X and Zt.,, respectively. Our main result is as follows:
Theorem Let q (Z X) be a stationary distribution of the latent variables induced by Eq.
When the mapping fz1x meets the following conditions, q (Z X) satisfies (Z X) oc
exp (-U (x, Z)) = exp (-E (z))))))()))
1. The mapping has the form of fax (xx; 1) bg (z), where & is de x d matrix, g is mapping from Rd to Rdd, and d is the dimensionality of g (x)).
2. The rank of G is n, where G is a matrix with (x) in row G
See Appendix A for the proof. Theorem T suggests that samples obtained by ALD asymptotically converge to the true posterior when we construct the inference model fa|x with an appropriate form. Practically, we can implement such function using neural network whose parameters are fixed except for the last linear layer. In this implementation, the last linear layer takes role of the parameter +, and the preceding feature extractor takes role of the function g. our experiments, randomly initialize weights of the feature extractor and freeze them throughout the training.
In addition, the dimensionality of the last linear layer should be sufficiently large to meet the second condition. Therefore, the second condition derives trade-off between approximation quality and
"The relationship hetween the naive SGLD and the preconditioned version is almost identical to the naive stochastic gradient descent and RMSProp.
4

---

Under review as a conference paper at ICLR 2022
computational costs. It is worth noting that similar trade-off is also known in the context of AVI In AVI, the approximation quality is influenced by the capacity of the inference model, and the gap between the optimal variational distribution and the amortized distribution is often denoted as the amortization gap (Cremer et al.] 2018). In experiments, we confirm that preserving the condition does not become significant computational overhead in practice. In addition, we should note that decaying the step size 1) is needed to ensure convergence when we perform the simulation with discretization as described by /Welling& Teh (2011).
3.3 EXTENSION TO UNCONDITIONAL CASES
Currently, we have introduced ALD as sampling algorithm for conditional posterior distributions. We can also apply ALD to sampling from unconditional unnormalized distributions, namely energy- based models (EBMs). Consider an unconditional distribution over random variable z defined by an energy function fa as follows:
(6) where fe maps the variable into scalar value. To obtain samples from this EBM using ALD, we prepare a sampler function Szju (u: v) that maps its input u into the variable z. Here, the input vector is fixed, whereas observations are used in the posterior case. To run multiple MCMC chains in parallel, we prepare & fixed inputs u function as follows:
p (z) x exp (-fx (z))
(b) and update the parameter of the sampler
V Ja (I2 (eu((u))) (x)); v)) 2n
N
V
Typically, the fixed input vectors (2) u (k) are chosen from standard Gaussian distribution. As in the posterior case, we can guarantee the stationary distribution matches the EBM by choosing an appropriate form for the function fzju: and such a function can be implemented using a neural network whose parameters are fixed except for the last linear layer. For minibatch training, we can substitute the gradient for all k chains with the stochastic gradient of m minibatch chains:
m
Σ V (Jaiu (all(a)) (u))
V (Faju v)
(8)
1=1
The advantage of using amortization in the unconditional case is that we can run massive chains parallel using minibatch training.
LANGEVIN AUTOENCODERS
Using ALD for sampling from both the posterior and the energy-based prior, we derive novel framework for learning DLVMs. We here consider a latent variable model defined as follows:
(z + exp (-fe (z: Q))
(x 2, θ) N (x; fx|= (z; θ) diag (o)).
(9)
where is variance parameter of Gaussian distributior To learn the latent variable model, need to estimate both the model parameter O and the latent variable z. In Bayesian learning, the estimation is represented as the joint posterior distribution p (0, (!)) ..... z(n) .(),......
To obtain samples from the posterior, we can combine traditional L.D and ALD as follows:
O' N (O': Vo (X, S2|x
X: ②) θ) 2nT).
(10)
T N ( VU (x, S2|x (X; + + 2n
(11)
U (X,2,00) = logp ( log p
+ log p (i)
20) +
(12)
क) in its i-th row. If omit the Gaussian noise injection in Eq. (10). it corresponds to gradient ascent for maximum a posteriori (MAP) estimation
where fa|x (X; b)) is matrix with faix (x)
We also include the variance a into e and treat it as a learnable parameter. We also provide the learning algorithm of maximum likelihood in Appendix
5

---

Under review as a conference paper at ICLR 2022
Algorithm 2 Langevin Autoencoders
θ + v
Initialize parameters
repeat
+
O' ~
2n W V/L &, 2)
Update the generative model
4
D Update the inference model Update the sampler model
V V ( L (,, 2)
until convergence of parameters return O $ v
of O if additionally flat prior for (e). yields the maximum likelihood estimation (MLE). In this study, we assume a flat prior for p (+) and omit the notation for simplicity.
In Eq. (TO), cannot calculate the derivative of the potential function VeU in closed-form because the latent prior p ( ))) is defined using an unnormalized energy function. However, we obtain the unbiased estimator of the derivative by obtaining samples from the prior follows:
VeU (x, O)
Volz () Ve logg (0))
(0)
T Σ Vol (20),)
(13)
where z(1), _()).... (n) are sampled from the latent prior p (z
()) (see Appendix for the derivation). also ALD by preparing sampler function fuju that
To get samples from the latent prior,
takes fixed inputs u u 2u (k), as described in Section 3.3 Here, we set number of chains equal to the number of data points for simplicity, i.e.,
In summary, the encoder fz|x: the decoder /x/2- and the latent energy function fa are trained by min- imizing the following loss function L, whereas the latent sampler fzju is trained by maximizing it, while stochastic noise of the Brownian motion is injected in their update order to avoid shrinking to MAP estimates:
C (O, Φ V
(14)
Σ fa (Jaix (0) 10) - Sa (Ja (/au(u))) :) 1=1
llogp (z0(0) Se 10);;; ;
We refer to this framework of learning DLVMs as the Langevin autoencoder (LAE). We summarize the algorithm of the LAE in Algorithm [ LAE is closely related to the traditional autoencoder (AE) and other deep generative models, such as the variational autoencoder (VAE) and the generative adversarial network (GAN). We discuss the relationship the next section detail.
5 RELATED WORKS
Amortized inference is well-investigated in the context of variational inference. It is often re- ferred as amortized variational inference (AVT) (Rezende & Mohamed 2015 Shu et al. 2018). The basic idea of AVI is to replace the optimization of the datapoint-wise variational parameters with the optimization of shared parameters across all data points by introducing an inference model that predicts latent variables from observations. The AVI is commonly used in generative models (Kingma & Welling 2013), superisse learning (Kingma et al. 2),4), anomaly detection (An & Choo 2015), machine translation (Su et al. 2018)) and neural rendering (Eslami et al.) 2018 Ku- mar et al. 2018). However, in the MCMC literature, there are few works on such amortization.
Han et al. (201)) use traditional LD to obtain samples from posteriors to train deep latent variable models. Such Langevin-bars based algorithms for deep latent variable models are called alternating back propagaroonatiion (ABP) and applied in several fields (Xie et al. 2019 Zhang et al. 2020 Xing jet al., 2018 Zhu et al. 2019). However, ABP requires datapoint-we wise Langevin iterations, causing slow convergence. Moreover, when we perform inference for data in test time, ABP requires

---

Under review as a conference paper at ICLR 2022
MCMC iterations from randomly initialized samples again. Although Li et al. (2017) and Hoff- man (2017) propose amortization methods for MCMC, they only amortize the initialization cost in MCMC by using an inference model. Therefore, they do not entirely remove datapoint wise MCMC iterations.
be seen as particular case of LAEs, wherein the Gaussian noise injection to the update of the inference model (encoder) and the gener- ative model (decoder) is omitted in Eqs. 110g and (1I), and flat prior is used for p (z )) When different distribution is used as a latent prior, it is known as sparse autoencoders (SAEs) (gg et al., 2011]. In these cases, the dynamics in Eqs. (IO) and (IT) are dominated by gradient VU; hence, both the latent variables and the model parameter converge to MLE or MAP estimates (or other stationary points). Therefore, AEs (and SAEs) can be considered MLE (and MAP) algorithms for the parameter @ and the latent variables Z.
Autoencoders (AEs) (Hinton & Salakhutdinov, 2006)
Variational Autoencoders (VAEs) are based on AVI, wherein an inference model (encoder) is de- fined as a variational distribution (z x; .) using neural network. Its parameter 4 is optimized by maximizing the evidence lower bound. Interestingly, there a contrast between VAE and LAE when stochastic noise is used in posterior inference. In VAE, noise is used to sample from the stochastic inference model in calculating the potential U. i... in the forward calculation. However, in LAE, the inference model itself is deterministic, and stochastic noise is used for its parameter update along with the gradient calculation VaU, i.e., in the backward calculation. The advantage of LAE over VAE is that LAE flexibly approximate complex posteriors by obtaining samples, whereas VAE's approximation ability is limited by choice of variational distribution (z x; ②) because it requires a tractable density function. Although there are several considerations in the improvement of the approximation flexibility, these methods typically have architectural constraints (e.g., invertibility and ease of Jacobian calculation in normalizing flows (Rezende & Mohamed, 2015 Kingma et al. [016 Van Den Berg et al.. (2018) Huang et al. 2018)) Tiitsias & | (2019), or they incur more computational costs (e.g.. MCMC sampling for the reverse conditional distribution in unbiased implicit variational inference (Titsias& & Ruiz, 2019)).
Energy-based Models' training is challenging, and many researchers have been studying method- ology for its stable and practical training. A significant challenge is that it requires MCMC sampling from EBMs, which is challenging to perform in high dimensional space. Our LAE avoids this dif- ficulty by defining the energy function in latent space rather than data space. A similar approach is taken in several works (Pang et al. 2020a b). but they use traditional LD to obtain latent samples without amortization.
Generative adversarial networks (GANs) are closely related to our LAE because both are trained using adversarial loss functions. For detailed discussion, see Appendix D
6 EXPERIMENT
In our experiment, we first test our ALD algorithm on toy examples to investigate its behavior, then we show the results of its application to the training of deep generative models.
6.1 TOY EXAMPLES
We perform numerical simulation using toy examples to demonstrate that our ALD can properly ob- tain samples from target distributions in conditional and unconditional cases. First, we examples where the posterior density can be derived in a closed We initially generate three synthetic data 1 22, 33, where each x; is sampled from a bivariate Gaussian distribution as follows:
(2) = N (z; Mx, Ex),
p ( 2) = N (a;; 2, Σx)
In this case, calculate the exact posterior follows:
p= x) NN (z: (> + ) (Σ_² Ue + Σx x) (Σ,² + +xx') --))
] Σ = 1 0 and Σx =
0
0.7 0.6
In this experiment, we set μ2= ALD algorithm for this setting to obtain samples from the posterior. We neural network
0.7 0.8 We simulate
7

---

Under review as a conference paper at ICLR 2022
12
-05
-3
-10
-65 60
-15
-10
60
(A) Conditional Case
(B) Unconditional Case
Figure 2: Visualization of ground truth density (left) and samples by ALD (right) in the conditional
case (A) and the unconditional case (B) toy examples.
SCALD
Samou vall paluue un
Sample values
Dround Emoan
valuse 10000
Tuth Imear)
TLED
Tuth Ineen. atd)
2900 1000 1500 sebcetrsoe cicoowo0 Number MCMC steps
M0ns Raio 7005 wnwowoiionniieeeiioioooo Number of MLMC. steps
000 7500 te0001i50 1000 S0000000
Number of MCMC steps
Figure 3: Evolution of sample values across MCMC iterations for traditional LD and our SGALD in univariate Gaussian examples. The black lines denote the ground truth posteriors (the solid lines show the mean values, and the dashed lines show the standard deviation).
of three fully connected layers of 128 units with ReLUU activation for the inference model f2|xi setting the step size to x 10-4, and update the parameters for 3,000 steps. We omit the first 1,000 samples as burn-in steps and use the remaining 2,000 samples for qualitative evaluation. The result is shown in Figure 2 (A). ALD produces samples that match the shape of the target distributions well, even though ALD does not perform direct updates of samples the latent space. We also performed similar experiment in univariate setting see the convergence speed of our SGALD, the minibatch version of AL.D (see Appendix F.T for the detailed experimental setting). Figure shows the evolution of obtained sample values by traditional I.D) and our SGALD. It can be observed that SGALD's s samples converge much faster than traditional LD.
In addition to the simple conjugate Gaussian example, we experiment with a complex posterior, wherein the likeli- hood is defined with randomly initialized neural net- work. For comparison, we also implement the amortized variational inference (AVT) method, in which the pos- terior is approximated with a Gaussian distribution pa- rameterized by a neural network (see Appendix F22 for more experimental details). Figure 4 shows typical ex- ample, which characterizes the difference between AVI and ALD. The advantage of our ALD over AVI is the flexibility of posterior approximation. AVI methods typ- ically approximate posteriors using variational distribu- tions, which have tractable density functions. Hence, their approximation power is limited by the choice of variational distribution family, and they often fail to approximate such complex posteri- ors. On the other hand, ALD can capture such posteriors well. The results in other examples are summarized in Figure
GT
AVI
ALD
Figure 4: Visualizations of a ground truth posterior (left), an approximation by AVI (center), and samples by ALD (right) in the neural likelihood example.
i in the appendix.
Furthermore, we also test our ALD for sampling from an unconditional distribution. this experi- ment, mixture distribution of eight Gaussians and obtain samples using ALD, shown in Figure (B). We can observe that ALD adequately captures the actual density's multimodality and works well in the unconditional case.
8

---

Under review as a conference paper at ICLR 2022
Table 1: Quantitative results of the image generation for SVHN, CIFAR-IO, and CelebA-HQ. We report the mean and standard deviation of the Fréchet Inception Distance in three different seeds.
SVHN
CIFAR-10
CelebA-HQ
Description
32 32
32 x 32
32 32
64 x 64
VAE
VI amortization
47.19 1 0.96
106.0 0.5
102.6 + 1.4
174.9 + 0.9
VAE-flow
VI+ amortization flow
46.69 + 0.69
105.5
101.2 - 1.1
174.5 0.8
ABP
LD
46,90 1.07
0.2
99.27 2.42
135.7 2.0
DLGM
LD amertized init.
46.86 + 1.04
102.3 1.76
73.64
139.9 3
LEBM
LD FBM
38.79 + 2.48
97.02 0.36
32.59
0.30 0.30
53.31 + 2.26
LAE
LD EBM + amortization
46.66 1.33
95.85 + 1.06
40.33 + 1.33
61.38 + 1.20
6.2 IMAGE GENERATION
To demonstrate the applicability of our LAE to the generative model training, we experiment on image generation tasks using SVHN, CIFARIO. and CelebA-HQ datasets. Note that our goal here is not to provide the state-of-the-at results on image generation benchmarks but to verify the ef- fectiveness of our ALD this aim, we compare our LAE with five baseline methods, as shown in Table [ VAE (Kingma & Welling. 2013) is one of the most popular deep latent variable models in which the posterior distri- bution is approximated using the AVI. VAE-flow is an extension of VAE in which the flexibility of AVI is improved using normalizing flows. In addition AVI-based methods, based on Langevin dynamics (LD). The alternating back propagation (ABP) uses traditional LD to approximate the posterior, and the deep latent Gaussian model (DLGM) uses a VAE-like inference model to initialize L.D. The latent energy-based model (LEBM) uses an EBM for the latent prior, and the EBM and posterior sampling performed via traditional L.D. LEBM can be regarded as a non-amortization version of our LAE.
method of approximate inference in deep latent variable models. For
use three methods
We apply a commonly used convolutional neural network-based architecture for all models and a multi-layer perceptron for an energy-based model in the latent space of LAE and LEBM. Please refer to Appendix [3 for more detailed experimental settings. For quantitative evaluation of the sample quality, report the Fréchet Inception Distance (FID) (Heusel et al. 2017).
The results are summarized in Table I can be observed that LAE outperforms VI-based methods in terms of FID, although it does not reach LEBM's performance in SVHN and CelebA-HQ. In training speed, LAE takes 34.12 seconds per epoch on average to train with CIFAR- 10, while LEBM and DLGM take 84.29 and 60.66 seconds per epoch, respectively. This result shows that LAE is approximately 2.47 times faster than the non-amortized LEBM and 1.78 times faster than the partially amortized DLGM.
7 CONCLUSION
This paper proposed amortized Langevin dynamics (ALD), an efficient MCMC method for deep latent variable models. The ALD amortizes the cost of datapoint-wiss iterations by using inference models. We showed that our ALD algorithm could accurately approximate posteriors with both theoretical and empirical studies. Using ALD, we derived novel scheme of deep generative models called the Langevin autoencoder (LAF).. We demonstrated that our LAE performs better than VI- based methods in sample quality and can be trained faster than non-amortized LD methods.
This study will be the first step to further work on efficient MCMC for latent variable models with large-scale datasets. For instance, deriving a Metropolis- rejection step for ALD and algo- rithms based on Hamiltonian Monte Carlo methods an exciting direction of future work. More- over, developing more sophisticated way of choosing the feature extractor of the inference model is also important. In our experiments, we used a randomly initialized neural network that is fixed throughout the training, but there could be better way improve the performance of LAF.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Jinwon An and Sungzoon Cho. Variational autoencoder hased anomaly detection using reconstruc- tion probability. Special Lecture on IE, 2(1), 2015.
Martin Arjovsky, Soumith Chintala, and Léon Bottou.
Wasserstein gan.
arXiv preprint
arXiv:1701.07875 2017.
Chris Cremer, Xuechen Li, and David Duvenaud. Inference suboptimality in variational autoen- coders. In International Conference on Machine Learning, pp.
PMLR, 2018.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 360(6394):1204-1220
2018.
lan Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Tian Han, Yang Lu, Song-Chun Chun Zhu, and Ying Nian Wu. Alternating back propagattnn for generator network. In Proceedings of the AAAI Conference Artificial Intelligence, volume 31, 2017.
Hao He, Hao Wang, Guang-He Lee, and Yonglong Tian. Probgan: Towards probabilistic gan with theoretical guarantees. In International Conference on Learning Reprexentations, 2018.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(578)):504 500-50000557-5000 2006.
Matthew D Hoffman. Learning deep latent gaussian models with markov chain monte carlo. In International conference on machine learning, pp. 1510-1519, PMLR, 2017.
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In International Conference on Machine Leurning, pp. 2078-2087. PMLR, 2018.
Ric Johnson and Tong Zhang. Composite functional gradient learning of generative adversarial models. In International Conference Machine Learning, pp. 2371-321122 23799. PMLR, 2018.
Diederik P Kingma and Max Welling.
Auto-encoding variational bayes.
arXiv preprint
1rXiv:1312.611 14, 2013.
Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pp- 3581-3589, 2014.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling, Im- proved variational inference with inverse autoregressive flow. In Advances in neural information processing systems, pp. 4743-4751, 2016.
Peter E Kloeden and Eckhard Platen. Numerical solution of stochastic differential equations, vol- ume 23. Springer Science & Business Media, 2013.
Ananya Kumar, SM Eslami, Danilo Rezende, Marta Garnelo, Fabio Viola, Edward Lockhart, and Murray Shanahan. Consistent generative query networks. arXiv preprint arXiv: arXiv:1807.02033 2018.
Chunyuan Li, Changyou Chen, David Carlson, and Lawrence Carin. Preconditioned stochastic gradient langevin dynamics for deep neural networks. In Thirtieth AAAI Conference Artificial Intelligence, 2016.
Yingzhen Li, Richard E Turner, and Qiang Liu. Approximate inference with amortised meme. arXiv preprint arXiv: /00i:/702.088800 08343, 2017.
10

---

Under review as a conference paper at ICLR 2022
Radford M Neal. Meme using hamiltonian dynamies. Handbook of Markov Chain Monte Carlo, PP. 113, 2011.
Andrew Ng al. Sparse autoencoder. CS294A Lecture notes, 72(2011):1-19) 2011.
Bo Pang, Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy-based prior model. arXiv preprint urXiv:2006.08203 2020a.
Bo Pang, Erik Nijkamp, Jiali Cui, Tian Han, and Ying Nian Wu. Semi-supervised learning by latent space energy-based model of symbol-vector coupling. arXiv preprint arXiv:2010.09359, 2020b.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Swish: self-gated activation function. arXiv preprint arXiv:1710.0544 7:1, 2017.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Interna- tional conference on machine learning, pp. 1530-1538. PMLR, 2015.
Yunus Saatci and Andrew Wilson. Bayesian gans. In Advances in neural information processing systems, pp. 3624-3633, 2017.
Rui Shu, Hung Bui, Shengjia Zhao, Mykel Kochenderfer, and Stefano Ermon. Amortized infer- ence regularization. Proceedings of the 32nd International Conference on Neural Information 2018.
Processing Systems, pp.
Jinsong Su, Shan Wu, Deyi Xiong, Yaojie Lu, Xianpei Han, and Biao Zhang. Variational recurrent neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Michalis K Titsias and Francisco Ruiz. Unbiased implicit variational inference. In The 22nd Inter- national Conference Artificial Intelligence and Statistics, pp. 167-176. PMLR, 2019.
Rianne Van Den Berg, Leonard Hasenclever, Jakub M Tomczak, and Max Welling, Sylvester nor- malizing flows for variational inference. In 34th Conference on Uncertainty Artificial In- telligence 2018, UAI 2018, pp. 393-402 Association For Uncertainty in Artificial Intelligence (AUAI), 2018
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681-688. Citeseer, 2011.
Jianwen Xie, Ruiqi Gao, Zilong Zheng, Song-Chus Chun Zhu, and Ying Nian Wu. Learning dynamic generator model by alternating nack-propagation through time. In Proceedings of the AAAI Con- ference Artificial Intelligence, volume 33, pp. 5498-5595 5507, 2019.
Xianglei Xing, Ruiqi Gao, Tian Han, Song-Chun Zhu, and Ying Nian Wu. Deformable gen- erator network: Unsupervised disentanglement of appearance and geometry. arXiv preprint arXiv: 1606.0698. 2018.
Jing Zhang, Jianwen Xie, and Nick Barnes. Learning aoise-aware encoder-decoder from noisy labels by alternating pack-propagation for saliency detection, arXiv preprint S.Ki::0007.1221:1 2020.
Yizhe Zhu, Jianwen Xie, Bingchen Liu, and Ahmed Elgammal. Learning feature-to-feature trans- lator by alternating back- tokkproppgation for generative zero-shot learning. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 9844-9854, 2019.
A PROOF OF THEOREM
First, we prepare lemmas.
11

---

Under review as a conference paper at ICLR 2022
Lemma 2. Let h R + Bdaxa be linear map defined by h (Φ) &G. When the rank of G
n, there exists orthogonal linear map RR TRax such that T ($)
($(1) $12)
satisfies ker = span 0n.xn: (1(2) where = hor hor (
Φn
=
[ant1..... क0 and Φb E R for i = l....,,
A (dd-xxn) where
Proof. The singular value decomposition of G' is represented by TGTT A is n x n diagonal matrix A = diag (AI,..., An), and T and T are orthogonal matrices. Since the rank of G is n, At....y An are non-zero. When we set (T) = TT we obtain
0
=
TTG = (TGTT) T
$
A
T.
(15)
(ddnnxn)
From the above equation, ker h span
0 Od' xn: 112)
holds.
Lemma 3. For + 12 xd, let & satisfy:
T
T (1) 212) (T))
map h()) :
Rd.xn + Rd defined by h() ($(1)) h ((001),)
h
o)) satisfies 4G
7001) ((1) and is linear isomorphic.
Proof. From the definition, we have
GG -
G TTG
= ④ (TGT))
T
a (2)
A
T
0 Odd-
T (1)
A
T
0 (d-nxn)
([ [
--
G
= hor-1
((
[ 0)
(( ($00), ($0)))) .l)
= k(00 (Φ(1).
By the definition, 7,00) linear. Here, 7(0)) injective, since ker ī span 0x $12)
and
hence, dim Im h() Z dz x n. Since Im h0) gd 700) is surjective.
Lemma 4. For V RD $ + V ( = U fxp (X: RR (11,5(2)
512)
(T))
-or- and (()) (Φ(1) v ([00) (1) (ddx( Eq. 4 equivalent to
d$ (1) V & (1) (m(1) dt + 2dB
(16)
( V2dB.
(17)
12

---

Under review as a conference paper at ICLR 2022
Proof. By direct calculation, obtain
V
(( 8 (1)
$00) = V VoT ( $
U (x, falx (X:--1 (( ( (
1 h (T-1 ( ($(1) $ (
U [x, h orr ([ ([5(1) 00) + hho-- (0,
= (x,h (T-1 (( (
U (xlx fulx (x:--² ([ [
= Vor (1 ,))
v (15t0,0) .0)
(18)
Then, the following equivalence holds:
dΦ = - V V (o) dt + V2dB
+ dt (o) = -T T V ( dt + v2dB
+ dΦ = T T V V + dt + 2dr (B)
(19)
V V +
dt + v /dB,
where we used T TOT = id because T is orthogonal. From Eq. 18 the dynamics in Eq. 19 is equivalent to Eq. (16) and Eq. 17
In the following, we prove Theorem using the above lemmas. The latent variables Z T,00) (11) is independent of (2), and the probability distribution (Z X) of Z is given by
the pushforward measure (R(W) (p.2)
(Z) of the probability distribution p(1) of o by h1),, The
# amortized Langevin dynamics has q [Z G') as its stationary distribution of Z. Then, we have
( x) (R(1)
# (p.
(Z)
(1)) ((nc0)- (2))
det d(hha)) -1
dZ
((R0)) -1 (2))
det (h(
(1
(1)
(R())) (2))
det
- ((x20) ((x)))((2))
X
exp
7-1 (-' ([(x0) -1 ((2).0 00)
-v
exp
-U exp
X x ([((0))
-1 (Z) (2)) G))
exp (-U (x, Z)),
where used that 2b z. The last equation is derived as follows. From Lemma 3 fG = (((1) (Tb11) holds when
7,(1) because of the linearity of (1))) and is constant with respect to
13

---

Under review as a conference paper at ICLR 2022
Algorithm Amortized Langevin dynamics (test time)
+ faix (x; x*)
Initialize sample using trained inference model >> Initialize a sample set
++ Ø
repeat
ト - ~ (22:=- -=
(x, 2n))
> Update the sample using traditional LDD > Add samples
z
until convergence of parameters return
i (22) (Ø). Thus, when 4 = -- (( [ 0)).
obtain h()) (T(()
DG
r-1
([ (b())) 00)
G. In particular, for ((1) = (1()) (Z), we have
Z a2) (n -1 (2))
-- ((()0)) (2),0)
(2) G.
-1
B MAXIMUM LIKELIHOOD TRAINING FOR LAE
In Section we derive Bayesian learning algorithm of LAE, where the whole model is formulated as a Bayesian neural network. We can also think of the frequentist approach, where the training is defined as maximum likelihood. In this case, the objective is to maximize the evidence log p (æ ().
and its derivative is as follows.
Ve logp (x
(20)
Vo log
p (xx, Oz dz
(21)
/
Vep (z, 1 |
(22)
( ()
p p(r,
+
Ve logp (x, 2 ( dz
(23)
+
Ve log (x. z [)]
(24)
V log (x
z, ①) + Ve logp O)
(25)
Ve log (a
)) Vefz (z; 0)] + Ep Vef (z; O)]
(26)
We can approximate the derivative by obtaining samples from the posterior ( x. θ) and the latent prior (z )) using LAE as in Eq. 5 and
Hence, the maximum likelihood version of
LAE summarized shown in Algorithm
C DERIVATION OF EQ. [13
VaU (X, 2,
Vela (21110) Vo logp ( 2((1) 0) + Vo log (()),
14

---

Under review as a conference paper at ICLR 2022
Algorithm 4 Maximum likelihood version of Langevin Autoencoders
θ $ V
Initialize parameters
repeat
repeat
Φ イ- 4 ( - (, y L ~ N ( L - 2n
b Update the inference model
Update the sampler model
until convergence of o and y
θ + @ V/ $ 4)
Update the generative model
until convergence of θ return O 4 V
where C' (e)= = (xxx((=xx22; dz. By direct calculation, we obtain
Ve Vo C (0)
(eC
(27)
C (())
(- dz exp (- (z: 9)
(28)
(
Vefz (z: θ) dz
(29)
(())
p(z () Velz (z: 0) dz
(30)
= Ez-p(z(e) Volu (z; θ)
(31)
Vel2 (
(32)
where {(1) y... ((n) are samples drawn from (z ().
D ADDITIONAL RELATED WORK
Generative Adversarial Network (GAN) (Goodfellow et al., 2014) is similar to LAE in that both trained by minimax game between two functions (i.e., the energy function and the sampler func- tion in LAE; the discriminator and the generator in GAN). However, there are some differences be- tween them. First, the minimax game performed in the latent space in LAE, while is performed in the observation space in GAN. In other words, the latent variable is identical to the observation (i.e.. p (z 2²) 1==x) in GAN. Note that the latent variable is different from the input of GAN's generators. Here, the input of the GAN's generators is denoted as u for the analogy with LAE. Second, the loss function slightly different. In GAN, the loss function is as follows:
(1- LGAN 1) = Σ log fx (z))) θ) + log (1 - fx (frlu (2: V 2 O)):
(33)
where fx/u denotes the generator that maps its inputs into the observation space, and fx denotes the discriminator that maps from the observation space into (0, 1). and ( N (w; 0, I). The discriminator is trained to minimize this loss function, whereas the generator is trained to maximize it. The main difference to the loss function of LAE is the second term. When we substitute it with
-log fx (fx (x) y ()), it becomes more similar. This modification is often used to alleviate gradient vanishing and stabilize the training of GAN's generator (Goodfellow et al., (0144) Johnson & Zhang 2018). In this formulation, the counter parts of the energy function and the sampler function are log fx (:: and fx (:; )T), respectively.
Another difference between LAE and GAN is that the input vector of the sampler function is fixed through the training in LAE, whereas the input of the generator changes per iteration by sampling from N (w; 0, I) in GAN. Furthermore, LAE is trained using noise injected gradient, whereas GAN
15

---

Under review as a conference paper at ICLR 2022
00
Ground Truth
d > n
d d=
d < n
Figure 5: Analysis on amortization gap.
trained with a standard stochastic optimization method like SGD, In GAN, the discriminator also trained with standard stochastic optimization, which corresponds to the MLE case where noise injection is omitted. Although there are some investigations to apply Bayesian approach to GAN (Saatci& & Wilson 2017 He et al. 2018), their discriminators not defined as energy functions.
Wasserstein GANs (WGANs)
(Arjovsky et al. 2017) also have loss function similar to LAE's:
LWGAN (θ, V)
Jx x 10 6
fx (Jaju (v 1 0),
(34)
where denotes the discriminator WGANs that maps from the observation space into the real space R. In this case, the counter part of the energy function is -D (æ;; θ), although D has constraint of 1-Lipschitz continuity, which the energy function of LAE does not has.
E ADDITIONAL EXPERIMENT
We perform additional experiment investigate the amortization gap when the capacity of the inference model is not enough to meet the second condition of Theorem We use the same experi- mental setting with the bivariate Gaussian example in Section
6.1 and change the dimensionality of the last linear layer of the inference model from 2 (< d) to 128 (> d). The results are summarized in Figure S It can be observed that the sample quality is good when the dimensionality of the last linear layer is equal to or greater than the number of data points (i.e., d 2 n). When the dimension-
ality is smaller than the number of data points, the samples for some data points shrink to small area, while good samples are obtained for the remaining data points.
F EXPERIMENTAL SETTINGS
F.1 CONJUGATE UNIVARIATE GAUSSIAN EXAMPLE
In the experiment of conjugate univariate Gaussian example, we initially generate 100 synthetic data (1) x (100), where each (i) is sampled from univariate Gaussian distribution as follows:
P (2) N (zi Mz: oz) ( 2) (z;
In this experiment, we set 14 = 0, 0, 1, 0.01. In this case, we can calculate the exact posterior as follows:
( x)
= W
2;
T
In this experiment, obtain 20,000 samples using SGALD. We use four fully-connected layers of 128 units with tanh activation for the inference model and set the step size n to 0.001. We set the batch size 10.
16

---

Under review as a conference paper at ICLR 2022
AVI (diag.)
AVI (full)
ALD (hist.)
ALD (sample)
GT
Figure 6: Neural likelihood experiments.
Table 2: Neural network architectures
Encoder
Decoder
a E Rdx
E Rd
Conv3x1x2x64 Conv4x2x2x128
Deconv6x1xxxx112
Conv4x2x2x256 -> Conv4x2x2x512
+ DeConv4x2x0x256 - DeConv4x2x0x128
Conv6x1x0xm - Linear dz
DeConv4x2xxx61 Conv3x1x(xxl
Energy
Sampler
z E Rdd
Rdd
+ Linear2048
+ Linear2048 + Linear n
+ Linearl
Linear dz
F.2 NEURAL LIKELIHOOD EXAMPLE
We perform an experiment with a complex posterior, wherein the likelihood is defined with ran- domly initialized neural network fe. Particularly, we parameterize fo by four fully-connected layers of 128 units with ReLU activation and two dimensional outputs like p (x z) = N (fo (z), I) We initialize the weight and bias parameters with A (0, 0.27) and N (0, 0.17). respectively. In ad- dition, set the observation variance a₂ to 0.25, We used the same neural network architecture for the inference model f Other settings are same as the previous conjugate Gaussian experiment.
The results are shown in Figure 6 The left three columns show the density visualizations of the ground truth or approximation posteriors of AVI methods; the right two columns show the visual- izations of 2D histograms and samples obtained using ALD. For AVI method,
two different models. One uses diagonal Gaussians, i.c., A (μ (æ; Φ) diag (a² (æ; Φ))). for the variational dis- tribution, and the oher uses Gaussians with full covariance N (μ (x; Φ), (æ; Φ)). From the den- sity visualization of GT, the true posterior is multimodal and skewed; this leads to the failure of the Gaussian AVI methods notwithstanding considering covariance. In contrast, the samples accurately capture such complex distribution, because ALD does not need to assume any tractable distributions for approximating the true posteriors. The samples ALD capture well the multimodal and skewed posterior, while Gaussian AVI methods fail even when considering covariance.
ALD
17

---

Under review as a conference paper at ICLR 2022
2
3
3
3
53
6
5
6
1
6
22
9
4
S
ii SVHN
(ii) CIFAR-10
(iii) CelebA-HQ (32 32)
(iv) CelebA-HQ (64 64)
Figure 7: Generated images by LAE.
F.3 IMAGE GENERATION
Our implementation of baseline models is based on the official public code of Pang et al. (0020a) which is available at /ttp::://qithae
com com/bpuclal latent space EBM prior The architecture of neural networks for LAE is summarized in Table 2 Cony x x x c denotes convolutional layer with & x A kernel, x stride, p p padding, and output channels. Deconv & x x p x denotes a transposed convolutional layer with x h kernel, x stride, p p padding, and output channels. Linear d is fully connected layer of output dimension d. We apply the swish function (Ramachandran et al. 2017) as activation after each convolution, transposed convolution linear layer except the last one. dx. dz and dy are the dimensionality of x, z and u respectively. We fix the parameters in the encoder and the sampler except for the last linear layer meet the condition of Theorem [ For all datasets, we set the minibatch size m to 100. The latent dimensionality d2 is set to 64 for SVHN and 128 for CIFARIO and CelebA-HO. We set d,, = 2048 throughout the experiments. The training of LAE performed via the Langevin sampling iterations explained in Section 4 We use preconditioned stochastic gradient Langevin dynamics (pSGLD) (Li al. 2016) as the sampling algorithm. The implementation is available at https://bit. 1y/ http:///bb..112Sww005 2Swow0F'
18


Reviews:
[{"review_id": "vS63yFItIQl", "reviewer": "Reviewer_vRp6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed the overall paper. I felt the presentation was perhaps a little unecessarily complex. I have a few technical concerns\n\n# Good points\n- I felt the writing and presentation of existing literature was generally very good, clear and concise.\n\n- the theoretical concept \"one chain for all datapoints\" (and linear projections) also feels like a good idea to me. The theorem requires that all samples for all posteriors $z^{(i)}$ must be varying linear projections ($g(x)$) of a single communal chain $\\Phi$. Or alternatively, all data item specific posterior distributions are linear projections of a single higher dimensional distribution. This sounds really cool!\n\n- I enjoyed the theorem and the discussion of the rank of the G matrix.\n\n\n# Concerns\n\n- In algorithm 2, the generative model weights, $\\Theta$, are also sampled using the Euler-Maruyama method, however this does not correspond to the theorem 2.1 which requires that the generative model $\\Theta$ is fixed (otherwise there is no true posterior to approximate). (Upon further inspection, it looks like the given loss function (14) is the evidence lower bound of the data and hence sampling model weights is similar to Bayesian neural network sampling of weights with the ELBO and not model evidence).\n\n- P4, line 3: Test time sampling: why not pregenerate a markov chain during/after training that can be reused again and again at test time? Surely the whole point of amortization is the “one-chain fits-all” hypothesis? If the authors propose resampling at test time then the proposed method is not strictly amortized and there is no actual speedup?\n\n- In the experiments, would it be possible to also report estimated marginal dataset likelihood $\\log p(x)$? There are many generative modelling works that use this as a metric (e.g. PixelCNN).\n\n- I am surprised by the *extremely* high variance of of the Frechet Inception distance among baselines, I struggle to believe that all the standard methods are *that* bad, presumably either the error bars are not accurate or the implementation of baselines is not ideal. For me, unfortunately, this make the paper hard to accept. (I realise this may contradict other venues/reviewers that emphasise empirical results).\n\n- in the experiments, I think it is valuable to know why LD and LD+EMB are worse than LAE? Surely LD + EMB is the gold standard (one MCMC chain per dataitem) that the LAE strives to achieve, while this may be computationally expensive, I believe that such a baseline should at least be implemented to outperform (or match) LAE, (i.e. agreeing with the theory) this could also show the MCMC amortization gap that this new amortization method introduces (again agreeing with the theory). Currently these are unanswered question that feels rather central to the contribution of the paper.\n\n- regarding Theorem 2.1, and the implication that all posteriors are linear projections of a bigger common distribution, this feels very sensitive to the big distribution dimension, currently this is not properly investigated. If the dimension is large, then the communal MCMC chain is no cheaper than parallelized low dim chains across dataitems, if the dimension is too small, this imposes too much similarity across all the posteriors and the method suffers, this trade off is not clear.\n\n# Minor Points\n\n- there appears almost no mention of Bayesian neural networks. MCMC sampling of network weights has been extensively studied, the proposed method is also MCMC sampling network weights while just using a different  distribution to sample from. Presumably, any Bayesian neural network approach for the encoder may be used? \n\n- If I understand correctly, training is using the Eurler Maruyama method and so the algorithm simplifies to SGD with Gaussian noise, presumably this makes convergence much slower than a VAE or VAE-flow which may use RMSprop or Adam or any method with momentum across minibatches?\n\n- (very minor) regarding the disadvantages of VAEs, the ELBO training objective is the KL divergence of true and approximate posterior, i.e. it makes a inference network Gaussian that looks like the posterior but it also, *theoretically*, it makes a posterior that looks Gaussian,  hence the claims that VAEs are existentially flawed feel slightly overstated to me, though I realise this is not a common opinion and in practice this is indeed very often not the case.\n\n- the presentation of the proposed method seems a little overly complex or difficult to follow, as I have described, I personally found it much easier to think in terms of a communal chain and linear projections, I am sure there must be many other intepretations that are simpler and easier to follow than the paper in its current form.\n\n\nI enjoyed the paper and I am less well read in the area hence absolute novelty I cannot confidently judge (I wouldn't be surprised if this idea is well established in the more pure \"non-deep\" statistics community) though I do believe the idea has merit and is worth accepting. However the rather chaotic numerical results are very disturbing.\n", "summary_of_the_paper": "Typically, approximating the (analytically intractable) posterior of deep latent variable models can be done in two ways, variational inference methods learn an (analytically tractable) distribution to approximate the true posterior. For large datasets where each dataitem has its own posterior distribution, the parameters of the optimal approximate distribution may be predicted as a function of the dataitem, the comon practice of “amortization”. Thus the approximate posterior is quick to infer at test time however it is also limited by the flexibility of the familty of distributino used for the analytic approximation.\n\nMeanwhile, Markov Chain Monte-Carlo methods generate a collection of samples to approximate true posterior. As each dataitem in a large dataset has a unique posterior, each would require its own Markov chain of samples which is a more accurate approximation, however is time consuming.\n\nThe author propose a mixture of these approaches. At a very high level, the authors propose to combine a single “communal” Markov chain, $\\Phi$, together with a function that maps dataitems into a weight vector, $g(x)$, that linearly maps the communal markov chain onto a markov chain that approximates the posterior for the given dataitem p(z|x)~$\\Phi g(x)$. Replace one chain per item (traditional MCMC) with a communal chain and one linear mapping per dataitem, where the mapping is amortized.\n\nThe authors describe application of this new trick to latent space of an autoencoder latent variable model and the sampling from an unconditioned density such as an energy based model. The MCMC algorithm of choice is the well known Langevin Dynamics solved by the Euler-Maruyama method. The authors describe training in minibatches simulataneously regenerating the MCMC chains which reduces to standard SGD with noise. Toy experiments show proof of concept and image generation tasks show positive results as measured by Frechet Inception distance.\n", "summary_of_the_review": "I like the paper, I feel that the writing could be simplified somewhat, the method treats latent posteriors over data points as unique linear projections of a global distribution over network final layer weights.\n\n- to me, the proposed method is an instance Bayesian neural networks with a different distribution (the ELBO) over weights (eg https://arxiv.org/abs/1902.02476, http://proceedings.mlr.press/v48/gal16.pdf)\n\n- the experimental results have very high variance amongst baselines, this is concerning, particularly, the \"truth\" that the proposed method aims to approximate performs *worse* that the approximation.\n\n- treating all dataitem posteriors as a linear projections of a single communal distribution seems risky, further comment would be appreciated.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "sLE1tvD9dc6", "reviewer": "Reviewer_siJZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\n(1) The idea of the paper is novel and interesting. Theoretical analysis is provided. \n\n(2) The proposed method can be applied to sampling both the posterior distribution and the energy-based model. \n\n(3) The proposed method is different from variational approximation. \n\nWeaknesses: \n\n(1) One thing I am concerned with is, if all the randomness is accounted for by the Langevin sampling of the parameters, will the generated samples be correlated? \n\n(2) Can you always find a good sampler function to approximate complex target distribution? ", "summary_of_the_paper": "This paper proposes an amortized Langevin dynamics for sampling from a posterior distribution of a top-down generative model or from an energy-based model. The key is to recruit a sampler function that generates many samples directly in parallel, and run a single Langevin dynamics on the parameters of this sampler function. The method is illustrated by image generation. ", "summary_of_the_review": "The paper proposes a new idea on sampling from unnormalized densities. It can be useful for learning deep generative models. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Nqap9Km0Dt7", "reviewer": "Reviewer_Lr3z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(+) The paper considers an important and challenging problem. The idea discussed is quite interesting and somewhat new. \n\nHowever, there are some downsides of the current submission: \n\n(-) The presentation of the paper needs to be improved. Quite a few places are unclear which need to be better motivated or to be further described. For example, what does $u$ represents in Eqn. 14? Any motivations/insights on why consider Langevin process on the parameter space (the parameter space usually has much higher dimensions compared to the latent codes itself)? How to choose $g(x)$ as in theorem 1? If only the last layer is trainable, does the inference model parametrized in this way has very limited expressive power? Also, in algorithm 2, all the update is on the parameter space including generator as well as encoder. The paper should also need to be well motivated on why performing Langevin on their parameter spaces (as in Eqn 10, 11) as well. \n\n(-) The experiments need to be improved. The sample quality as well as score is relatively weak. Why not use reported FID of the baselines in table 1? For example, the LEBM achieves lower FID on cifar10 on their paper, why not choose their model structure and retrain the proposed model for the comparison?  ", "summary_of_the_paper": "The paper proposes to use amortized MCMC to learn latent space energy-based model (EBM). Specifically, instead of running Langevin dynamics on (latent) data space, such sampling is performed on the parameter space of inference model and is shown to converge to the true posterior under several mild conditions. The experiments demonstrate the generation quality of the proposed model.  ", "summary_of_the_review": "The paper study an important problem, but the current presentation makes it hard to follow. Also the extra experiments and analysis need to be added to motivate and back-up their claims. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CKfl_fACzft", "reviewer": "Reviewer_ZHba", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is not clear to me how to get the function g in the inference network. Is the function g obtained through some preprocessing? How does the quality of it affect the posterior approximation result? How do the authors get it in the experiment section? It will be better to provide more explanation and details about it, since it seems to play an important role in approximating the posterior as shown in Theorem 1.\n\nThe second condition for ALD to converge to the true distribution does not seem mild. Since the dataset size is typically very large, it is impractical to have such high dimension for the linear layer. Therefor the claim seems a bit misleading that “ALD has the target posterior as a stationary distribution with a mild assumption”. \n\nHow does the storage cost of the proposed method compared to LD without amortization and AVI? By looking at Algorithm 1, the storage cost of the proposed method seems much higher, since it has to save Z^1 to Z^n and each Z^i contains several samples.\n\nFor the experiments, it will be better to add the results of SGALD in Figures 1 and 4 to check the posterior estimation when using minibatch of data. Since SGALD is the one being used in LAE in practice. \n\nOn the synthetic distributions, the dimension of the linear layer is much larger than the dataset size (128 vs 3). However in practice the dataset size will be much larger than the dimension of the linear layer (as what the authors did for the experiment on image generation). It will be much convincing to simulate this scenario on synthetic data and verify that with small dimension ALD/SGALD can still approximate posterior well. It will be very helpful to further show how the posterior estimation changes with respect to the dimension of the linear layer.\n\n", "summary_of_the_paper": "This paper introduces amortized Langevin dynamics for latent variables and extends it to unconditional distributions. A Langevin autoencoder is further developed by using amortized Langevin dynamics for prior and posterior sampling. The proposed method has been tested on synthetic distributions and image generation.", "summary_of_the_review": "In summary, I think the idea is reasonable and developing a cheap MCMC-based autoencoder will be of interest to the community. However, I have some concerns with respect to the methodology and the experiments mentioned above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an amortization strategy for MC sampling from a single chain rather than per-datapoint chains, and uses this strategy to define a new Bayesian autoencoder based on Langevin dynamics.

The reviewers find the line of thought very promising, and a potentially interesting addition to the latent variable literature, while also raising some concerns. The dimension of the single chain must match the dataset size, which limits the computational benefits coming from amortization, and in fact this restriction seems hard, as empirical results (added in the discussion period) are qualitatively worse in the `d<n` case. This could be emphasized much more strongly in the current version, and seems worth deeper investigation. In the discussion, the authors agreed that in the case when the feature matrix G is the identity matrix then there can be no amortization improvement, but for other choices of (fixed) features, amortization *can* yield improvements; this is quite unclear. In addition, in response to the reviewers' observation, the authors improved in the discussion period the implementation of the EBM baseline, leading to much less clear cut differences on metrics. To improve the work further, the authors should clarify the source of amortization improvement, and discuss more the relationship to Bayesian Neural Networks (perhaps by evaluating against Bayesian / hyper-net / hyper-GAN generative models.)
Decision: Reject