Adaptive Multi-stage Density Ratio Estimation for Learning Latent Space Energy-based Model
Authors: Zhisheng Xiao
Tian Han
Venue: NeurIPS 2022 Accept (2022)

Abstract: This paper studies the fundamental problem of learning energy-based model (EBM) in the latent space of the generator model. Learning such prior model typically requires running costly Markov Chain Monte Carlo (MCMC). Instead, we propose to use noise contrastive estimation (NCE) to discriminatively learn the EBM through density ratio estimation between the latent prior density and latent posterior density. However, the NCE typically fails to accurately estimate such density ratio given large gap between two densities. To effectively tackle this issue and further learn more expressive prior model, we develop the adaptive multi-stage density ratio estimation which breaks the estimation into multiple stages and learn different stages of density ratio sequentially and adaptively. The latent prior model can be gradually learned using ratio estimated in previous stage so that the final latent space EBM prior can be naturally formed by product of ratios in different stages. The proposed method enables informative and much sharper prior than existing baselines, and can be trained efficiently. Our experiments demonstrate strong performances in terms of image generation and reconstruction as well as anomaly detection.

Adaptive Multi-stage Density Ratio Estimation for Learning Latent Space Energy-based Model
Zhisheng Xiao University of Chicago Chicago, IL, 60637 zxiao@uchicago edu
Tian Han
Stevens Institute of Technology Hoboken, NJ, 07030 than6@stevens.cet edu
Abstract
This paper studies the fundamental problem of learning energy-based model (EBM) in the latent space of the generator model. Learning such prior model typically requires running costly Markov Chain Monte Carlo (MCMC). Instead, we propose use noise contrastive estimation (NCE) discriminatively learn the EBM through density ratio estimation between the latent prior density and latent posterior density. However, the NCE typically fails to accurately estimate such density ratio given large gap between two densities. To effectively tackle this issue and learn more expressive prior models, we develop the adaptive multi-stage density ratio estimation which breaks the estimation into multiple stages and Iearn different stages of density ratio sequentially and adaptively. The latent prior model can be gradually learned using ratio estimated in previous stage so that the final latent space EBM prior can be naturally formed by product of ratios different stages. The proposed method enables informative and much sharper prior than existing baselines, and can be trained efficiently. Our experiments demonstrate strong performances in image generation and reconstruction as well as anomaly detection.
Introduction
Deep generative model provides powerful framework for representing complex data distributions and have seen many successful applications in image and video synthesis 43144355 [1], representation
learning [34] well unsupervised semi-supervised learning [19 37 Such model, referred to
as generator model, usually consists of low-dimensional latent variables that follow non-informative prior distribution, and top-down network that maps such latent vector to the observed example. The informative prior model in the latent space [36] I can be learned to further improve the expressive power of the whole model. Specifically, we consider learning energy-based model (EBM) in the latent space as our informative prior for the generator model.
Learning latent space EBM can be challenging and requires iterative Markov Chain Monte Carlo (MCMC) sampling step which is computationally expensive and sensitive to hyperparameters. In this paper, we instead propose use noise contrastive estimation (NCE) [12] for learning EBM prior via density ratio estimation. The EBM is learned discriminatively by classifying the latent vector sampled from the prior density and the latent sampled from posterior density. Instead of variational learned inference [23] 41 which needs a separate inference network designed, we obtain the posterior latent sample through short-run Langevin dynamics [33] ensure more accurate inference. However, the success of NCE depends on the closeness of prior density and posterior density [18). Given large gap between two densities, NCE typically fails accurately estimate such density ratio which leads to inaccurate EBM modeling.
To effectively tackle the inaccurate estimation issue and further learn more expressive prior model, develop the adaptive multi-stage density ratio estimation for latent space EBM training. The proposed
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

model breaks the density estimation into multiple stages and learn different stages of density ratio sequentially. Thanks the wwwddmeenionaalty of the latent space and the short-run style posterior inference, in each stage, the gap between prior and posterior density could be kept in check which makes NCE easier. The density ratio estimated in previous stage can further integrated into the current prior model as correction term to build more expressive prior density for later stage. With such framework, the final latent space EBM prior can then be naturally formed by product of ratios in different stages top of the initial base prior.
Contributions: 1) we propose an EBM prior on generator model which is modelled through estimation of density ratios in multiple stages. 2) We develop the adaptive multi-stage noise contrastive estimation learn different stages of ratios sequentially and adaptively. The ratio estimated in previous stage can be integrated to form the more informative prior in the later stage. 3) we demonstrate strong empirical results to illustrate the proposed method.
2 Background
2.1 Maximum likelihood learning of deep latent variable models
Let x E RD be an observed example such as an image, and z E Rdd be the latent variables where d < D. A latent variable generative model (a.k.a, generator model) factorize the joint distribution of
(x, 2) as
pe (x, z) = p(z)po 2);
(I)
where p(z) the prior distribution over latent variables 2 Po [x (z))) is the top-down generation model with parameters 6. Usually the prior distribution is chosen to be simple one such as N (0, Ia), but it
can also be more expressive with learnable parameters [36]. The generation model the same as that
in VAE 223 i.e., x go(z) + with go to be the decoder network and e ~ N (0, In),, so that
po (x/2) N' (go(z), o ID) As in VAE, o² takes a pre-specified value.
Given set of N training samples {x;, i = IJ. from the unknown data distribution Paua (x). the model pe can be trained by maximizing the log likelihood over training samples
[(0)= = N Σ log Po (x;). Maximizing the log likelihood L()) can be accomplished by gradient
ascent where the gradient can be obtained from
V Vopo(x)= / No log log, Po (xx,2) 2) dz
V log))o()) (x(
po px(x)
= =Eps (x)x) Vo logg z)].
Ve log (x,, z) can be easily computed according to the form of log (o z). however, approximating the expectation requires drawing samples from pn(z)x), which can be difficult. Sampling from the
intractable posterior pz(x)x) requires MCMC, and one convenient MCMC algorithm is Langevin Dynamics (L.) (30] Given a step size 8 > 0, and an initial value Zo, the Lanegvin dynamics iterates
A-1 z Vz logp)(z)x) + Swiks
where we (O, I). For sufficiently small step size the marginal distribution of zμ will converge to (z(x)) as kt - 0c. However, it is not feasible to run Langevin dynamics until convergence, and in practice the iteration in Eq. 3 is run for finite iterations, which yields Markov chain with an invariant distribution approximately close to the original target distribution. When Zu is initialized from the noise distribution, the algorithm is called
short-run LD 32 33
2.2 Learning EBMs with discriminative density ratio estimation
Suppose there are two distributions with density functions p(x) and g(x) from which we can sample, p(x)
we can estimate the density ratic r((x
by training a classifier to distinguish samples from
x(x
and [44). Specifically, cross-entropy loss
train the binary classifier D R + (0, by minimizing the binary
min - Ex D(x)) Expp(x)) [lg(1 - D(x))
Asssuiing (x)) when p(x)

---

The objective is minimized when Dxx)
(x))p(()) and denoting the classifier at optimality by
((x)
D' (x), we have r(x) p(x]
1 (x)
Equivalently, the ratio r(x)
p(x) can be estimated by o(x)
[(x)
D'(x)
directly minimizing
(x) 1 ro (x) (x))
T (x)
(()) = 1 Exx log
- -xx log 1 tra (x)
where (x(x) is non-negative ratio estimating model implemented as the exponential of an uncon- strained neural network with scalar output. The minimizer o* satisfies Ti- (x)
xxx 12
Such technique can be useful for training Energy-based models (EBMs). Given samples from the true data distribution Paata (x) and base distribution g(x) that we can sample from, we consider EBMs of the form (x(x)= = Tx (x)(x))(x). where Z is the normalizing constant and rg is an unconstrained positive function. With this parametrization, obviously the optimal rs equals the density-ratio (au(x)
g(x]
In fact, if (x) is trained with density ratio estimation, the normalizing constant Z is simply 1. Therefore, the problem of learning an EBM becomes the problem of estimating density-ratio, which can be solved by discriminative density ratio estimation. Typically the base distribution g(x) is chosen be Gaussian, resulted in so-called noise contrastive estimation (NCE) [12]
Although NCE provides promising way to train EBMs without running MCMC, the accuracy of the density ratio estimation depends on the closeness between the two distributions. The ratio estimator is often severely inaccurate when the gap between and q is large. Rhodes et al. [42] propose Telescoping density- Ratio Estimation (TRE), which breaks the density ratio estimation task into a collection of harder sub-tasks and show improvement over simple NCE on density ratio estimation. However, it is still difficult to apply the technique to energy-based modeling. On one hand, EBMs in high- high-dimensional data space such as image space can be highly complex and multi- modal, making them extremely far away from simple noise distribution. On the other hand, the intermediate distributions are pre-designed through linear transition, making them less effective to connect complicated target densities. In Rhodes al. [42] TRE only obtains limited success on training EBMs through density estimation on MNIST dataset.
3 Adaptive Multi-stage Desnity Ratio Estimation
In this section, we introduce adaptive multi-stage density ratio estimation on latent space in details.
3.1 Multi-stage density ratio estimation in latent space
Instead of modeling directly on nigh-dimensional data space, it easier introduce low-dimensional latent variables and learn an EBM in latent space, while also learning a mapping from the latent space to the data space 3 26). We follow this approach and attempt to model latent space EBM using contrastive estimation.
The latent EBM can be learned discriminatively by estimating the ratio between prior density and the posterior density. Due to cow-dimensionality of the latent space, such densities can be much easier to deal with than those in high dimensional data space. However, it presents new challenges. Firstly, while the target density in data space is given and fixed (i.e., empirical data distribution), posterior density in latent space is driven by the prior density and the inference the posterior can be hard. Secondly, while the prior is typically assumed to be un-informative and fixed (e.g., unit Gaussian), the expressiveness of the model is limited.
Inspired by 42 we propose to learn the latent space EBM of the below form through multiple stages
(z) = II row (z) Po(2).
(5)
k=
where pz(z) is the unit Gaussian base distribution, and re, the intermediate density ratio learned in each stage. Such proposed model shares the similar root as the Product-of-Exxe (PoE) [17] where To in each stage can be treated as individual expert model, and has the potential to produce much sharper distribution than the one with single expert model built such as (36).
3

---

stage
stage &
stage
pw(e) N(O. F)
Ps (()r
= pu((iir((
NGE
(n)
(t)
(a))
~ ax(4)
~~ (+)
I (ui(a)
Figure I: Training adaptive multi-stage density ratio estimation. We estimate the density ratio I'k (z) in each stage using contrastive estimation which trains a classifier to distinguish samples from the prior Pe. (z) and samples from the aggregate posterior qu (z). Posterior samples are obtained by short-run LD (blue dashed curve), prior samples can be obtained either by short-run LD (orange dashed curve) or using persistent chain (orange dashed line). The ratio estimated in stage k can be integrated to form a new prior in stage l + 1. The whole prior is adapted across multiple stages and learned sequentially.
Although the formulation of EBM in Eq. | related to the TRE proposed in Rhodes al. 42 training method is fundamentally different. In the next section, we will introduce the training of our model, and highlight the distinction with TRE.
3.2 Learning latent EBMs with adaptive multi-stage density ratio estimation
Our proposed generator model specifies the distribution on joint space (x, p z) = Po (x(z). where pa the prior model specified in Eq. and Ø {ioo.... mm- that
collects parameters for all intermediate learned ratios.
is tempting to apply maximum likelihood estimation (MLE) to train such model. However, there several challenges: (1) learning of latent EBM (z(z) needs costly and hard mixing MCMC sampling. (2) the prior (z) needs have a fixed form during training and cannot be adaptively adjusted. To alleviate the aforementioned limitations, we therefore break the density ratio estimation of pz(z) into m stages, learn and build the prior sequentially and adaptively. Specifically, the 1tth stage, we consider the generator model of the form
Pe.ox (x, z) Pau (x)2)(x(z),
(6)
where Ps. -1)= k-1 likelihood estimation of generation model 0 and the sequential contrastive estimation of prior ø.
To. (z) The whole training procedure iterates between the maximum
MLE for generation model 0: The generation model can be trained by maximizing the marginal log-likelihood po (x). In ttth stage, the complete data log-likelihood of the model po.o, (x, 2) can be expressed as
log pe.cu (x, z) log [oo (z) log Po. (z) 1 | ge(z)]1 /02//o/)]]]0 +C
where is the decoder and C' is a constant independent of 0. The generation model parameter 0 is then updated using the gradient based on Eq. with a batch of training n samples x,:
i 00
Oc+1 0e + 1/
E po, (z. (xc.)
log pe du (xi, 2i)
8=0.
where 7/ is the learning rate. The expectation over the posterior can be approximated by running short-run Lanegvin dynamics in Eq. 3 Note that the running LD to sample from (z (z)x) is equivalent to sample from (x, z) with fixed x.
Adaptive multi-stage NCE for prior şi The prior model pa (2) can be sequentially and adaptively learned to bridge the gap between prior and posterior densities in the previous stages. Specifically. in th stage, the correction term Too be trained to estimate the density ratio between Po. (z) and its

---

aggregated posterior Q& (z) through contrastive estimation using Eq. 4 The appealing advantage of this estimator that it simply trains binary classifier rather than using expensive MCMC sampling. The optimality of such logistic loss leads to the estimated roa. (z)
ge
Poy
The prior model in the (k + 1)th stage can then be sequentially adapted to match the previous aggregated posterior qu (z(2), i.e., letting P (z) Toh (2)Po, (z) match qu (z). Given the new prior, we similarly infer the posterior using short-run LD in Eq. 3 Then, the next density ratio estimator Tóv-1 can be learned through contrastive estimation match the updated prior and its aggregated posterior, which is further used to adapt the prior in next stage. Particularly, we have
qm- 1Z)
4m (z) 9m (()) (z Pdm-1 (z) P
go(z) Po(z)
where (z) is the aggregated posterior for prior Póu. (z). The above telescoping product holds since the new prior is designed match the aggregated posterior in previous stage, i.c., Poa41 (z) ss qx
Each stage estimates the ratio re, (z) =
via contrasive estimation. Then the aggregated Poc (x)
(x)
posterior qm- (z) can be obtained via
m
Qm-1 1 (z) = T T1 (z) Prbm
(z)
II Tox (z) po)); A=
Our final prior model can then be obtained by matching such aggregated posterior qm-1 (z) which has the same form as Eq. []] The proposed training is illustrated in Figure [ and the algorithm is
detailed in Algorithm 1
Comparison with TRE Rhodes et [42 The most significant difference between our training method and TRE is that TRE assumes fixed target distribution and construct multiple stages simultancously via interpolation, whereas our model considers adaptive targets and learn multi-stage density ratio estimators sequentially via NCE. We conduct empirical comparisons in Sec. 5.5
Sampling from prior Pe. (z): The density ratio estimation of res each stage requires the samples from the prior Pe. (z) and posterior. The posterior samples are inferred through short-run Langevin dynamics which can be efficient and accurate. For drawing prior samples from po. (z). we can either use short-run prior Langevin dynamics or persistent update.
One one hand, could directly utilize the short-run Langevin on the Po; (z) obtain prior samples. On the other hand, the samples from prior can also be obtained in persistent chain manner to avoid the prior Langevin altogether. When introducing the (k + 1)th stage of density ratio estimation, we assume that the current estimator rex (z) performs well in modeling the ratio between the current aggregated posterior distribution (z(z) and current prior Pou (z). Therefore, we simply use samples from (z) to approximately serve as samples from the new prior P (z) for the learning of
Tol-1 (z). In practice, it achieved by maintaining memory matrix that stores posterior samples 2; associated to each data point x,. Note that we only need to keep one memory matrix throughout the training, as only the posterior samples from the previous stage are needed.
Test time sampling: After obtaining the density ratio estimators in each stage and form the final EBM prior Po (z), we can sample latent variables ~ (2) and produce sample by decoding z. Sampling from (z) can be done by either running Langvin dynamics with V, lo) =
Vz
log ro. (z) 12) or Haapliing-mmprraane-RRRsssa (SIR) techniques.
4 Related Work
Latent variable deep generative models: Our proposed method aims to improve the performance of latent variable deep generative models. Such models consist of a decoder for generation, and
require an inference mechanism to infer latent variables. VAEs 23 46) learn the decoder network by training a tractable inference network (encoder) to approximate the intractable posterior distribution of the latent variables. Alternatively, Han et al. latent variables by Langevin sampling from the posterior distribution without using a encoder. Our method follows the latter approach that uses Langevin sampling to infer latent variables.
13 14]. Xie et al. [52). Nijkamp et al. (33 infer the
5

---

Algorithm 1: Adaptive Multi-stage Density Ratio Estimation.
input :l.aarnig iterations T, number stages K, observed examples (x. Fi=1, number of posterior sampling steps initial prior model pz(2)
output: Estimated parameters 0, cie,, k ,,.
k =0 for T
1. Mini-hatch: Sample observed examples (x+
Posterior sampling for gı((2) eace x.. sample 2, po, (2)x.) using Eq. (3 for steps with
current prior (z).
3. Learning density ratio réy (z): Update du using contrastive estimation between qu (z) and (z)
via Eq.
4. Learning generation model: update 0 according Eq.
if is a mudtiple of of K then
Stage transition and prior update: Construct new stage k update the prior
[z) T (Z)
Discriminative contrastive estimation for learning generative models: The efforts has been made to combine the discriminative and generative models [27 20 47,, particularly, introduced in Section 2.2 discriminative contrastive estimation can be applied learning EBMs. Gao et al. 7 use a normalizing flow [38] as the base distribution for contrastive estimation. Aneja et al. [ refine the prior distribution of a pre-trained VAE by noise contrastive estimation. However, such method may fail if the empirical latent distribution (called aggregated posterior) is far away from the Gaussian noise. Rhodes et al. 442 propose telescoping density-ratio estimation, which breaks the estimation into several sub-problems. The method is connected to range of methods leverage sequences of intermediate distributions such as 8 29 24].
Generator model with flexible prior: Our method trains an energy-based prior on the latent space by proposed adaptive multi-stage NCE, so our work is related the broader line of previous papers on introducing flexible prior distribution. Tomczak and Welling 45 parameterized the prior based on the posterior inference model, and 2 proposed to construct priors using rejection sampling. Some previous work adopt wwo-stage approach, which first trains a latent variable model with simple prior, and then trains a separate prior model to match the aggregated posterior distribution. For example, 2s- [4) trains another VAE in the latent space; Ghosh et al. 99 fit Gaussian mixture model on latent codes. Additional work in this line include
50
Pang et al. [36] have the closest connection to our work. Similar to us, they introduce an EBM on the latent space. Both the latent space EBM and the generator network are learned jointly by maximum likelihood, and in particular the training involves short-run MCMC sampling from both the prior and posterior distributions. In contrast, we sequentially learn a more expressive EBM with our novel adaptive multi-stage NCE, which avoids running MCMC for EBM prior. We also show improved results on image generation and outlier detection tasks.
Experiments
In this section, we present set of experiments which highlight the effectiveness of our proposed method. We want to show that our method can (i) learn an generator model with expressive prior distribution from which visually realistic images can synthesized, (ii) generalize well by faithfully reconstructing test images during training, and (iii) successfully perform anomaly detection. To show the performance of our method, we mainly include SVHN [31], CelebA (28] and CIFAR-10 (25) in our study. Besides, we also include studies on the training dynamics and the Langevin sampling, as well as ablation studies to better understand our method. Details about the experiments, including network architecture, the choices of the model each dataset can be found in Appendix A
and the optimization method for
5.1 Image Synthesis and Reconstruction
We evaluate the quality of the generated and reconstructed images. Ideally, if the model is well-trained, the FBM prior on latent space will fit the marginal distribution of latent variables, which in turn leads to realistic samples and faithful reconstructions. We benchmark our model against variety

---

Table 1: MSE(L) and FID(L) obtained from models trained on different datasets. For reported results, the FID is computed based on 50k generated images and 50k real images and the MSE computed based on 10k test images.
SVHN
CelebA
CIFAR-I0
MSE
FID
MSE
FID
MSE
FID
VAE
23
0.019
46.78
0.021
65.75
0.057
106.37
ABP
49.71
51.50
SRI 33
0.018
44.86
0.020
61.03
SRI (L=5)
33
0.011
35.32
0.015
47.95
2s-VAE
4
0.019
42.81
0.021
44.40
0.056
72.90
RAF O
0.014
40.02
0.018
40.95
0.027
74.16
NCP- VAE
0.020
33.23
0.021
42.07
0.054
78.06
LEBM
36
0.008
29.44
0.013
37.87
0.020
70.15
Adaptive CE (ours)
0.004
26.19
0.009
35.38
0.008
65.01
2 2
34 3
10 24
8:19:
728
66
15
94
9
66
95
1 1386
15 2
3:
9
21
5 21 1Z
29
4
L5 822
(a) SVHN
(b) CelebA
(c) CIFAR-10
Figure 2: Samples generated from our models trained on SVHN, CelebA and CIFAR- 10 datasets.
of previous methods including VAE 2x
Alternating Back-propogation (ABP) 13 and Short-run
Inference (SRI) (33] which assume simple standard Gaussian prior distribution for the latent vector, well recent two-stage methods such as 2-stage VAE 44, RAE 9 and NCP- (T], whose prior distributions are learned with posterior samples in a second stage after the generator is trained. We also compare method with LEBM (36), which learns EBM prior adaptively during training the generator, while the EBM prior is trained by maximum likelihood instead of density ratio estimation. To make fair comparisons, follow the protocol in 36
We report the quantitative results of FID [16] in Table [ where we observe that across all datasets, proposed method achieves superior generation performance compared to baseline models based with simple or learned prior distribution.
Synthesis:
We show qualitative results of generated samples in Figure 2 where we observe that our model can generate diverse, sharp and high-quality samples. Additional qualitative samples are presented in Appendix C To test our method's 's scalability, we trained a larger generator on CelebA-HO (128 x 128) and show samples in Appendix C the model can produce realistic samples.
Reconstruction: Note that the posterior Langevin dynamics should not only help to learn the latent space EBM prior model but also produce samples that approximately come from true posterior distribution (z of the generator model. To verify this, we evaluate the accuracy of the posterior inference by looking at reconstruction error on images. We quantitatively compare reconstructions of test images with baseline models using mean square error (MSE) in Table [ We observe that our method consistently obtain lower reconstruction error than competing methods do. We also provide qualitative results of reconstruction in Appendix B
5.2 Anomaly Detection
Anomaly detection another task to evaluate the generator model. With a generator and an EBM prior model trained on in-distribution data, the posterior pe (z|x) would have separated probability
7

---

Table AUPRC scores for unsupervised anomaly detection on MNIST. Numbers are taken from and results for our model are averaged over last 10 trials to account for variance.
36
Heldout Digit
4
VAE
0.063
0.337
0.325
0.148
0.104
ABP 13
0.095 = 0.03
0.138 + 0.04
0.147
0.03
0.138 + 0.02
0.102 = 0.03
MEG 26
0.281 = 0.04
0,401 = 0.06
0,402 + 0.06
0.290 + 0.04
0.342 = 0.03
BiGAN-o 55
0.287 + 0.02
0.443 + 0.03
0.514
0.317 0.02
0.307 0.03
I.BBM
36
0.336 + 0.01
0.630 + 0.02
0.619
0.463 0.01
0.413 0.01
Adaptive CE (ours)
0.531 = 0.02
0.729
0.02
0.742 + 0.01
0.620 0.02
0.499 0.01
densities for in-distribution and out-of-distribution (anomalous) samples. In particular, we decide whether a test sample x is anomalous or not by first sampling from the posterior pe pz (z|xx) x) by short-run Langevin dynamics, and then computing the joint density pe px, z) pa (x p (z). A
higher value of log joint density indicates the test sample more likely be normal sample. Some
prior work on using latent variable generative model for anomaly detection includes [49 15 36].
Following the experimental settings in [26 55], anomalous class and leave the other classes as normal. Note that it is a challenging task and all previous methods do perform well. To evaluate the performance, we to compute the area under the precision-recall curve (AUPRC) [6]. We compare our method with where observe that method obtains significant improvements.
set each class in the MNIST dataset
the log-posterior density
related models in Table 2
5.3 Analyzing Training Loss
we plot the evolution of the density ratio estimation loss (Eq. 4 for each stage of estimation during training. Our experiment has estimation stages, resulted estimators. We observe that the loss for the first stage, which estimates the density ratio between unit Gaussian prior (z) and aggregated posterior is significantly lower than later stages, which estimate the ratio between the updated prior and updated posterior. This observation is consistent with intuition: directly discriminating between Gaussian prior and posterior is very easy, while introducing additional stages of estimation make the task more difficult, and hence the estimated density ratio is more reliable. Higher losses in later stages also suggests that the prior is getting close to aggregated posterior, the discrimination becomes harder.
In Figure 4
density ratio
5.4 Analyzing Langevin Dynamies
stage
stage
stage 2
stage
0.8
06
0.4
02
2000
4000
6000
8000
10000
Figure 3: Transition of Langevin dynamics ini- tialized from (z) towards P:o (z) for 200 steps.
Figure 4: Density ratio estimation loss for each estimation stage.
In Figure 3. we visualize the transition of Langevin dynamics initialized from (z(z) towards Ps (z) on model trained on CelebA. The LD iterates for 200 steps, which longer than the LD for training (30 steps). We expect that with well-trained pz(2), the trajectory of a Markov chain should transit towards samples of higher quality. Indeed, we observe that the quality of synthesis improves significantly with as the LD progresses. In addition, we observe human faces with different
8

---

Table 3: Results for ablation study on CelebA dataset.
Method
Adaptive
Non-ad Non-adaptive
# of stages
0
2
8
0
2
8
FID
62.78
44.17
39.85
35.38
35.84
62.78
43.84
42.61
42.48
43.06
identities along the LDD, suggesting that the Markov chain can mix between different modes of the prior distribution. This indicates that the density function of learned EBM prior has a smooth geometry that allows MCMC to mix well.
5.5 Ablation Study
To better understand our proposed method, we conduct ablation study on number of density ratio estimators and training methods. We use CelebA for the ablation experiments.
Number stages. The most important hyper-parameter of our method the number of density ratio estimators, equivalently, the number of training stages. We present the FID score of models trained with different number of stages the first part of T The line of stage means no latent EBM at all, i.e., simply training generator model by short-run inference and sampling from it by decoding z ~ pz()).
We make the following observations. Firstly, directly sampling latent variables from leads to poor results, while any latent EBM trained by density ratio estimation significantly improves the performance, suggesting the necessity of learning the latent EBM. Secondly, we see that multi-stage density ratio estimation can further significantly improve the performance of single-stage estimation. The results indicate that multi stage density ratio estimation facilitates the training of latent EBM by gradually making the estimation task harder. We observe that the FID score does not improve for more than 4 stages, and therefore we choose 4 as the number of stages for our main experiments.
Training method: adaptive vs. non-adaptive. is important to distinguish our method from TRE in Rhodes et al. [42]. TRE assumes the target distribution to be fixed, therefore, if we adopt TRE, the posterior distribution (z(z)x) will be fixed one throughout the training. In contrast, our training method is adaptive in the sense that the target posterior is updated by incorporating the current EBM prior into the joint distribution when new stage introduced. To quantitatively compare these two approaches, we also train non-adaptive version of the model and report the numbers in the second part of B We observe that models trained with non adaative multi-stage density ratio estimation obtain significantly worse results. Therefore, we believe that it is crucial to learn density ratios sequentially with adaptive posterior.
5.6 Parameter Efficiency
One potential disadvantage of our method is its parameter inefficiency from multiple estimator networks. Moreover, since the training is sequential, we cannot share parameters between estimators done in Rhodes et al. [22. Fortunately, our EBM is the latent space so the network is light- weighted. For example, with 4 density ratio estimators, the number of parameters in the prior EBM is only around 1% of the number of parameters in the generator. In addition, confirm the larger number of parameters the latent EBM is not the cause of improvements, as we train a single stage model with 4x size and observe no improvement.
Conclusions
In this paper, we propose adaptive multi- state- density ratio estimation, which is an effective method for learning a EBM prior for generator model. Our method learns the latent EBMs by introducing multiple density ratio estimators that learn the density ratio between prior and posterior sequentially and adaptively. We demonstrate the effectiveness of our method by conducting comprehensive experiments, and empirical results show the advantage of our method on generation, reconstruction and anomaly detection tasks.
As future directions, our method can potentially be applied modeling the latent space of generator models in other domains, such as text [35] and graph. We also tend to develop more advanced and efficient inference schemes for posterior density estimation.

---

References
[1] Jyoti Aneja, Alexander Schwing, Jan Kautz, and Arash Vahdat. A contrastive learning approach for training variational autoencoder priors. In Neural Information Processing Systems (NeurIPS), 2021.
[2] Matthias Bauer and Andriy Mnih. Resampled priors for variational autoencoders. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 66-75. PMLR, 2019.
[3] Yoshua Bengio, Grégoire Mesnil, Yann Dauphin, and Salah Rifai. Better mixing via deep representations. In International conference on machine learning, pages 552-560. PMLR, 2013.
[4] Bin Dai and David Wipf. Diagnosing and enhancing vae models. arXiv preprint arXiv:1903.05789, 2019.
[5] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 12873-12883, 2021.
[6] Tom Fawcett. An introduction to roe analysis. Pattern recognition letters, 27(8):861-874, 2006.
[7] Ruiqi Gao, Erik Nijkamp, Diederik Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu. Flow contrastive estimation of energy-based models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7518-7528, 2020.
[8] Andrew Gelman and Xiao-Li Meng. Simulating normalizing constants: From importance sampling to bridge sampling to path sampling. Statistical science, pages 163-185, 1998.
[9] Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. arXiv preprint arXiv: /2436, 2019.
[10] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pages 446..56..1 JMLR Workshop and Conference Proceedings, 2010.
[11] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[12] Michael U. Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of Machine Learning Research, 13(1 11:307737-3 2012.
[13] Tian Han, Yang Lu, Song-Chun Zhu, and Ying Nian Wu. Alternating back propagation for generator network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[14] Tian Han, Yang Lu, Jiawen Wu, Xianglei Xing, and Ying Nian Wu. Learning generator networks for dynamic patterns. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 809-818. IEEE, 2019.
[15] Jakob D Havtorn, Jes Frellsen, Søren Hauberg, and Lars Maaløe. Hierarchical vaes know what they don't 't know. In International Conference on Machine Learning, pages 4117-4128, PMLR, 2021.
[16] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler. and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[17] Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800, 2002.
[18] Mathew D Hoffman and Matthew J Johnson. Elbo surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference,
NIPS, volume 1, 2016.
[19] Pavel Izmailov, Polina Kirichenko, Mare Finzi, and Andrew Gordon Wilson. Semi-supervised learning with normalizing flows. In International Conference on Machine Learning, pages 4615-4630. PMLR, 2020.
[20] Long Jin, Justin I Lazarow, and Zhuowen Tu. Introspective classification with convolutional nets. Advances in Neural Information Processing Systems, 30, 2017.
10

---

[21]
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4401-4410, 2019.
[22] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6968 2014.
[23] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. In The International Conference on Learning Representations (ICLR). 2014.
[24] Scott Kirkpatrick, C Daniel Gelatt, and Mario P Vecchi. Optimization by simulated annealing. science, 220(4598):671 1660, 1983.
[25] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 10 (canadian institute for advanced research). URL http://www. cs. toronto. edu/kriz/cifar himl, 5:4, 2010.
[26] Rithesh Kumar, Sherjil Ozair, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Maximum entropy generators for energy-based models. arXiv preprint arXiv: 1901.0850,88 2019.
[27] Justin Lazarow, Long Jin, and Zhuowen Tu. Introspective neural networks for generative modeling. In Proceedings of the IEEE International Conference Computer Vision, pages 2774-2783, 2017.
[28] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015.
[29] Enzo Marinari and Giorgio Parisi. Simulated tempering: new monte carlo scheme. EPL (Europhysics Letters), 19(6):51,, 1992.
[30] Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 2011.
[31] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 201 2011.
[32] Erik Nijkamp. Mitch Hill, Song-Chun Chun Zhu, and Ying Nian Wu. Learning non-convergent non- persistent short-run meme toward energy-based model. In Proceedings of the 33rd International Conference Neural Information Processing Systems, pages -232-5242, 2019.
[33] Erik Nijkamp, Bo Pang. Tian Han, Linqi Zhou, Song-Chun Zhu, and Ying Nian Wu. Learning multi-layer latent variable model via variational optimization of short run meme for approximate inference. In European Conference on Computer Vision, pages 361-378, 2020.
[34] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. arXiv preprint arXiv:1711.00%
2017.
[35] Bo Pang and Ying Nian Wu. Latent space energy-based model of symbol-vector coupling for text generation and classification. In International Conference Machine Learning, pages 8359-8370. PMLR, 2021.
[36] Bo Pang, Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy-based prior model. NeurIPS, 2020.
[37] Bo Pang, Erik Nijkamp, Jiali Cui, Tian Han, and Ying Nian Wu. Semi-supervised learning by latent space energy-based model of symbol-vector coupling. arXiv preprint arXiv:20100.93359 2020.
[38] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, (2(5)):1-64 2021.
[39] Giorgio Patrini, Rianne van den Berg, Patrick Forre, Marcello Carioni, Samarth Bhargav, Max Welling, Tim Genewein, and Frank Nielsen. Sinkhorn autoencoders. In Uncertainty in Artificial Intelligence, pages 733-743. PMLR, 2020.
[40] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 2015.
[41] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. In International conference on machine learning, pages 1278-1286. PMLR, 2014.

---

[42] Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratie estimation. arXiv preprint arXiv:2006. 12204, 2020.
[43] Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kobayashi. Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal gan. International Journal of Computer Vision, 128:2586-2666, 2020.
[44] Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.
[45] Jakub Tomezak and Max Welling. Vae with a vampprior. In International Conference on Artificial Intelligence and Statistics, pages 1214-1223, PMLR, 2018.
[46] Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Neural Information Processing Systems (NeurIPS), 2020.
discriminative, descriptive and generative models. arXiv preprint arXiv: arXiv:1810.04200 04261, 2018. [47] Ying Nian Wu, Ruiqi Gao, Tian Han, and Song- Chun Zhu. A tale of three probabilistic families:
[48] Zhisheng Xiao, Qing Yan, and Yali Amit. Generative latent flow. arXiv: 1905. 10485, 2019.
arXiv preprint
[49] Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An out-of-distribution detection score for variational auto-encoder. Advances in neural information processing systems, 33: 20685-20696, 2020.
[50] Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. Vaebm: A symbiosis between variational autoencoders and energy-based models. In International Conference on Learning Representations, 2021.
[51] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. In International Conference on Learning Representations, 2022.
[52] Jianwen Xie, Ruiqi Gao, Zilong Zheng. Song-Chun Zhu, and Ying Nian Wu. Learning dynamic generator model by alternating baco-propag through time. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5498-5507 2019.
[53] Peiyu Yu, Sirui Xie, Xiaojian Ma, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu. Unsupervised foreground extraction via deep region competition. Advances in Neural Information Processing Systems, 34: 1424-1142799 2021.
[54] Peiyu Yu, Sirui Xie, Xiaojian Ma, Baoxiong Jia, Bo Pang, Ruigi Gao, Yixin Zhu, Song-Chun Zhu, and Ying Nian Wu. Latent diffusion energy-bascc model for interpretable text modeling. arXiv preprint arXiv:2206. 05895, 2022.
[55] Houssam Zenati, Chuan Sheng Foo, Bruno Lecouat, Gaurav Manek, and Vijay Ramaseshan Chandrasekhar. Efficient gan-based anomaly detection. arXiv preprint arXiv:/802.06222 2018.
12


Reviews:
[{"review_id": "qr5S4nOOiHDR", "reviewer": "Reviewer_PebL", "summary": "This paper proposes a new adaptive multi-stage density ratio estimation framework for learning latent space energy-based model (EBM) in order to overcome the density-chasm problem. Namely, whenever the gap between p (prior distribution) and q (posterior data distribution), the classifier can obtain almost perfect accuracy with a relative poor estimate of the density ratio. The adaptive multi-stage density ratio estimation follow the telescope density-ratio estimation (TRE) approach, learns the latent space EBM through multiple stages. They claim that their work shares similar root as the Product-of-Expert (PoE), each intermediate ratio is treated as individual expert model and it has the potential to produce much sharper distribution than a single model. In training their proposed work, instead of using MLE, they resort to \nmulti-stage NCE for obtaining prior models (also avoid using expensive MCMC sampling) and short-run Langevin Dynamics (LD) to infer the posterior. Their empirical results show the effectiveness of the proposed work. They also do some ablation studies to show how to determine stage numbers and the importance of adaptiveness. ", "questions": "1. Following the sequential nature of this work, we wonder if this work is really much more efficient than MCMC. \n2. The selection of stage number as four based on FID scores is not so convincing. As we mentioned before, this lacks theoretical guarantee and looks not generic either. \n3. Further, they still resort to short-run LD for posterior inferences while current standard EBM usually also make use of short-run LD for inferences. So, one question we have is when the number of stages is quite many, can is it more efficient than the standard EBM models, ", "limitations": "1. Please add theoretical analysis or proofs on the stage choices or sample efficiencies if possible \n2. Looks that the constraint of adaptive to sequential learning is not convincing. Is it possible to combine sequential and distributional? \n3. They did not run experiments with TRE, which is the most closely related work. So, we do not have ideas whether their work can really work better than related works. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The idea of learning EBM via multi-stage density estimation is somewhat novel and also workable for overcoming the density-chasm problem. \n2. This work has done intensive ablation studies to show the importance of stage numbers and whether to use adaptive or non-adaptive. \n3. They have done experiments in SVHN, CelebA and CIFAR-10 to demonstrate the effectiveness of their proposal. \n4. They also provide the workable code and this make reproduction and extensions more feasible.\nWeaknesses:\n1. This work can still be regarded as an incremental research from the Telescoping density-ratio estimation. \n2. Compared to TRE, it seems that it is much less efficient since each divided component of TRE is done simultaneously and can be trained in a distributional fashion while this work is done sequentially and later stage depends on previous stages. \n3. This work lacks theoretical analysis and proofs to understand the effectiveness and efficiency of proposed approaches. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AMnquyVmDTI", "reviewer": "Reviewer_o7M9", "summary": "The authors propose a new method for unsupervised learning of latent models that consist of an energy based prior model (EBPM) and a (stochastic) deep decoder.  For the EBPM estimation, the authors propose to extend the approach of Telescoping density-Ratio Estimation (TRE) (Rhodes et al 2020).  The utility of the approach is evaluated on several image modeling benchmarks (SHVN, CelebA /CelebA-HD, CIFAR-10) and on an outlier detection task on MNIST.  ", "questions": "* what is the tradeoff between the increased complexity of EBPM vs a more complex decoder?  Current generative SoTA for CelebA-HD is certainly not based on the simple decoder model architectures used here (c.f., StyleGAN type of models).\n* Please comment on the increase in computational complexity of learning (time-complexity) compared to eg simple Vamp prior\n", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The proposed approach is clearly described, well-motivated, and appears technically correct\n+ While the approach uses the same telescoping/recursive approximation idea as TRE, the idea is applied in a sufficiently novel manner to EBPM + generative model estimation\n+ All results point to clear benefits of the proposed approach; competing approaches constitute a reasonable selection of SoTAs for unsupervised latent model estimation + most related approach (LEBM)\n+ authors provide insightful analyses of ablation wrt #stages and ratio estimation \"hardness\"\n\nWeaknesses:\n- while I do not see any major downsides to the paper, it would be useful to comment on the increased complexity (during learning and inference) introduced by the telescoping ratio-based EBPM\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "90-dYOFdiSd", "reviewer": "Reviewer_JvzX", "summary": "This paper proposes using EBMs in the latent space before being pushed through a latent variable model. The paper proposes learning this EBM using NCE instead of via MCMC sampling. The difference between the prior and the latent posterior is estimated using short-run langevin instead of a recognition network (variational inference). This technique is applied repeatedly to obtain several “stages” of density ratio estimation. \n\nThe paper demonstrates their approach on image generation and reconstruction, as well as on anomaly detection. \n", "questions": "Based on figure 4, it sounds like there are decreasing marginal returns to using additional stages. Is this a correct assessment? How do you propose to choose the number of stages?\n", "limitations": "I see no potential for negative societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The experiment in Section 5.4 on analyzing langevin dynamics is very interesting. It uniquely shows the benefit of using a latent variable model, with the prior learned by an EBM, and how this results in stable, well-mixed markov chains.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept