Year: 2021

Abstract:
This paper studies the cooperative learning of two generative flow models, in which the two models are iteratively updated based on the jointly synthesized examples. The first flow model is a normalizing flow that transforms an initial simple density into a target density by applying a sequence of invertible transformations. The second flow model is a Langevin flow that runs finite steps of gradient-based MCMC toward an energy-based model. We start from proposing a generative framework that trains an energy-based model with a normalizing flow as an amortized sampler to rapidly initialize the MCMC chains of the energy-based model. In each learning iteration, we generate synthesized examples by using a normalizing flow initialization followed by a short-run Langevin flow revision toward the current energy-based model. After that, we treat the synthesized examples as fair samples from the energy-based model and update the model parameters with the maximum likelihood learning gradient, while the normalizing flow directly learns from the synthesized examples by maximizing the tractable likelihood. We show that, under the short-run non-mixing MCMC scenario, the estimation of the energy-based model actually follows the perturbation of maximum likelihood, and both the short-run Langevin flow and normalizing flow form a two-flow generator that we call CoopFlow. We provide an understating of the CoopFlow algorithm by information geometry and show that it is a valid generator because it converges to a moment matching estimator. We demonstrate that the trained CoopFlow is capable of synthesizing realistic images, reconstructing images, and interpolating between images.

heading: 1 INTRODUCTION AND MOTIVATION
text: Normalizing flows (Dinh et al., 2015; 2017; Kingma & Dhariwal, 2018a) are a family of generative models that construct a complex distribution by transforming a simple probability density, such as Gaussian distribution, through a sequence of invertible and differentiable mappings. Due to the tractability of the exact log-likelihood and the efficiency of the inference and synthesis, normalizing flows have gained popularity in density estimation (Kingma & Dhariwal, 2018a; Ho et al., 2019; Yang et al., 2019; Prenger et al., 2019; Kumar et al., 2020) and variational inference (Rezende & Mohamed, 2015; Kingma et al., 2016). However, for the sake of ensuring the favorable property of closed-form density evaluation, the normalizing flows typically require special designs of the sequence of transformations, which, in general, constrain the expressive power of the models.
Energy-based models (EBMs) (LeCun et al., 2006) define an unnormalized probability density function of data, which is the exponential of the negative energy function. The energy function is directly defined on data domain, and assigns each input configuration with a scalar energy value, with lower energy values indicating more likely configurations. Recently, with the energy function parameterized by a modern deep network such as a ConvNet, the ConvNet-EBMs (Xie et al., 2016) have gained unprecedented success in modeling large-scale data sets (Gao et al., 2018; Nijkamp et al., 2019; Du & Mordatch, 2019; Grathwohl et al., 2020; Gao et al., 2021a; Zhao et al., 2021) and exhibited stunning performance in synthesizing different modalities of data (Xie et al., 2017; 2018b; 2021a; Du et al., 2020). The parameters of the energy function can be trained by maximum likelihood. However, due to the intractable integral in computing the normalizing constant, the eval-
uation of the gradient of the log-likelihood typically requires Markov chain Monte Carlo (MCMC) sampling (Barbu & Zhu, 2020), such as gradient-based Langevin dynamics (Neal et al., 2011), to generate samples from the current model.
However, the Langevin sampling on a highly multi-modal energy function, due to the use of deep network parameterization, is generally not mixing. When sampling from a density with a multimodal landscape, the Langevin dynamics, which follows the gradient information, is apt to get trapped by local modes of the density and is unlikely to jump out to explore other isolated modes. Relying on non-mixing MCMC samples, the estimated gradient of the likelihood can be biased, and the resulting learned energy-based model may become an invalid model, which is unable to approximate the data distribution as expected.
Recently, Nijkamp et al. (2019) propose to train an EBM with a short-run non-convergent Langevin dynamics, and show that even though the energy function is invalid, the short-run MCMC can be treated as a valid flow model that generates realistic examples. This not only provides an explanation of why EBMs with non-convergent MCMC are still capable of synthesizing realistic examples, but also suggests a more practical computationally-affordable way to learn useful generative models under the existing energy-based learning frameworks. Although EBMs have been widely applied to different domains, learning short-run MCMC in the context of EBM is still underexplored.
In this paper, we will accept the fact that MCMC sampling is not mixing in practice, and decide to give up the goal of training a valid EBM of data. Instead, we treat the short-run non-convergent Langevin dynamics, which shares parameters with the energy function, as a flow-like transformation that we call the Langevin flow because it can be considered as a noise-injected residual network. Even though implementing a short-run Langevin flow is surprisingly simple, which is nothing but a design of a bottom-up ConvNet for the energy function, it might still require a sufficiently large number of Langevin steps (each Langevin step consists of one step of gradient descent plus a diffusion step) to construct an effective Langevin flow, so that it can be expressive enough to represent and synthesize the data. Motivated by reducing the number of Langevin steps in the Langevin flow for computational efficiency, we propose the CoopFlow model that trains a Langevin flow jointly with a normalizing flow in a cooperative learning scheme, in which the normalizing flow model learns to serve as a rapid sampler to initialize the Langevin flow so that the Langevin flow can be shorter, while the Langevin flow teaches the normalizing flow by short-run MCMC transition toward the EBM so that the normalizing flow can accumulate the temporal difference in the transition to provide better initial samples. Besides, the motivation of using the normalizing flow instead of a generic generator, is because the cooperative learning framework need to incorporate the MLE algorithm of an EBM and the MLE algorithm of a generator. The MLE of a normalizing flow generator will be much more tractable than the MLE of any other generic generator. The latter might resort to either MCMC-based inference to evaluate the posterior distribution or another encoder network for variational inference. In the proposed CoopFlow, the Langevin flow can overcome expressivity limitation of the normalizing flow due to the invertibility constraint. Such a benefit also motivates us to study the CoopFlow framework. To be specific, we understand the dynamics of cooperative learning with short-run non-mixing MCMC by information geometry. We provide justifications that the CoopFlow trained in the context of EBM with non-mixing MCMC is a valid generator, because it converges to a moment matching estimator. Experiments,including image generation, image reconstruction, and latent space interpolation, are conducted to support our justification.

heading: 2 CONTRIBUTIONS AND RELATED WORK
text: Our paper studies amortized sampling for training a short-run non-mixing Langevin sampler toward an EBM. We propose a novel framework, the CoopFlow, which cooperatively trains a short-run Langevin flow as a valid generator and a normalizing flow as an amortized sampler for image representation. We provide both theoretical and empirical justifications for the proposed CoopFlow algorithm. The following are some closely related work. We will point out the differences between our work and the prior arts to further highlight the contributions and novelty of our paper.
Learning short-run MCMC as a generator Recently, Nijkamp et al. (2019) propose to learn an EBM with short-run non-convergent MCMC that samples from the model, and they eventually keep the short-run MCMC as a valid generator and discard the biased EBM. Nijkamp et al. (2020b)
use short-run MCMC to sample the latent space of a top-down generative model in a variational learning framework. An et al. (2021) proposes to correct the bias of short-run MCMC inference by optimal transport in training latent variable model. Pang et al. (2020) adopts short-run MCMC to sample from both the EBM prior and the posterior of the latent variables. Our work studies learning a normalizing flow to amortize the sampling cost of a short-run non-mixing MCMC sampler (i.e., Langevin flow) in data space, which makes a further step forward in this underexplored theme.
Cooperative learning with MCMC teaching Our learning algorithm is inspired by the cooperative learning or the CoopNets (Xie et al., 2020), in which an EBM and a top-down generator (Han et al., 2017) are jointly trained by jump-starting their maximum learning algorithms. Xie et al. (2021b) replaces the generator in the original CoopNets by a VAE (Kingma & Welling, 2013) for efficient inference. The proposed CoopFlow algorithm is different from the above prior work in the following two aspects. (i) In the idealized long-run mixing MCMC scenario, our algorithm is a cooperative learning framework that trains an unbiased EBM and a normalizing flow via MCMC teaching, where the updating of the normalizing flow with a tractable density is more efficient and less biased than updating a generic generator via variational inference as in Xie et al. (2021b) or MCMC-based inference as in Xie et al. (2020). (ii) Our paper has a novel emphasis on studying cooperative learning with short-run non-mixing MCMC, which is more practical and common in reality. Our algorithm actually trains a short-run Langevin flow and a normalizing flow together toward a biased EBM for image generation. We use information geometry to understand the learning dynamics and show that the learned two-flow generator (i.e., the CoopFlow) is a valid generative model, even though the learned EBM is biased.
Joint training of EBM and normalizing flow There are other two works studying EBM and normalizing flow together. To avoid MCMC, Gao et al. (2020) propose to train an EBM using noise contrastive estimation (Gutmann & Hyvärinen, 2010), where the noise distribution is a normalization flow. Nijkamp et al. (2020a) propose to learn an EBM as an exponential tilting of a pretrained normalizing flow, so that neural transport MCMC sampling in the latent space of the normalizing flow can mix well. Our paper proposes to train an EBM and a normalizing flow via short-run MCMC teaching. More specially, we focus on short-run non-mixing MCMC and treat it as a valid flow-like model (i.e., short-run Langevin flow) that is guided by the EBM. Disregarding the biased EBM, the resulting valid generator is the combination of the short-run Langevin flow and the normalizing flow, where the latter serves as a rapid initializer of the former. The form of this two-flow generator shares a similar fashion with the Stochastic Normalizing Flow (Wu et al., 2020) that consists of a sequence of deterministic invertible transformations and stochastic sampling blocks.

heading: 3 TWO FLOW MODELS

heading: 3.1 LANGEVIN FLOW
text: Energy-based model Let x ∈ RD be the observed signal such as an image. An energy-based model defines an unnormalized probability distribution of x as follows:
pθ(x) = 1
Z(θ) exp[fθ(x)], (1)
where fθ : RD → R is the negative energy function and defined by a bottom-up neural network whose parameters are denoted by θ. The normalizing constant or partition function Z(θ) =∫ exp[fθ(x)]dx is analytically intractable and difficult to compute due to high dimensionality of x.
Maximum likelihood learning Suppose we observe unlabeled training examples {xi, i = 1, ..., n} from unknown data distribution pdata(x), the energy-based model in Eq.(1) can be trained from {xi} by Markov chain Monte Carlo (MCMC)-based maximum likelihood estimation, in which MCMC samples are drawn from the model pθ(x) to approximate the gradient of the log-likelihood function for updating the model parameters θ. Specifically, the log-likelihood is L(θ) = 1n ∑n i=1 log pθ(xi). For a large n, maximizing L(θ) is equivalent to minimizing the Kullback-Leibler (KL) divergence DKL(pdata||pθ). The learning gradient is given by
L′(θ) = Epdata [∇θfθ(x)]− Epθ [∇θfθ(x)] ≈ 1
n n∑ i=1 ∇θfθ(xi)− 1 n n∑ i=1 ∇θfθ(x̃i), (2)
where the expectations are approximated by averaging over the observed examples {xi} and the synthesized examples {x̃i} generated from the current model pθ(x), respectively.
Langevin dynamics as MCMC Generating synthesized examples from pθ(x) can be accomplished with a gradient-based MCMC such as Langevin dynamics, which is applied as follows
xt+1 = xt + δ2
2 ∇xfθ(xt) + δϵt; x0 ∼ p0(x), ϵt ∼ N (0, ID), t = 1, · · · , T, (3)
where t indexes the Langevin time step, δ denotes the Langevin step size, ϵt is a Brownian motion that explores different modes, and p0(x) is a uniform distribution that initializes the MCMC chains.
Langevin flow As T → ∞ and δ → 0, xT becomes an exact sample from pθ(x) under some regularity conditions. However, it is impractical to run infinite steps with infinitesimal step size to generate fair examples from the target distribution. Additionally, convergence of MCMC chains in many cases is hopeless because pθ(x) can be very complex and highly multi-modal, then the gradient-based Langevin dynamics has no way to escape from the local modes, so that different MCMC chains with different starting points are unable to mix. Let p̃θ(x) be the distribution of xT , which is the resulting distribution of x after T steps of Langevin update starting from x0 ∼ p0(x). Due to fixed initial distribution p0(x) and fixed K and δ, the distribution p̃θ(x) is well defined, which can be implicitly expressed by
p̃θ(x) = (Kθp0)(x) = ∫ p0(z)Kθ(x|z)dz, (4)
where Kθ denotes the transition kernel of T steps of Langevin dynamics that samples pθ. Generally, p̃θ(x) is not necessarily equal to pθ(x). p̃θ(x) is dependent on T and s, which are omitted in the notation for simplicity. The KL-divergence DKL(p̃θ(x)||pθ(x)) = −entropy(p̃θ(x))− Ep̃θ(x)[f(x)] + logZ. The gradient ascent part of Langevin dynamics, xt+1 = xt + δ2/2∇xfθ(xt), increases the negative energy fθ(x) by moving x to the local modes of fθ(x), and the diffusion part δϵt increases the entropy of p̃θ(x) by jumping out of the local modes. According to the law of thermodynamics (Cover & Thomas, 2006), DKL(p̃θ(x)||pθ(x)) decreases to zero monotonically as T →∞.

heading: 3.2 NORMALIZING FLOW
text: Let z ∈ RD be the latent vector of the same dimensionality as x. A normalizing flow is of the form x = gα(z); z ∼ q0(z), (5) where q0(z) is a known prior distribution such as Gaussian white noise distribution N (0, ID), and gα : RD → RD is a mapping that consists of a sequence of L invertible transformations, i.e., g(z) = gL ◦ · · · ◦ g2 ◦ g1(z), whose inversion z = g−1α (x) and log-determinants of the Jacobians can be computed in closed form. α are the parameters of gα. The mapping is used to transform a random variable z that follows a simple distribution q0 into a flexible distribution. Under the change-of-variables law, the resulting random variable x = gα(z) has a probability density qα(x) = q0(g −1 α (x))|det(∂g−1α (x)/∂x)|. Let hl = gl(hl−1). The successive transformations between x and z can be expressed as a flow z g1←→ h1 g2←→ h2 · · · gL←→ x, where we define z := h0 and x := hL
for succinctness. Then the determinant becomes |det(∂g−1α (x)/∂x)| = ∏L
l=1 |det(∂hl−1/∂hl)|. The log-likelihood of a datapoint x can be easily computed by
log qα(x) = log q0(z) + L∑ l=1 log ∣∣∣∣det(∂hl−1∂hl )∣∣∣∣ . (6)
With some smart designs of the sequence of transformations gα = {gl, l = 1, ..., L}, the logdeterminant in Eq.(6) can be easily computed, then the flow-based model can be trained by maximizing the data log-likelihood L(α) = ∑n i log qα(xi).

heading: 4 COOPFLOW: COOPERATIVE TRAINING OF TWO FLOWS

heading: 4.1 COOPFLOW ALGORITHM
text: We have moved from trying to use a convergent Langevin dynamics to train a valid EBM. Instead, we accept the fact that the short-run non-convergent MCMC is inevitable and more affordable in
practice, and we treats a non-convergent short-run Langevin flow as a generator and propose to jointly train it with a normalizing flow as a rapid initializer for more efficient generation. The resulting generator is called CoopFlow, which consists of both Langevin flow and normalizing flow.
Specifically, at each iteration, for i = 1, ...,m, we first generate zi ∼ N (0, ID), and then transform zi by a normalizing flow to obtain x̂i = gα(zi). Next, starting from each x̂i, we run a Langevin flow (i.e., a finite number of Langevin steps toward an EBM pθ(x)) to obtain x̃i. x̃i are considered as the synthesized examples that are generated by the CoopFlow models. We then update α of the normalizing flow by treating x̃i as training data, and update θ of the Langevin flow according to the learning gradient of the EBM, which is computed with the synthesized examples x̃i and the observed examples. Algorithm 1 presents a description of the proposed CoopFlow algorithm. The advantage of this training scheme is that we only need to minimally modify the existing code for the MLE training of EBM pθ and normalizing flow qα. The density of the CoopFlow π(θ,α)(x) is well defined, which can be implicitly expressed by
π(θ,α)(x) = (Kθqα)(x) = ∫ qα(x ′)Kθ(x|x′)dx′. (7)
If we increase the length of Langevin flow, π(θ,α) will converge to EBM pθ(x). The network fθ(x) in Langevin flow is scalar valued and is of free form, whereas the network gα(x) in normalizing flow has high-dimensional output and is of a severely constrained form. Thus Langevin flow can potentially provide a tighter fit to pdata(x) than normalizing flow. On the flip side, sampling from Langevin flow requires MCMC, whereas normalizing flow can synthesize examples directly via ancestral sampling. It is thus desirable to train these two flows, where normalizing flow serves as an approximate sampler to amortize the sampling cost of Langevin flow. Meanwhile the normalizing flow can be updated by a temporal difference MCMC teaching scheme provided by Langevin flow.
Algorithm 1 CoopFlow Algorithm Input: (1) Observed images {xi}ni ; (2) Numbers of Langevin steps T ; (4) Langevin step size δ; (5) Learning rate for Langevin flow ηθ; (6) Learning rates for normalizing flow ηα; batch size m. Output: Parameters {θ, α}
1: Randomly initialize θ and α. 2: repeat 3: Sample observed examples {xi}mi ∼ pdata(x). 4: Sample noise examples {zi}mi=1 ∼ p0(z), 5: Start from zi, generate x̂i = gα(zi) via normalizing flow. 6: Start from x̂i, run a T -step Langevin flow to obtain x̃i by following Eq.(3). 7: Given {x̃i}, update α by maximizing ∑ i log qα(x̃i) in Eq.(6). 8: Given {xi} and {x̃i}, update θ by following the gradient∇θ in Eq.(2). 9: until converged

heading: 4.2 UNDERSTANDING THE LEARNED TWO FLOWS
text: Convergence equations In the traditional contrastive divergence (CD) algorithm, MCMC chains are initialized with observed data so that the CD learning seeks to minimize DKL(pdata(x)||pθ(x))− DKL((Kθpdata)(x)||pθ(x)), where (Kθpdata)(x) denotes the marginal distribution obtained by running the Markov transition K, which is specified by the Langevin flow, from the data distribution. In the proposed CoopFlow algorithm, the learning of the EBM (or the Langevin flow model) follows a modified contrastive divergence, where the initial distribution of the Langevin flow is modified to be a normalizing flow qα. Thus, at iteration t, the update of θ follows the gradient of DKL(pdata||pθ) − DKL(Kθ(t)qα(t) ||pθ) with respect to θ. Compared to the traditional CD loss, the modified one replaces pdata by qα in the second KL divergence term. At iteration t, the update of the normalizing flow qα follows the gradient of DKL(Kθ(t)qα(t) ||qα) with respect to α. Let (θ∗, α∗) be a fixed point the learning algorithm converges to, then we have the following convergence equations
θ∗ = argmin θ DKL(pdata||pθ)− DKL(Kθ∗qα∗ ||pθ), (8) α∗ = argmin α DKL(Kθ∗qα∗ ||qα). (9)
Ideal case analysis In the idealized scenario where the normalizing flow qα has infinite capacity and the Langevin sampling can mix and converge to the sampled EBM, Eq.(13) means that qα∗ attempts to be the stationary distribution of Kθ∗ , which is pθ∗ because Kθ∗ is the Markov transition kernel of a convergent MCMC sampled from pθ∗ . That is, minα DKL(Kθ∗qα∗ ||qα) = 0, thus qα∗ = pθ∗ . With the second term becoming 0 and vanishing, Eq.(12) degrades to minθ DKL(pdata||pθ) and thus θ∗ is the maximum likelihood estimate. On the other hand, the normalizing flow qα chases the EBM pθ toward pdata, thus α∗ is also a maximum likelihood estimate.
Moment matching estimator In the practical scenario where the Langevin sampling is not mixing, the CoopFlow model πt = Kθ(t)qα(t) is an interpolation between the learned qα(t) and pθ(t) , and it converges to π∗ = Kθ∗qα∗ , which is an interpolation between qα∗ and pθ∗ . π∗ is the short-run Langevin flow starting from qα∗ towards EBM pθ∗. π∗ is a legitimate generator because Epdata [∇θfθ∗(x)] = Eπ∗ [∇θfθ∗(x)] at convergence. That is, π∗ leads to moment matching in the feature statistics∇θfθ∗(x). In other words, π∗ satisfies the above estimating equation.
Consider a simple EBM with fθ(x) = ⟨θ, h(x)⟩, where h(x) is the feature statistics. Since ∇θfθ(x) = h(x), the MLE of the EBM pθMLE is a moment matching estimator due to Epdata [h(x)] = EpθMLE [h(x)]. The CoopFlow π
∗ also converges to a moment matching estimator, i.e., Epdata [h(x)] = Eπ∗ [h(x)]. Figure 1 is an illustration of different model distributions that correspond to different parameters at convergence. We first introduce three families of distributions: Ω = {p : Ep[h(x)] = Epdata [h(x)]}, Θ = {pθ(x) = exp(⟨θ, h(x)⟩)/Z(θ),∀θ}, and A = {qα,∀α}, which are shown by the red, blue and green curves respectively in Figure 1. Ω is the set of distributions that reproduce statistical property h(x) of the data distribution. Obviously, pθMLE , pdata, and π
∗ = Kθ∗qα∗ are included in Ω. Θ is the set of EBMs with different values of θ, thus pθMLE and pθ∗ belong to Θ. Because of the short-run Langevin flow, pθ∗ is not a valid model that matches the data distribution in terms of
h(x), and thus pθ∗ is not in Ω. A is the set of normalizing flow models with different values of α, thus qα∗ and qαMLE belong to A. The yellow line shows the MCMC trajectory. The solid segment of the yellow line, starting from qα∗ to Kθ∗qα∗ , illustrates the short-run non-mixing MCMC that is initialized by the normalizing flow qα∗ in A and arrives at π∗ = Kθ∗qα∗ in Ω. The dotted segment of the yellow line, starting from π∗ = Kθ∗qα∗ in Ω to pθ∗ in Θ, shows the potential long-run MCMC trajectory, which is not really realized because we stop short in MCMC. If we increase the number of steps of short-run Langevin flow, KL(π∗||pθ∗) will be monotonically decreasing to 0. Though π∗ stops midway in the path toward pθ∗, π∗ is still a valid generator because it is in Ω.
The EBM with MLE pθMLE is the intersection between Θ and Ω. It is the projection of pdata onto Θ because θMLE = argminθ DKL(pdata||pθ). The normalizing flow with MLE qαMLE is also a projection of pdata onto A because αMLE = argminα DKL(pdata||qα). Generally, qαMLE is far from pdata due to its restricted form network, whereas pθMLE is very close to pdata due to its scalar-valued free form network. Because the short-run Langevin flow is not mixing, θ∗ ̸= θMLE and α∗ ̸= αMLE. Specifically, θ∗ and α∗ are perturbations of θMLE and αMLE, respectively. pθMLE shown by an empty dot is not attainable unfortunately. Note that Ω, as well as Θ and A, is high-dimensional manifold instead of 1D curve as depicted in the Figure, and π∗ may be farther away from pdata than pθMLE is. During learning, qα(t+1) is the projection of Kθ(t)qα(t) on A. At convergence of the learning algorithm, there is an infinite looping, i.e., Kθ∗qα∗ (i.e., π∗) lifts qα∗ off the ground A, and the projection drops Kθ∗qα∗ back to qα∗ .

heading: 5 EXPERIMENTS
text: In this section, we showcase the experiment results on various tasks. We start from a toy example to illustrate the basic idea of the CoopFlow algorithm in Section 5.1. Then we show the image generation results in section 5.2. Section 5.3 demonstrates the learned CoopFlow can be useful for image reconstruction and inpainting, while Section 5.4 shows that the learned latent space is meaningful so that it can be used for interpolation.

heading: 5.1 TOY EXAMPLE STUDY
text: We first demonstrate our idea using a two-dimensional toy example where data lies on a spiral. We train 3 CoopFlow models with different lengths of Langevin flows. As shown in Figure 2, the rightmost box shows the results obtained with 10,000 Langevin steps. We can see in this situation, both the normalizing flow qα and the EBM model pθ can capture the underlying distribution perfectly. This validates that, with a sufficiently long Langevin flow, the CoopFlow algorithm can learn both a valid EBM and a valid normalizing flow. The leftmost green box represents the model trains with 100-step Langevin updates. This is a more typical non-convergent MCMC setting. In this case, neither qα nor pθ are perfect. Instead, they need to work together and the Langevin updates process over EBM model actually works as an extra generator that modifies the initial proposal by the normalizing flow model. We can see that the samples from qα are not perfect, but after the modification, the samples from π(θ,α) follow the underlying distribution well. The middle green box shows the results obtained with 500-step Langevin steps. It is in the middle of short-run and long-run. pθ becomes better but still not perfect. Using the toy example here, we want to make the points that short-run non-convergent MCMC situation might still exist even when we have a generator for initial proposal and using relatively long MCMC chain. Besides, if we use the MCMC on EBM as a special generator, i.e. Langevin Flow, then together with the normalizing flow, we can get pretty good samples even with few MCMC updates.

heading: 5.2 IMAGE GENERATION
text: We test our model on 3 image datasets for image synthesis. (i) CIFAR-10 (Krizhevsky et al., 2009) is a dataset containing 50k training images and 10k testing images in 10 classes; (ii) SVHN (Netzer et al., 2011) is a dataset containing over 70k training images and over 20k testing images of house numbers; (iii) CelebA (Liu et al., 2015) is a celebrity facial dataset containing over 200k imagess. We downsample all the images to the resolution of 32× 32. For our model, we show results of three different settings. CoopFlow denotes the setting where we train a normalizing flow and a Langevin flow together from scratch and use 30 Langevin steps. CoopFlow(long) denotes the setting that we increase the number of Langevin updates to 200. In CoopFlow(Pre) setting, we first pretrained a normalizing flow from observed data, and then we train the CoopFlow with the parameters of the normalizing flow being initialized by the pretrained one. We use a 30-step long Langevin flow in this setting. For all the three settings, we slightly increase the step size of Langevin update at the test stage for better performance. For more training details, please refer to Appendix.
We show both the qualitative results in Figure 4 and quantitative results in Table 1. To calculate FID Heusel et al. (2017) scores, we generate 50,000 samples on each dataset. From these results, we can see that our models outperform most of the baseline algorithms. We get lower FID scores comparing to the purely normalizing flow models and the prior works that try to jointly train a normalizing flow with an EBM, e.g., Gao et al. (2020); Nijkamp et al. (2020a). We also achieve comparable results with the SOTA EBMs. Among our different settings, we can see using a longer Langevin flow
consistently improves the performance. And using a pretrained normalizing flow can provide the lowest FID scores among all the three settings. More results can be found in Appendix.
5.3 IMAGE RECONSTRUCTION
We show that the learned CoopFlow model is able to reconstruct observed images. We may consider the CoopFlow model π(θ,α)(x) to be a latent variable generative model: z ∼ p0(z); x̂ = gα(z);x = Fθ(x̂, e), where z denotes the latent variables, e denotes all the injected noises in the Langevin flow, and Fθ denotes the mapping realized by a T -step Langevin flow that is actually a T -layer noise-injected residual network. Since the Langevin flow is not mixing, x is dependent on x̂ in the Langevin flow, thus also dependent on z. The CoopFlow is a generator x = Fθ(gα(z), e), so we can reconstruct any x by inferring the corresponding latent variables z using gradient descent on L(z) = ||x−Fθ(gα(z), e)||2, with z being initialized by p0. However, g is an invertible transformation, so we can infer z by an efficient way, i.e., we first find x̂ by gradient descent on L(x̂) = ||x− Fθ(x̂, e)||2, with x̂ being initialized by x̂0 = gα(z) where z ∼ p0(z) and e being set to be 0, and then use z = g−1α (x̂) to get the latent variables. These two methods are equivalent, but the latter one is computationally efficient, since computing the gradient on the whole two-flow generator Fθ(gα(z), e) is difficult and time-consuming. Let x̂∗ = argminx̂ L(x̂). The reconstruction is given by Fθ(x̂
∗). The optimization is done using 200 steps of gradient descent over x̂. The results are shown in Figure 3. We do experiment on CIFAR10 test set. The right column displays the original images x that need to be reconstructed. The left column and the middle column display x̂∗ and Fθ(x̂∗), respectively. We can see that our model can successfully reconstruct the observed images, which verifies that the CoopFlow with a non-mixing MCMC is a valid latent variable generator.
We further show that our model is also capable of doing image inpainting. Similar to image reconstruction, given a masked observation xmask with a binary matrix M indicating the positions of the unmasked pixels, we optimize x̂ to minimize the reconstruction error between Fθ(x̂) and xmask in the unmasked area, i.e., L(x̂) = ||M ⊙ (xmask − Fθ(x̂))||2, where ⊙ is the element-wise multiplication operator. x̂ is still initialized by the normalizing flow. We do experiment on Celeba training set. In Figure 5, each row shows one example of inpainting with a different initialization of x̂ provided by the normalizing flow and the first 17 columns display the inpainting results at a
different optimization iteration. The last two columns show the masked images and the original images respectively. We can see that our model can reconstruct the unmasked area faithfully and simultaneously fill in the blank areas in the input images. The recovered images are meaningful and given different initializations, our model can inpaint diversified patterns.

heading: 5.4 INTERPOLATION IN THE LATENT SPACE
text: The CoopFlow model is capable of doing interpolation in the latent space z. Given an image x, we first find its corresponding x̂∗ using the reconstruction method described in section 5.3. We then infer z by the inversion of the normalizing flow z∗ = g−1α (x̂
∗). The results of interpolation in latent space experiments on Celeba (training set) are shown in Figure 6. Comparing the first and second columns to the left (or to the right), We can see that our reconstructions of the original images are good. Further, our interpolations are smooth and the generated images are meaningful.

heading: 6 CONCLUSION
text: This paper studies an interesting problem of learning two types of deep flow models in the context of energy-based framework for image representation and generation. One is the normalizing flow model that generates synthesized examples by transforming Gaussian noise examples through a sequence of invertible transformations, while the other is the Langevin flow that generates synthesized examples by running a non-mixing, non-convergent short-run MCMC toward an EBM. We propose the CoopFlow algorithm to train the short-run Langevin flow model jointly with the normalizing flow serving as a rapid initializer in a cooperative manner. The experiments show that the CoopFlow is a valid generative model that can be useful for image generation, reconstruction, and interpolation.

heading: A APPENDIX

heading: A.1 MODEL ARCHITECTURE
text: We use the same model structure for all the experiments. For the normalizing flow model, we use the Flow++ (Ho et al., 2019) structure designed for the CIFAR-10 dataset. For the EBM model, we use the structure shown in Table 2.

heading: A.2 EXPERIMENTAL SETTING
text: We have tried three different settings. In CoopFlow setting and CoopFlow(long) setting, we train both normalizing flow model and Langevin flow model from scratch. The difference between them are the number of Langevin steps use. The CoopFlow(Long) use a relatively long chain of 200 steps MCMC updates. Following Ho et al. (2019), we use data-dependent parameter initipalization (Salimans & Kingma, 2016) for our normalizing flow model. On the other hand, in the CoopFlow(Pre) setting, we first pretrained the normalizing flow model using true observed data. Then we use this pretrained normalizing flow model as initialization and train it together with our Langevin flow model using Algorithm alg1. To make this switching smoother, we include a warm-up phase for Langevin flow model. During this phase, instead of updating both the normalizing flow model and Langevin flow model from the beginning, we first fix the weight of pretrained normalizing flow model and only update the weight of EBM. After a certain number of epochs of updating, the Langevin flow model itself has been able to roughly capture the underlying data distribuion. Then we beginning to update both the normalizing flow and Langevin flow together. This may prevent the Langevin flow from proposing bad samples at the begining of training and affecting the pretrained normalizing flow model.
During training, we use 1e-4 as learning rate for both the normalizing flow and Langevin flow model, we use Adam (Kingma & Ba, 2014) as our optimizer. We use βs = (0.9, 0.999) for normalizing flow model and βs = (0.5, 0.5) for Langevin flow. We keep monitoring the FID scores during training and pick the best one over the training trajectory. For data augmentation, we only include random horizontal flip for the CIFAR-10 dataset and do not apply other tricks. Given we are actually using the short-run MCMC setting, we remove the noise term in Langevin updates so that the updates becomes gradient descent over the energy function. The other hyper-parameter setting can be found in Table 3.
We further show the influence of Langevin step size and number of Langevin steps on the CIFAR10 dataset using the CoopFlow(Pre) model. We first fix the Langevin step size to be 0.03 and try different number of Langevin updates from 10 to 50. The results are shown in Table 4. On the other hand, we show the influence of Langevin Step size in Table 5. Here, we keep the number of Langevin updates to be 30 and we adjust the training Langevin step size. We keep the testing step size and training step size to have a ratio of 43 . We can see that our choice of 30 as number of Langevin steps and 0.03 as Langevin step size is reasonable. Based on the comparison between CoopFlow and CoopFlow(Long), further increase the number of Langevin steps to a relatively long chain may helps with the FID score, but given long MCMC chain can be computationally expensive, we keep to use 30 steps here.

heading: A.3 ABLATION STUDY
text: Here we compare the results of training normalizing flow and Langevin flow using or Cooperative algorithm with training them separately. Training flow itself follows the original normalizing flow training algorithm (we use the flow++ (Ho et al., 2019) structure here, which is the same as the one used in our CoopFlow model). And for training Langevin flow individually, we use the short-run MCMC proposed by Nijkamp et al. (2019) with our own EBM structure. Through our experiments, we find that removing the noise injection to both observed data and Langevin updates can give us better result. We show generated samples of normalizing flow and short-run EBM on CIFAR-10 dataset in Figure 7 and FID scores in Table 6. We can see that the CoopFlow outperforms both the normalizing flow and short-run EBM by a large margin.

heading: A.4 MORE GENERATION RESULTS
text: In section 5.2, we show the samples from CoopFlow(Pre). Here, we show the samples generated by CoopFlow in Figure 8 and samples generated by CoopFlow(Long) in Figure 9. We can see that all the samples are meaningful. On the other hand, we the initial samples from the normalizing flow trained under the CoopFlow setting in Figure 10. Comparing Figure 4 and Figure 10, we can see that there is a gap between the normalizing flow samples and the final CoopFlow outputs. The initial samples from normalizing flow are blurred and after modified by the Langevin flow, the samples become sharp and clear. This agrees with our discussion in section 4.2. The CoopFlow(Long) setting and CoopFlow(pre) setting give similar observations. We show more image inpainting results in Figure 11. In Figure 12, we show the FID score curve over training epochs. The FID keep decreasing as the learning proceeds. We can see the FID curves converge to a low value for both CoopFlow setting and CoopFlow(pre) on CIFAR-10 dataset. This is an evidence to show that our algorithm is a convergent algorithm.

heading: A.5 QUANTITATIVE RESULTS FOR THE IMAGE RECONSTRUCTION EXPERIMENT
text: Following Nijkamp et al. (2019), we calculate the per pixel MSE on 1,000 examples in the leave-out testset of CIFAR-10 dataset. We use a 200-step gradient descent to minimize the reconstruction loss. We plot the MSE per update iteration in Figure 13 and we report the final per pixel MSE in Table 7. For a baseline, we test the reconstruction of the same 1,000 images using the short-run EBM model which has the same EBM structure as our CoopFlow and uses 100 steps of MCMC. We can see the CoopFlow works better than the short-run EBM in this image reconstruction task.

heading: A.6 COMPARISON OF MODEL SIZES
text: In Table 8, we present an comparison of different models according to their size and the FID scores they got. Here we mainly compare those models that employ a normalizing flow as a generator, e.g. EBM-FCE, NT-EBM, GLOW or EBM jointly trained with other generator structure, e.g., VAEBM. We can see our CoopFlow achieves a good computation and performance balance. On one hand, we have similar number of parameters as EBM-FCE but has a much lower FID. Also, the Flow++ only represents the setting that we train the normalizing flow model with the same structure as the one used in our CoopFlow. Comparing it to CoopFlow, we can see that introducing a Langevin Flow increases the FID a lot. On the other hand, the VAEBM model achieves better FID than us, but they need to use a pretrained NVAE model (Vahdat & Kautz, 2020), which is much larger than the one we used here.

heading: A.7 ANALYSIS OF THE NUMBER OF LANGEVIN STEPS
text: In this section, we compare our method with other baselines in generating examples with the same number of Langevin steps. We compare our methods with two closely related baselines, (1) the single EBM with short-run MCMC starting from noise distribution (Nijkamp et al., 2019), and (ii) Cooperative learning of an EBM and a generic generator (Xie et al., 2018a). The numbers of MCMC steps and FID reported in Table 9 are collected from the original papers. In Table 10, we also report the FID scores of different baseline methods using the different numbers of MCMC steps. We can see that with the same number of Langevin steps, the CoopFlow can generate much more realistic image patterns than the other two baselines. The results of the experiments show that the proposed CoopFlow can use less number of MCMC steps to achieve better results.

heading: A.8 MORE CONVERGENCE ANALYSIS
text: The CoopFlow algorithm simply involves two MLE learning algorithms: (i) the MLE learning of the EBM pθ and (ii) the MLE learning of the normalizing flow qα. The convergence of each of the two learning algorithms has been well studied and verified in the existing literature, e.g., Younes
(1999); Xie et al. (2016); Kingma & Dhariwal (2018b). That is, each of them has a fixed point. The only interaction between these two algorithms in the proposed CoopFlow algorithm is that, in each iteration, they feed each other with their synthesized examples and use the cooperatively synthesized examples in their parameter update formulas. To be specific, the normalizing flow uses its synthesized examples to initialize the MCMC of the EBM, and the EBM feed the normalizing flow with its synthesized examples as training examples. The synthesized examples from the Langevin flow are considered the cooperatively synthesized examples by two models, and used to compute their learning gradients. Unlike other amortized sampling methods (e.g., Han et al. (2019); Grathwohl et al. (2021)) that uses variational learning, the EBM and normalizing flow in our framework don’t back-propagate each other through the cooperatively synthesized examples. They just feed each other with some input data for their own training algorithms. That is, each learning algorithm will still converge and have a fixed point.
Now let us analyze the convergence of the whole CoopFlow algorithm that alternates two maximum likelihood learning algorithms. We first analyze the convergence of the objective function at each learning step, and then we conclude the convergence of the whole algorithm.
The convergence of CD learning of EBM. The learning objective of the EBM is to minimize the KL distance between the EBM and the data distribution. Since the MCMC of EBM in our case is initialized by the normalizing flow, it follows a modified contrastive divergence algorithm. That is, at iteration t, it has the following objective,
θ(t+1) = argmin θ DKL(pdata||pθ)− DKL(Kθ(t)qα(t) ||pθ). (10)
No matter what kind of distribution is used to initialize the MCMC, it will has a fixed point when the learning gradient of θ equals to 0, i.e., L′(θ) = 1n ∑n i=1∇θfθ(xi) − 1 n ∑n i=1∇θfθ(x̃i) = 0. The initialization of the MCMC only affects the location of the fixed point of the learning algorithm. The convergence and the analysis of the fixed point of contrastive divergence has been studied by Hinton (2002); Carreira-Perpiñán & Hinton (2005).
The convergence of MLE learning of normalizing flow The normalizing flow’s objective is to learn to minimize the KL divergence between the normalizing flow and the Langevin flow because, in each learning iteration, the normalizing flow uses the EBM’s synthesized examples as training data. At iteration t, it has the following objective
α(t+1) = argmin α DKL(Kθ(t)qα(t) ||qα), (11)
which is a convergent algorithm at each t. The convergence has been studied by Younes (1999).
The convergence of CoopFlow The CoopFlow alternates the above two learning algorithms. The EBM learning will reduce the KL distance between EBM and data, i.e., pθ → pdata; while the MLE learning of normalizing flow will reduce the KL distance between normalizing flow and EBM, i.e., qα → pθ. Therefore, the normalizing flow will chase the EBM toward the data distribution
gradually. Because the process pθ → pdata will stop at a fixed point, therefore qα → pθ will also stop at a fixed point obviously. Such a chasing game is a contraction algorithm, therefore the fixed point of CoopFlow exists. Empirical evidence has also supported that our EM-like learning algorithm is a contraction. If we use (θ∗, α∗) to denote the fixed point of the CoopFlow, according to the definition of a fixed point, (θ∗, α∗) will satisfy
θ∗ = argmin θ DKL(pdata||pθ)− DKL(Kθ∗qα∗ ||pθ), (12) α∗ = argmin α DKL(Kθ∗qα∗ ||qα). (13)
The convergence of the cooperative learning framework (CoopNets) that integrates MLE algorithm of an EBM and MLE algorithm of a generic generator has been verified and discussed in Xie et al. (2020). The CoopFlow that uses a normalizing flow instead of a generic generator has the same convergence property as that of the original CoopNets. The major contribution in our paper is to start from the above fixed point equation to discuss where the fixed point will be in our algorithm when the MCMC of the EBM is non-mixing and non-convergent. This will go beyond all the prior works about cooperative learning.

heading: A.9 DISCUSSION ON THE BIAS OF THE CONTRASTIVE DIVERGENCE OBJECTIVE FOR EBM
text: Bias in CD learning. The EBM trained by contrastive divergence (CD) still follows the “analysis by synthesis” scheme to update the model parameters θ. The learning gradient of θ is still computed by
L′(θ) = Epdata [∇θfθ(x)]− Epθ [∇θfθ(x)] ≈ 1
n n∑ i=1 ∇θfθ(xi)− 1 n n∑ i=1 ∇θfθ(x̃i), (14)
except that the synthesized examples x̃ are sampled by a short-run MCMC initialized by the training data pdata. For example, the CD-1 is commonly used to train Restricted Boltzmann machines (Hinton, 2002), and “1” means the single-step MCMC. We denote x̃ ∼ Kθpdata(x), where Kθ is the MCMC transition kernel of T steps of MCMC that samples pθ, and (Kθpdata)(x) is the distribution of x after T steps of MCMC update starting from qdata. Because x̃ ∼ Kθpdata(x) ̸= pθ(x), therefore, the estimated θ using the above equation is biasd (i.e., it is not an MLE solution, but it still a convergent algorithm with a fixed point.) Hinton (2002) point out that the CD learning actually minimizes the following objective, which is a perturbation of the MLE,
min θ DKL(pdata||pθ)︸ ︷︷ ︸ MLE loss −DKL(Kθpdata||pθ)︸ ︷︷ ︸ bias due to the gap , (15)
where the first term is the MLE due to the use of Eq.(14), the second term is the bia due to the gap between the actually used sampling distribution Kθpdata and the correct distribution pθ(x) we should sample from. Only if DKL(Kθpdata||pθ) = 0, the CD estimate will be unbiased. However, the original CD doesn’t explicitly minimize the second term. Recently, Du et al. (2021) improves the CD algorithm by explicitly minimizing the second term that has been neglected in the original CD to further reduce the bias. We need to point out that, even though the CD is biased, it is still a convergent algorithm, whose convergence has been well studied in its original paper (Hinton, 2002). “Biasd” doesn’t mean it is wrong. “Biasd” is just because DKL(Kθpdata||pθ) ̸= 0. Bias in amortized inference Actually, many useful algorithms and models using amortized sampling or amortized inference are biased (with respect to MLE). For example, variational autoencoder (VAE) (Kingma & Welling, 2013), whose objective is also a perturbation of MLE,
min α,ϕ DKL(pdata(x)||qα(x))︸ ︷︷ ︸ MLE loss +DKL(πϕ(z|x)||qα(z|x))︸ ︷︷ ︸ bias due to variational inference
(16)
where qα is the distribution of the decoder/generator in VAE, and πϕ(z|x) is the encoder to approximate the true posterior qα(z|x). Even though VAE explicitly minimizes the second term (i.e., bias) in its loss, but in practice, this term can not be zero due to the mean-field design of the inference model πϕ, the model α is still biased to MLE. Even though our model is not related to VAE, we just want to mention that “bias” is not a bad thing, it is just a trade-off between computation efficiency and the exact MLE solution due to the usage of amortized sampling/inference.
Bias in CoopFlow Now let us go back to our CoopFlow algorithm. The update of the EBM in our CoopFlow can be considered a modified CD algorithm, because its MCMC starts from the normalizing flow qα. At iteration t, the update of the EBM follows the following objective
θ(t+1) = argmin θ DKL(pdata||pθ)︸ ︷︷ ︸ MLE loss − DKL(Kθ(t)qα(t) ||pθ)︸ ︷︷ ︸ bias due to amortized sampling , (17)
which is biased with respect to MLE if DKL(Kθ(t)qα(t) ||pθ) ̸= 0. However, the CoopFlow algorithm has the other part that can handle the second term. It is the MLE learning of the normalizing flow qα(x), with x̃ ∼ Kθ(t)qα(t) as training examples. That is, at iteration t, we fix Kθ(t)qα(t) and minimize the following objective
α(t+1) = argmin α DKL(Kθ(t)qα(t) ||qα.) (18)
Eq.(18) tries to reduce the KL distance between the flow and the EBM. In an ideal case where capacities of all models are large enough and the MCMC Kθ is mixing and convergent MCMC, minimizing Eq.(18) will lead to DKL(Kθqα||qα)→ 0, which means qα = pθ because qα has become the stationary distribution of Kθ. If qα = pθ = 0, then the second term of Eq.(17) will be vanishing because DKL(Kθqα||pθ) = DKL(Kθpθ||pθ) = 0, the Eq.(17) only has the first term left, which is the MLE solution. Since qα = pθ, thus α is also MLE solution. Therefore, our estimates of θ and α in the CoopFlow are unbiased in the ideal case.
However, in practice, the conditions, such as large enough capacity of models and convergent MCMC are very hard to get. Eventually, the CoopFlow might converge to a fixed point but not the fixed point of MLE. We have used information geometry to illustrate the fixed point in Figure 1 of the main text, and analyze the biased estimator in the paragraph “moment matching estimator” of section 4.2. That is, even though the CoopFlow is not MLE solution, but it is a moment matching estimator (MME), which is a perturbation of MLE shown in Figure 1. MLE is shown as pθMLE in Figure 1, and the obtained MME solution is marked as a star π∗. Again, bias doesn’t means wrong. πθ,α with MME is a valid generator.

heading: A.10 COMPARISON BETWEEN COOPFLOW AND SHORT-RUN EBM VIA INFORMATION GEOMETRY
text: Figure 14 illustrates the convergences of both CoopFlow and the EBM with a short-run MCMC starting from an initial noise distribution q0. We define the short-run EBM in a more generic form (Xie et al., 2016)
pθ(x) = 1
Z(θ) exp[fθ(x)]q0(x), (19)
which is an exponential tilting of a known reference distribution q0(x). In general, the reference distribution can be either a Gaussian distribution or a uniform distribution. When the reference distribution is a uniform distribution, q0 can be removed in Eq.(19). Since the initial distribution of the CoopFlow is a Gaussian distribution q0, which is actually the prior distribution of the normalizing flow. For a convenient and fair comparison, we use Gaussian distribution as the reference distribution of the EBM in Eq.(19). And both CoopFlow and the baseline short-run EBM use the same EBM defined in Eq.(19) in their frameworks. We use p̄θ̄ to denote the baseline short-run EBM.
There are three families of distributions: Ω = {p : Ep[h(x)] = Epdata [h(x)]}, Θ = {pθ(x) = exp(⟨θ, h(x)⟩)q0(x)/Z(θ),∀θ}, and A = {qα,∀α}, which are shown by the red, blue and green curves respectively in Figure 14. Figure 14 is an extension of Figure 1 by adding the following elements. (i) q0, which is the initial distribution for both CoopFlow and short-run EBM. It belongs to Θ because it corresponds to θ = 0. q0 is just a noise distribution thus it is far under the green curve. That is, it is very far from qα∗ because qα∗ has been already a good approximation of pθ∗ . (ii) gα∗ , which is the learned transformation of the normalizing flow qα, and is visualized as a mapping from q0 to qα∗ by a directed brown line segment. (iii) The MCMC trajectory of the baseline shortrun EBM p̄θ̄, which is shown by the yellow line on the right hand side of the blue curve. The solid part of the yellow line, starting from q0 to π̄∗ = Kθ̄∗q0, shows the short-run non-mixing MCMC starting from the initial Gaussian distribution q0 in Θ and arriving at π̄∗ in Ω. The dotted part of the yellow line is the potential long-run MCMC trajectory that is unrealized. To compare the MCMC
trajectories of the CoopFlow and the short-run EBM, we can find that CoopFlow has a much shorter MCMC trajectory than that of the short-run EBM, since the normalizing flow amortizes the sampling workload for the Langevin flow in the CoopFlow model.

Review summaries:
The authors propose a cooperative learning framework of normalizing flow and energy-based Langevin MCMC models (CoopFlow). Specifically, the normalizing flow suggests a better initialization for the Langevin flow, then the latter generates synthetic samples via short-run MCMC. While the Langevin flow is trained with a standard MCMC likelihood toward the data distribution, the normalizing flow chases the Langevin one by maximizing the likelihood of the synthetic samples. The proposed CoopFlow seems to outperform each of its components.

The paper proposes a generative model that consists of two components: a normalizing flow and an energy-based model with short-run MCMC as proposed in (Nijkamp, 2019). The whole procedure operates by pushing some simple distribution through the NF and then running the short-run chain on these samples. Both models are learned jointly as follows. The normalizing flow tries to capture the limiting distribution of the EBM by minimizing  $$\text{KL}(K_\theta q_\alpha \Vert q_\alpha),$$  where $q_\alpha$ is the density of the NF with parameters $\alpha$ and $K_\theta$ is the transition kernel of the short-run chain with parameters $\theta$. Simultaneously, the EBM is learned to capture the data by minimizing  $$\text{KL}(p_{\text{data}}\Vert p_\theta) - \text{KL}(K_\theta q_\alpha\Vert p_\theta).$$  In Section 4.2, the authors provide some analysis of the proposed procedure. Namely, the authors consider the perfect scenario (where the NF matches the EBM density and they both match data) and give some intuition for the case when the models are not expressive enough to perfectly match the data.  Finally, the authors provide an empirical study of the proposed procedure. They report metrics for image generation on CIFAR10, SVHN, Celeba (downsampled to 32x32); give several examples of image inpainting and latent space interpolation.

This paper proposes to train an energy-based model with a normalizing flow to achieve rapid and high-quality MCMC sampling.  It shows the trained CoopFlow is capable of synthesizing toy data and realistic images (reasonable compared to many baseline methods in terms of common evaluation metrics).  Applications to image reconstruction and interpolation are also included. 

This paper develops a new generative model, called CoopFlow, by combining the energy-based model (EBM), normalizing flow (NF) and Langevin dynamics (LD). The CoopFlow updates all the parts in an cooperative manner: Parameters of NF are updated by using samples modified by LD, LD is performed based on the potential function provided by the EBM, and the gradient of log-likelihood of EBM are estimated according to samples generated by NF + LD. Here, LD can overcome the limitation of the model capacity of NF, and NF + LD can provide a more accurate estimation of the gradient of EBM loss. After learning, a powerful sampler and an accurate EBM can be obtained.